>Does anyone know the hex value or the header file that contains this value for the layered bit of the window's extended style?

See thread #377423. The values you want are in winuser.h. Check the MSDN for SetLayeredWindowAttributes() for more information on using this function.