Imaging through narrow band hydrogen alpha filters can bring remarkable enhancement to objects with singular or dominant emission in the hydrogen-alpha line. These objects are mostly red emission nebulas but can also be complex nebulas with a dominant hydrogen-alpha emission component. Many imagers are content with monochrome display of these images but some may desire to go further and create a color image using the hydrogen alpha data. I no longer believe that luminance layering (LRGB) is the appropriate technique for h-alpha color composites. Instead I believe the best way to incorporate h-alpha data is by including the h-alpha directly into the RGB color data. Following these principles it is possible to create dramatically high contrast and detailed images without sacrificing richness of color.
There are several obstacles one must surmount to create quality color images using h-alpha data. One problem concerns the general suppression of stars in the h-alpha component and subsequent mismatching with stars in the RGB (color) data. A natural appearing image requires that stars match up in size, distribution, and color for both the luminance and color channels. The most pleasing results occur when star colors are accurately displayed. The second problem is the mismatch between the h-alpha and RGB components when using h-alpha as luminance. The H-alpha data is always deep and high in contrast which results in muted colors when matched with the lower contrast RGB data. Increasing the saturation usually does not help and often results in unnatural "peach" or "salmon" hues. This is the reason I no longer use LRGB technique as my preferred method for incorporating h-alpha data. From experience I've found the best results occur when a new red channel is created by using a blend of h-alpha with the traditional red channel. There are two primary benefits that occur from making this blend. The traditional red channel will bring more natural stars to the h-alpha data. The h-alpha data dramatically enhances nebula depth and detail but will be offset by the red data so a good color balance can achieved when making the RGB composite.
There are several creative solutions which can help the imager to incorporate H-alpha data successfully to create an aesthetically pleasing hydrogen alpha color composite. Our goal should be an image having the superior depth and contrast provided by the h-alpha but without sacrificing color richness and saturation. An outline of two separate photoshop techniques I have found useful are as follows:
1) (See example below) This technique is best for pure hydrogen alpha emitting nebulas. The H-alpha data and red channel data are blended in Photoshop using either "normal" mode or "lighten" mode. The red channel is the top layer. At this point I experiment with different opacities and blending modes until I reach my endpoint which is "normal" appearing stars and good nebula density and contrast. I have found that the percentage of Ha/Red can vary from 80/20 to 50/50 but each image is different. The numbers are arbitrary and should be adjusted to achieve a pleasing result. Once the layers are flattened this H-alpha-Red blend is then used as the "new red channel" to create a new RGB. I then bring this new red channel into Maxim along with the green and blue channels to create my RGB composite. As expected the h-alpha will boost the contribution of the red channel in relation to the green and blue. The traditional calculated RGB ratios no longer will be applicable. I usually start at a ratio of 1:1:1 and then repeat the process, each time gently tweaking the RGB ratios until I reach my endpoint which is rich nebula color and accurate star colors. I judge star colors to be close to accurate when the white stars show a "pure" white, there is a normal distribution of yellow and blue stars and the background is close to neutral. This "eyeball" technique of tweaking color balance might upset the "color purists" but most experienced imagers can easily tell when fairly good color balance has been reached. If it serves to help the image the h-alpha-red blend can then be reused again as luminance. At this stage the luminance should be used at opacities much less than 100%, usually anywhere from 10% to at most 50%.
2) (not illustrated) This technique may be best for complex nebulas having a dominant h-alpha component in addition to emission at other wavelengths. Start by using the H-alpha data as only partial luminance (opacity < 50%). The colors will not be muted if the h-alpha data is incorporated at less than 50% opacity. The downside is that much of the depth of the H-alpha component will be sacrificed if the luminance is not used close to 100%. A solution is to use the partial H-alpha-RGB as a new "superior" RGB for improved matching with the H-alpha luminance again. This new deeper RGB is now a better match for the high contrast h-alpha data. This process can be repeated several times, each time using the H-alpha data at a higher percentage luminance until you are satisfied with the results .
Some objects will work better with technique #1 (pure h-alpha emitters) and some with #2 (complex emitters). Some will benefit from a combination of both. It may take some time and experimentation to get a satisfying result. I have used the above techniques in photoshop but they may be applicable to any program which allows partial luminance layering and blending functions. These techniques have allowed me to produce images that display the deepness of the h-alpha component without compromising the aesthetics of rich and well saturated color.