Low poly hair is the bane of most game artists existence. Rarely is it done well, and the majority of the time, a bald head, or a slicked back look is taken as the easy way out. For three months I was in charge of creating the various hairstyles for Dark Age of Camelots: Catacombs expansion. Making the characters bald was not an option, so I set out trying to figure the best way to create low polygon count, heavily textured hair. Examples of my results can be seen here next to this text. The PSD of the final hair texture created in this tutorial is here.
First you need to set up your arsenal, your tool set. I used a few custom brushes for this. Credit for the idea, goes to Adobe with their hair tutorial. To create a brush, make a new canvas roughly the size of the brush you want. Create a black and white image, and go to Edit> Define Brush. The black areas will be the brush bristles, and the White areas will be transparent. You will also need to lower the spacing, and have your tablet pressure control the opacity of the hair. It works best to have it stay at a constant size, so do not let pressure control size. Here are my brushes (note that the last one is just a one pixel brush set to hard, no need to create that). Each is being displayed four times their actual resolution. You can actually drag these into Photoshop and just click edit>define brush, or you can download my set here. I encourage you to make your own and play around with the settings.
The first brush is for soft hair like fill. It works at different sizes, and it is fairly soft. The second is for adding a bit of sharpness and definition over top of the first. It should be used in quick strokes. The second and third are double hairs, used either horizontally or vertically, and can come in handy when the three hair brush is too much. The final is a single pixel brush set to hard, and is for precision placement when things just need a little touching up.
I contemplated doing an entire head, but each approach is specific to the hairstyle you are wanting to do. I can not do all the leg work for you, and figuring new ways to use old techniques is one of a Game artists greatest abilities. So I will be only showing you a small example of creating hair, but hopefully you will be able to extrapolate out how to tackle your specific problem.
Reference is KEY!
Always find reference of the color and rough style you want. then make a few rough drawings of the style you are going after. The less seams you can keep while creating your hair, the better. Its all about the quick uninterupted strokes.
First thing I do is pick a nice base color, normally this is fairly dark, as I tend to lighten things as I go.
Using the soft hair brush, I rough out a general flow of the hair. Lighter at the top, and darker at the bottom. Keep the root area (top) darker as well.
Now I try to pick out a few shapes to make it not just straight hair. Clump it together in a few spots, and make it deep in a few others. Every so often, I change the brush from normal mode, to soft light mode, in order to get some color variation in.
I used the three pixel hard hairbrush to go in and add some more fine detail. Keep in mind, you can and should use these brushes in both brush mode and smudge mode. Set pressure to control strength, and pick these brush shapes for your smudge tool. The lower resolution your hair is, the more you will need to smudge. At this resolution, a single pixel is fine for a hairs width, but in a much smaller space, a single pixel will be too large and will make the hair look like yarn. So smudging will keep the hair-ness look there.
Make a selection over the areas that the highlights should go using the polygonal lasso tool. Hit ctrl+alt+D for the feather option, and pick a feather size appropriate for your texture. On mine I am going to use a value of 5.
I used the three pixel hard hairbrush with a much lighter color to go in and add the initial specular. By making a selection and feathering it, you can use very fast and very sharp strokes, with perfect fading in and out. Hair specular is unique. Hairs are translucent, so not only do they shine, but they transmit that shine up and down the shaft of the hair. There is normally a base specular, which is the absorbed light. That is what I painted in this step. This highlight tends to be saturated the color of the hair.
Make another selection, this time smaller, and put it about the two thirds of the way up of the previous highlight area. Feather this a smaller amount, I used a value of 1.5.
I highlighted within this area, then deselected, and cleaned up in some other areas, using the two pixel vertical hairbrush. I took some redder colors and blended the highlights into the lower parts. This is the primary specular, and it is actually the hair reflecting the light source. It tends to be cooler and closer to white.
No hair is complete without some scalp showing. I cant think of many hairstyles where you would not see at least some scalp. I like to always show the scalp at the crown of the head where all the hair grows from, as well as any split going down the center. This is merely a new layer, filled totally with skin color. I applied a mask, and painted on the mask to reveal the skin. This is important because of the next step, to have it on its own later, and a mask revealing it.
Add a layer effect to this layer. Inner shadow, change the color to a dark skin color like a deep orange red. Keep the inner shadow set to multiply, and decrease the distance to zero, and the size to 1 or 2. This makes it look like the hairs are shadowing the scalp, and you can paint on your actual layer to show the roundess of the scalp, because all the hair shadowing comes from the layer effects.
This is a PIMM original technique. It just came to me while thinking about what makes most painted hair so fake. Real hair breaks, and the shorter pieces will tend to stick out. To simulate this, grab the three pixel brush or the one pixel, and change the opacity controls from pressure, to "fade" play with the number of steps, I use between 15-45 depending on the size of the texture. This will force the stroke to start at full opacity and fade to zero over that many steps. Now brush into the direction of the hair. This simulates the broken edes of the hair. I normally do this on a new layer, and then play with the opacity of the layer before flattening. Use both light and dark strokes, and it will help make the hair look more realistic.
Create a new layer set to softlight mode. Using your soft hairbrush, and a dark and light color (pick them appropriate to your hair style color) go in and reinforce the depth of those clumps you made earlier. This will help the hair read from further away in the game. The dark bits will make the hair seem like it has more of a sheen to it.
Now for tricks on making your 3D hair better. The hair looks good as is, but its kind of flat and lifeless, lets fix that. This is assuming you only have 1bit alpha, but it will work just as well if you have 8bit alpha. In fact it will look better if you do.
Here is my alpha mask for the hair. When you are painting in the alpha channel, hit "~" to get the color image overlayed on the alpha. Hit it again to go back to seeing the alpha channel only.
You can see the edge of the hair looks much nicer with the alpha map. Now comes the fun part, making the hair more volumetric.
Shift click these faces, and select "clone to element". Now convert your face selection to vertices, by ctrl clicking on the vertices element on the right control panel (this is all max specific).
Now convert your face selection to vertices, by ctrl clicking on the vertices element on the right control panel. (this is all max specific) Now subtract the top three verts from the selection, so that you can move just the bottom six. this will make it look like the hair all stems from the same place. This makes the hair look thicker because as the model moves, you will see the inner plane move behind the outer plane. This parralax will make the hair seem much more voluminous than it is, and helps to kill the plane illusion.
I did this again at the tip, with the bottom two planes. What can enhance the illusion even more, is to darken the under planes using vertex lighting if you have that available. Or you can flip the uvs so that you cannot see the mirroring of the texture from one layer down to the next. be sure to ofset them both in length, as well as slide them a bit to one side or the other, for a more random hair look.
I added one more flip outwards with a single plane. Here you can see what it will look like in a game with the wireframe turned off.
Create the initial volume quickly with the soft brush. It is very forgiving, and will give a rough hairlike look right away. Do not move past this stage until the hair reads as a volume from far away. Move on to picking out individual clumps next, so that the hair reads as fibers at any distance. Next use the sharper brushes to detail the hair a bit so that it looks nice from up close. Use soft light layers to enhance the sheen look, and deepen the volume whenever you need to. Use the smudge brush with these hairbrushes to blend. Add random tips if your resolution allows it. Use multiple planes to enhance the volume of the hair. Use alpha maps when possible to soften the edges of the hair. Use Lasso selections with a bit of feathering for highlights. Have fun, and explore new techniques!