Guide: Texture information for aspiring modders

Prologue

My background lies in traditional and digital art. This guide is meant for people like me, who don't have a background in game- or 3D design and won't inherently understand how things tie in together in this particular field. In retrospect, had I known what information to look for before beginning, I'd have had a less difficult time understanding how textures work and how they could be manipulated.

The Mass Effect series was created with an custom version of the Unreal3 Engine. The engine much determines how textures work and while there is some nuance between games, the general setup is largely the same. As an example, I've also done some modding for "Remember Me" and was easily able to with knowledge acquired modding the ME series.
I've experienced the same with "Dragon Age: Inquisition", which was built on the Frostbite 3 engine. So, while this guide was initially written for ME(3) texture modding, I've noticed that for DA:I specifically, the principle carries over with the exception of specmap material functions seeming more consistent and normal maps having a different format. The most important thing to remember is that most textures won't make sense until you view each channel separately and to always mirror against the channel usage of the original texture.

I've sat on this tutorial for a while, having it almost completed and it's sister sitting in my ME3 modding thread, with the rise of DA:I (texture) modding I figured it a good time to give a final edit and release it.

Texture maps in ME3 and beyond

The following information might not bring anything new to some but it helps to start from the beginning so we can better understand what ME3 (or most any game for that matter) does with their textures and how things tie together. Working with textures for any game isn't so much about knowing exactly what they've done as it is about learning to recognize why and how they've done it. Lets take a look how Mass Effect works with texture maps and what you are likely to run into.

Remember that the way a texture functions is determined by the material and cannot be changed on the fly from the texture itself, as such it's important to look at the original closely when planning to edit a texture yourself.

Diffuse map
The diffuse is the most frequently used texture mapping method. It's a bitmap image wrapped directly onto the 3D mesh while displaying its original pixel color. This means that you can use any image as diffuse map to represent photo realistic quality. While some diffuse textures in Mass Effect will indeed hold colour information, others will be greyscale diffuse maps that get their colour information from a separate tintmap.

Function can be linked to all the different channels on a texture. Any single texture has up to 4 channels, R (red), G (green), B (blue) and, A (alpha). When you use the R/G/B channels together you can create a texture that holds it's own colour information. Most diffuse textures will also not have a functional alpha (or lack it entirely) and as such are most often a DXT1 without alpha to save on diskspace (file format). Those with alpha are DXT5. There are exceptions where colour is not put in the diffuse directly and the diffuse can be fit onto one channel rather than needing all three and that's where it can get confusing.

When creating a custom diffuse map I use both brushes and images for textures.
Good to remember is to avoid images that have lighting effect in them. While some highlighting and shading is done in the diffuse, most of it is controlled through the specmap and when you put it in the diffuse where it wasn't originally, it'll look all kinds of wrong.
On the flipside, if your diffuse does have lighting or shading in places put it back in! If it's in there, it's in there for a reason! I get this is hard for people working with a mouse but a lot of people don't do this and you end up with a flat diffuse that actually makes the whole look less realistic. This is also the reason I would suggest to anyone looking to seriously start texture modding to buy a drawing tablet. It doesn't have to be big, a bamboo is absolutely fine, it'll make this kind of work TONS easier. A big chunk of work I do on a diffuse will be manually painted on, having a tablet is a must for me.

If you are looking to introduce more detailing on a flat surface with little to no texture, you can use layer style screen or overlay to more easily add structure to it without having to recreate the entire surface. Be careful that you don't overuse this! There's really only a handful of places I've been able to use it where it worked, instead of making the surface look like a bleeding mess. It's not an auto-win button, but when it works, it saves time. Very situational.

Some tips:

• Use 3DS! Load in your mesh and your diffuse texture so you can see what you are doing without having to take your texture ingame.
• Group your materials together. Metal with metal, cloth with cloth. It'll make it easier to create the specmap later on.
• Use masks to add colour to elements. This makes it easier to create recolours for your retexture later.
• Put shading and highlighting on a separate layer and/or use masks for this as well.
• Shading and highlighting, put it back on the diffuse when you retexture!
• Learn how to use the pen tool, I can't express how important this tool is.
• Make sure your patterned areas are the same size on each element. Element wrap sizes differ so having them all the same size on the texture does not mean they are the same size on the model, quite the opposite.
• Take your time finding free stock (seamless) textures you can work with. You can make smart use of content aware fill in Photoshop (CS5+) to make a texture larger than it originally was. It's success depends on the textures structure, but be sure to look into it. The quality of a retexture halfly hangs on the quality of the stock used and smart texture selection.

Find common diffuse maps below to get an idea of what one might look like:

Specular map
You use specmaps to define the shininess and highlight colour of a surface. The higher the value of a pixel (from black to white), the shinier the surface will appear in-game. Therefore, surfaces such as dry stone or fabric would have a very dark specular map, while surfaces like metal or plastic would have lighter specular maps. 
The specular map is a peculiar thing in Mass Effect and has confused quite a lot of people in the past. Specmaps are inherently grey-scale, though in Mass Effect they won't look like they are.

While a diffuse needs the R/G/B channels you don't need those channels on a specmap. A DXT5 for example has 4 possible channels (R/G/B and alpha) and seeing the information held in a texture's channels is in essence just grey-scale information it's often so that these channels are independently used to map different functions. So, while Mass Effect and us modders call these textures the specmap, they are always a combination of several different maps spread over the channels of one texture which will include the specular map.
Putting all these different functions in channels of the same texture is essentially a filesize consideration. A texture with R/G/B and a functional alpha will be a DXT5 with interpolated alpha. Because this holds information for what would otherwise be 4 unique R/G/B channels (thus textures) and a DXT5 is about twice the size of a DXT1, they are saving half the diskspace (roughly put) while gaining flexibility.

Seeing a specular map is in essence grey-scale you'll sometimes see that the information on the R/G/B channels is identical resulting in a desaturated image. However, when you map things per channel the channels together make quite a colourful whole. Check out the specmap for this torso piece:

In this case the "typical" specmap most definitely sits on the green channel; as you can see it's a logical grey-scale with the fabric being slightly darker (less reflective) than the metal and carbon elements. The red and blue channels both completely black out certain elements which means these are both not a traditional specmap but are more likely to be glowmaps or maps that control sheen. The alpha channel clearly maps lights here.

While Mass Effect will often use the same colour channels for the same kind of functions, they're not completely consistent about their channels usage and that is really the hardest part of working with a specmap and creating a custom one. Finding out what function they've attached to each individual channel through the material itself and how to manipulate them to get the effect you desire. It's good to remember that, because any channel information is greyscale knowing that white is positive and black is negative, making a channel completely white will flood it's particular function across the surface the texture maps, alternately make the channel black to cancel out the effect it controls. It can be an effective way of finding out what any one channel effectively controls and how it makes elements look ingame.

Now the function attached to a specmaps channels isn't something you can manipulate via the texture itself, somewhere in the game (material) they hardcoded which channel maps which effect. This means that when you replace a mesh with another you might have to redo the specmap as the channels might not be hooked up to map and control the same functions.
Here are some of the functions that you can encounter mapped to one of the specmap's channels:
Glow, (coloured) Sheen, (coloured) lights, colour overlay, glossiness, transparency, material overlay effects (like glass for visors), animation frames (for instance animated text overlay cutoff points).
Basically every channel has it's own 'setting' in the material that determines exactly what kind of overlay effect it's tracking, all to make different material types like metal/leather and cloth look as realistic as possible in the engine.

Now sometimes you'll run into diffuse textures that don't seem to have a specmap to go with them, an example of this are the Mass Effect facial textures. Let's look at one of them:

The game needs a specmap to process an object in the lighting-engine, here it's actually hiding in the diffuse alpha. When an object doesn't need a lot of functions mapped to it, the number of channels/textures needed naturally drops where sometimes the separate specmap isn't necessary anymore. Most specmap textures will be DXT5 including an alpha, very rarely you'll encounter a DXT1 spec without alpha or a DXT1 with a 1 bit alpha.

Normal map
Normal Maps provide 3D detailing to an object. They are generated from the original high poly count 3D model of the mesh in question. As such, it's useful when creating a new normal map to work off the base of the original. Usually I upscale the original, depending on the object about half the size of the diffuse/spec unless it maps a full-body surface in which case I sometimes end up going up to 4096 on the diffuse and normal map alike. I then use the diffuse and Nvidia's normal map filter for Photoshop to generate a normal map.
I only tend to replace the original normal map in places where I changed the textures to such an extend as it being unavoidable. All other times I will smooth out the jagged edges on the original and overlay detailing generated with the Nvidia normal map preserving the original normal map.
ME3's normal maps are V8U8's. I save my edited normal maps as a BMP in Photoshop, then use the Direct X texture tool to convert to V8U8 and generate MIPs. Using a different file format will wreak havoc on your texture seams ingame. If you are modding a different game you can assume the same applies for it's native normalmap format and replacing it with one in a different format.

Some tips:

• Using the Photoshop normalmap filter, dark elements are automatically indented and light elements extruded. If you are trying to generate a map for a metal surface with white scratches, on default settings the filter will extrude them instead of indent. Flip the Z axis if you want to reverse this. You can cut and paste bits together with the axis inverted or normal depending on what kind of material surface you are dealing with.
• You can run the filter several times on different hardness (5-60) settings, then overlay each layer on a percentage to get a nice and fluid normal map with more depth.
• If your normal map becomes too pitted and irregular due to the diffuse detailing, consider running a gaussian blur over the diffuse before you run the filter.

Find common normal maps below to get an idea of what one might look like:

I am adding a little note here in regards to DA:I normal maps as these have a different native format and will need to be edited differently. DA:I normals are in ATI2 (3Dc) format, it has two channels (X/Y instead of R/G/B). The Photoshop DDS plugin decompresses the X/Y blocks and stores them in the R/G channels (the wrong way around mind you). When the opened texture is flagged as a normal, R/G/B is reconstructed from the R/G channels. Unfortunately, this makes the normalmap look 'wrong' upon import into Photoshop.
Now I used to eyeball the changes necessary to convert the opened normal back to what it should be by manually switching the red and green channel and messing with the contrast for those channels. However, I've recently stumbled across this workflow by MediAsylum on DeviantArt which makes the process entirely foolproof. Go thank them for being awesome. Here's a step by step:

• Open your extracted normalmap in Photoshop.
• In the NVidia DDS Read Properties window select "Convert images to 32bits"
• Upon opening, go to "Image - Mode - 16 bits/Channel". 
• Select "Exposure and Gamma" from the dropdown and hit "OK".
• Go to "Image - Adjustments - Levels..." and change the center value from 1.00 to 0.45
• Go to "Image - Adjustments -  Channel Mixer" and switch the red and green channel info (select red in the dropdown, change red 100% to 0 and green 0% to 100, then do the opposite for the green channel).

Et voila. You can now edit your normals as you're used to. When you're happy with your changes, save it as a 3Dc through the DDS plugin.
For reference, below you'll see a normal as previewed in the tool, after export/opening in Photoshop, before flipping the channel info and after applying ALL changes as mentioned above. Note that an extracted normal opened in the Compressonator also has switched R/G channels (3rd image).

NB: If you previously got rid of the NVidia DDS Read Properties dialog popup (like I did), you can get it back by saving a random image as a DDS and pressing "Read Config..." in the save dialog window. This'll pop it back up and you can recheck the box to re-enable the dialog.

Tint map
Tintmaps are used to add colour to a surface with a greyscale diffuse. They function as a colour overlay on-top of the diffuse greyscale information (much like a colour overlay layer works in Photoshop) preserving all detail on the diffuse itself and making it possible to have multiple colours of an item without needing a custom diffuse for every variant.
In essence these are function maps, with information either positive (white) or negative (black).
Tintmap channels all mask function dictated by the material. As an example:

You can see how the R/G/B data together looks and how the channels separately look, if an alpha is present it would hold the same kind of black/white data as on the R/G/B, masking parts of the texture surface for material control. You can't manipulate the function each channel maps from the texture, this is determined by the material itself.

Stripe map 
Stripemaps are used to add pattern-colour to a 3D model with a greyscale diffuse and tintmap. They function as a colour overlay on-top of the diffuse greyscale information, much like the tintmap.