What you need to know before starting
When the modeling part is done, it might feels a bit like being set back to square one. The following task list including unwrapping, baking and texturing is no less challenging than the modeling part. It’s now even more crucial to take care as we further develop the project, as even a great looking model can lose a lot of it vivacity when shapes aren’t properly represented by maps and textures.
In the first section of this part I’ll give you some tips on how to unwrap the model efficiently and effectively. In the second section we will proceed with baking different maps in Marmoset Toolbag.
UV unwrapping with Headus UVLayout
The UV tools in Maya have been improved a lot in the latest versions, so there is no need to switch to third party tools anymore. I, out of habit, still prefer to use Headus UVLayout. The software is easy to use and delivers great results in no time. A trial version is available on the developer’s website.
It usually makes sense to divide the unwrapping task in two steps. The first step is all about quickly laying out UV shells, whereas the second step focuses on optimization. Throughout this task, it’s best to keep the following rules in mind:
- Generally speaking, the fewer UV seams there are, the better. If possible, hide them in places that will not be too visible on the final model.
- When dealing with hard-surface models, place the UV splits where you need to have hard edges. This reduces shading errors with tangent-space normal maps. In Maya, I use a script that later turns UV seams into hard edges.
- Look out for parts of the model that are symmetrical. Ideally, these shells can share the same UV space. We will optimize the mesh for that in the next step.
If repetition does not become too noticeable, symmetric geometry can share the same UV space. After quickly unwrapping the model in UVLayout, bring it back to Maya, select symmetrical faces and delete them. Next, duplicate their counterparts, mirror these faces along the corresponding axis, combine them with the initial mesh again and merge the vertices. This results in UV islands that perfectly overlay each other. The following clip illustrates the process in detail.
Keep in mind that when you mirror faces and triangulate the mesh for baking afterwards, you’ll have to make sure that the mirrored parts are triangulated the same way as their counterparts are!
Before you fit the shells into the 0 to 1 texture space, align the ones with straight edges to the X or Y axis to avoid a stair-stepping/aliasing effect on the borders. If possible, place elements that share the same textures/materials next to each other. This will allow you to find them a little bit easier in the 2D window when texture painting. Next, pack the shells as efficiently as possible into the UV tile to get the most out of the available UV space. Leave enough room between each island to apply decent pixel padding.
It also makes sense to scale up shells of small but distinctive elements of the model and scale down invisible or internal parts. In our case, we can decrease the size of the UV island that represents the inside of the barrel for instance.
Once everything is set up, I will run a little script that turns the borders of the UV shells into hard edges on the mesh. The code can be found here.
We are now only a few steps away from exporting both the highpoly and the lowpoly mesh for baking. To achieve a correct calculation of the normal maps, it’s necessary to make sure none of the elements are touching or intersecting each other. To ensure this, combine the parts in different groups as illustrated in the screenshot below. Do the same for the highpoly and the lowpoly version of your model.
Next, assign different materials to the highpoly parts that you want to colorize separately in the ID map. This map allows us to select and mask parts of the geometry easily in Substance Painter later on. It’s the same procedure we used to assign different vertex colors to the stock in ZBrush before, except that we are using different materials this time.
Last, but not least, triangulate the lowpoly mesh before you export it if you haven’t already done so. This prevents the game engine from triangulating the mesh itself upon import which can end up producing distorted textures. Now you are ready to export!
Baking maps and transferring details from highpoly to lowpoly can be done with different tools nowadays. The maps you can see in the screenshot below were created in Marmoset Toolbag. We need some of them to generate different textures in Substance Painter later on. Alternatively, you could also bake all maps directly in Substance Painter.
Marmoset Toolbag is not only a fantastic tool for presenting your models in a PBR environment, it has also become great for baking. It’s easy to set up, gives you fast previews of what the normal maps are going to look like before you actually bake them and allows you to make precise adjustments directly in the viewport.
First, bring the FBX files of the highpoly and lowpoly model to Marmoset. Then create a new Baker node and assign the parts accordingly. If you add the baker before you import the FBX files, you can use the Quick Loader to import and assign all elements automatically. To make this work, all parts of the lowpoly and highpoly mesh must be named accordingly on export.
( objectname_high/low_variation )
Before you begin tweaking the settings, set an output path in the Baker palette and make sure only Normals is checked in the Maps section. Hide the highpoly geometry and hit the little “P” icon on the top right of the baking window to compute a quick preview of the normal map.
In case you encounter any baking issues, you can now select the container of a lowpoly mesh and adjust the cage distance and skew values by using the brush tool directly in the viewport.
When you are happy with how the normals look in preview, it’s time to crank up the quality settings and add additional maps. For Ambient Occlusion, Ignore Groups should remain unchecked. If that’s NOT the case, Marmoset will include all compound parts when it calculates the AO. This is only useful when you’re dealing with solid objects that do not consist of movable components.
The GIF animation below illustrates the grouping issue that occurs when creating AO maps. The screenshot on the left shows the AO calculated when including mesh groups that we defined before we exported the model from Maya. Remember, no elements are touching each other, which allow us to create clean normal maps. The AO looks good, but it is not correct as the parts that belong together do not interact properly.
The image on the right shows what happens when Ignore Groups is checked. Now it correctly creates the AO and components that are close together are interacting with each other. But as soon as I isolate removable elements like the magazine for instance, you’ll notice that the AO turns black as no light is reaching the area inside the handle. Of course that’s not an issue as long as the magazine remains where it is. But if the game hero removes the magazine to refill the ammo, it’s definitely something you must avoid!
You can solve this problem by grouping the highpoly and lowpoly mesh differently and repeat the AO baking process for each element, or you can follow this quick workaround using only the lowpoly mesh for baking and a tool called Faogen.
The idea is to create a second AO map by quickly arranging the parts of the lowpoly mesh in a logical manner – everything that belongs to the magazine gets attached to the magazine, everything that belongs to the stock gets attached to the stock, etc. I then move the parts far enough away from each other, so they don’t affect another when the AO gets computed in Faogen.
Import the mesh in Faogen, set texture resolution and anti-aliasing correspondingly and watch the tool making its calculations. Faogen bakes AO directly into textures or vertex colors and it does that super fast by using your GPU power. When the process is complete, right click on the object names and save the AO as textures.
We can now combine the previously created base AO map from Marmoset Toolbag with the one from Faogen in Photoshop, set the layer blending mode of to Darken and the opacity to something like 60%.
This concludes part 2 of this tutorial. In PART 3 we will bring the model to Substance Painter, add additional height details to the gun, update the ambient occlusion map one last time and paint textures for the weapon.
Ben | 3D Gladiator