Art-free Character Generation for Unity3D: Part 2/Integration with MakeHuman: What Didn’t Work

Update: Per request by the MakeHuman admins, I would like to communicate that using the basemesh and .target files in any proprietary software is a violation of the AGPL. This post was not meant to promote any kind of license-breaking. (It didn’t work to begin with!) For more details, please see Mr. Haquier’s comments below.

Original post below.

Documenting what didn’t work is as valuable as documenting what did work in my book, so I’d like to describe the many failed iterations of making this whole thing work.

My first idea for making this work was pretty simple. After sifting through the MakeHuman repository, I realized that every possible change in the character’s appearance was dictated by a “target” file, which was basically a list of vertices and the offsets it would take to make that particular change of appearance happen in “full strength”. That’s easy (although tedious!) math, so I thought I’d do the following:

1) Import the “base” MakeHuman model into my Unity project, using Mecanim to match up the rigging, get all the texture imports right, etc.
2) Use those .target files and do the math myself to change the mesh around the armature to change the mesh only in realtime.
3) To save a character, just save the stack of the targets applied to it. You could then load in the file easily later.

More or less, I was looking to replicate the existing MakeHuman interface, only inside of Unity. There were many challenges with this approach, and ultimately it did not end up working, although I maintain that my work with this still might come in handy later. And I learned a lot about the inner workings on Unity, so that’s valuable.

Problem 1: The Unity Import Pipeline. The way Unity imports in meshes is different from the way the meshes are actually stored. This is fairly well-documented, but it still can wreak havoc. If a vertex has multiple normals, the vertex is split into multiple co-located vertices with one normal a piece. The way the target files work is that they are lists of vertex numbers and offsets. Since a vertex in the file is split into multiple vertices in Unity, if you only move the vertex listed in the file, the rest of the co-located vertices don’t move with it, which is problematic. This can be solved by preprocessing the vertex list alongside the input file, using a lookup table to determine all the co-located vertices, and then when one vertex in the group moves, moving the rest of the vertices with it. The other problematic import setting is that Unity, by default, changes the order of the vertices for optimal performance, which would of course cause strange things to happen when you apply the targets. To make it not change those orderings, you have to uncheck “Optimize Mesh” on the import settings.

Problem 2: Changing Relationships Between Targets. The relationships between all the targets is difficult to derive and constantly changing. At first I just copied and pasted the entire “targets” folder from the MakeHuman project into my codebase and tried to use the folder hierarchy to form some kind of organization on that basis. The folders are fairly descriptive (armslegs, cheek, chin, etc.) , the filenames help (for example l for left, r for right),  and there are icons for most of the targets. However, to divide it up in to manageable chunks that make sense (as in the existing MakeHuman interface), you have to impose some kind of extra order. In the “0_modeling_0_modifiers” plugin, you can find that the authors have hard-coded in a hierarchy and the relationships between all the different targets. To further complicate things, some targets have a dual (increase/decrease, in/out) while some do not (roundness of face, for example).

Well, that should have been my first hint that this wasn’t a terribly good idea. But I persisted. I copied and pasted all that hierarchy data into a text file and wrote a parser to read in all that stuff and make sense of it. This actually worked for the majority of the targets. Then two problems arose.  The first problem came when I got to the “macro” targets, where the value of one target influences another. These were the primary targets I was interested in–things like age, gender, height, proportion, race. Obviously these things influence each other–the maximum height of a female will be shorter than the maximum height of a male; the age of an individual will affect the way gender differences show up, etc. These targets also influence shapekey changes, like visemes and facial expressions. So when I went to implement in the macro targets and figure out all those dependencies, I basically hit a brick wall. It appears that the dependencies are half hard-coded in, half filename derived. I punted on that for the moment and went to solve other problems. Then, the second, happy problem arose–MakeHuman is a constantly changing and growing codebase, and new targets arrive from time to time! For example, as I was working on this, a “pregnant woman” target appeared on a nightly update. It’d be nice to keep up to date with the targets as they’re added, and that’s simply impossible to do with all the copy-pasting, dependency-hard coding that I was trying to do.

Problem 3: Lack of a Rigged Base Model. There is no “base model” representation of the plain vanilla human that you get when you open up MakeHuman that’s rigged quite right. There’s a base.obj file with the skeleton representation, but Unity3D doesn’t recognize the skeleton of that obj for the Mecanim system, which is the core of what I’m trying to do. So what I wanted to do was to export out an equivalent, rigged “base.fbx” model. But when you open up Makehuman, there are already targets applied by default. I dug into the code to find where those  targets were applied, disabled them, and then imported the result in as a FBX. But when I applied the targets to that result, the result was not the same as in the MakeHuman codebase. I’m not sure why this is, and by this point, I was reconsidering my approach, so I didn’t take the time to find out.

Problem 4: Implementation of Clothes & Hair. Even if I were able to accommodate for all those problems with coding (which I did to a large degree), all of these target applications would work just fine for the body, but would not apply for any “add-ons” like hair, teeth, tongue, clothes, shoes, or eyes. All of those objects’ positions and locations are represented as offsets from the body mesh’s vertices. So with enough work you could sort that all out, but it seemed a shame to reimplement a lot of what MakeHuman had already implemented, only in Unity. And as soon as MakeHuman changed its file formatting or algorithms, my project would have to be remade.

Problem 5: Non-Dynamic Rigging. My assumption was that once the rigging was set up, I could change the mesh around the rigging however I wanted and the rigging would still stay valid. This is simply not the case (and is an assumption I made without much understanding of how rigging works). When I made vertex changes, especially big ones as would be needed to change the base mesh to a child, the rigging simply didn’t work as it should have. 

So, with all that considered, I decided that an alternate approach was better. I wanted to keep as much of the MakeHuman code intact as possible and rely on that implementation to make all the mesh and rigging changes, so that way I wouldn’t have to be constantly changing my project as MakeHuman changed, and I wouldn’t have to re-solve the difficult problems that already had solutions in the MakeHuman project. There are a couple disadvantages. Every created model will be saved in the StreamingAssets folder, but in this case disk space is not an issue. Additionally, there will be a slow-down since the MakeHuman application will have to be loaded and run every time a new character is needed, but for my particular application it’s not a big deal–all my characters can be generated before the character interaction actually starts.

My working approach is to modify the MakeHuman source minimally to enable command-line generation of models, then place those models in the StreamingAssets folder of the Unity project. This means that Unity needs to call MakeHuman to make the virtual character, then load in the virtual character that MakeHuman spits out, all at runtime. In the next few posts I’ll describe how I implemented those two parts.

2 thoughts on “Art-free Character Generation for Unity3D: Part 2/Integration with MakeHuman: What Didn’t Work

  1. The first approach (replicating the MH interface in Unity) is considered a violation of the (A)GPL license under which the MakeHuman application and its source are distributed.

    We consider the basemesh and target data to be part of our source code, which is licensed under the AGPL3 license. It is only the output models that you produce with MakeHuman that are CC0 licensed.
    Source: https://www.makehuman.org/doc/node/makehuman_mesh_license.html

    The scenario of writing a plugin mimicking the MH behaviour and using its mesh source files (.target and the basemesh) is essentially integrating (linking) GPL code in a closed source engine (Unity). This is a pure violation of the license terms.

    Our main concern is keeping MakeHuman free and available to everyone, so we will primarily oppose unscrupulous developers stealing MH’s source files and selling them under the form of closed source products (for example a Unity plugin on the Unity asset store).

    All the while we support MH’s use in research projects. If one were to attempt such project for Unity it would have to be released under the same AGPL3 license that applies to the rest of MakeHuman, and its source code would have to be contributed back to the community. The license will also require you to distribute such Unity plugin separate, together with the source code, as otherwise it would be a case of linking GPL code to proprietary software.

  2. You might also be interested to know that the developers of MORSE have similar goals as yours.
    https://bitbucket.org/skadge/makehuman/overview

    Perhaps this could result in a collaboration.
    We from MakeHuman support this project since it is in the spirit of open and free software. In time these changes might be accepted in the official release if all the hacks can be refactored out.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>