New network architecture/model for this weeks build
No. It's not the gen2 model. The first and second gen models were intended as a stop-gap until we had a generative network/model that would allow us more control over the attributes of the image. After a week-long experiment with the new network architecture and training, we believe we have a model that works.
Background
First and second generation models were trained using a pix2pix technique. Pix2pix is a method of converting one image to another image. This can be used to convert pictures of horses to zebras or dogs to cats. In our case, we train images with human poses to convert to 3dx-style images. This technique can produce good results, but the pose images lack any context information such as location, hair-color, sex/gender, etc. The model will infer those attributes almost randomly. This is why the male character in current Catching Heat versions (version 0.0.4 and below) often looks like a female.
Solution
The new network architecture adds onto the previous. It uses a technique used by ManiGAN and RefinedGAN to combine text attributes (ie "blonde_hair") with image data (ie a human pose) called "Affine Combination Module" or ACM. For over a month we've been doing experiments with this technique trying to see if we can control the attributes of the generated image by adjusting the tags. Our first several models failed at this. Further, quality was poor. Our experiments last-week however and a new architecture showed both improved control and quality over previous efforts. Consider the following using the old ACM architecture:
The two images are essentially identical, though they have different tags. The first image has the tag "black_hair" while the one on the bottom has "blonde_hair". With the new architecture, not only is quality better, but we can control hair color, clothing and more:
tags: 1girl, solo, black_hair, long_hair, dress, jewelry, standing, blue_eyes, sleeveless, tanktop, heels
tags: 1girl, solo, blonde_hair, long_hair, jewelry, standing, blue_eyes, heels, denim, sleeveless, bare_shoulders
We ran these experiments using the deepfashion dataset with imperfect machine-created tags. Even with the inconsistent tags, the results are impressive.
Further Improvement
Once we started having good results, we began training two production models we will start using this week. The first is a smaller model that generates 256x256 pixel images upscaled to 1920x1080. The final quality of the game images won't be great, but this model will train fast.
The second model (we are simply calling "massive" for now) is one that takes over 20GB of GPU memory and is about 5 times slower to train. The benefit is this model produces 512x512 images and should be higher quality. Long-term, this is the model we want to use.
In this post are (extremely early) renders of some of the in-game images. Included is red-headed Sky (did you know Sky was a red-head?) and Elena with both blonde and black hair. The quality is still very poor and we should see some improvement before the build on Friday.
The dataset used to train both new models also have machine-generated tags like the deepfashion dataset. We have humans working on manually tagging a smaller dataset (only thousands of images vs tens of thousands). Once we have enough well-tagged images, we will start training on those and fine tune our models. This will allow more consistent control over more image attributes including background attributes and scene locations.
Why are we releasing builds with reduced image quality?
Because the model previously used (gen1) was always intended as a stop-gap and now that we know that our ACM models work, we are putting almost all of our resources in training those models and improving their quality. We have a some resources available we could continue to train those older models, but we are choosing instead to put those resources towards other experiments and essential work.