Generated image datasets don’t work well? Maybe you need to consider differences in content distribution

There are many achievements in exploring the differences between generated datasets and real datasets, such as learning common image features for different tasks, learning image style transfer, etc., which can make the images in the generated dataset look more like real images. However, the authors of this paper believe that the difference in image style is actually only a small factor. The more important difference lies in the difference in image content, and the generated images should be helpful for new tasks. Previous image generation methods can only cover limited scenes, limited objects, and limited changes, and the variability and attribute distribution of real-world objects are insufficient. Moreover, the authors proposed that, taking the KITTI dataset as an example, its data was collected in Germany, but perhaps other researchers use this dataset to train a system that they want to use in Japan, and the scene content must be different; even the task objectives of the service may be different. These are all aspects that the existing data generation methods have not solved or even considered. If it is completely replicated in a virtual environment, the financial and time costs are also very high.

Meta-Sim generates datasets that narrow the distribution between real and generated data and can be optimized for downstream tasks.

Therefore, in the paper "Meta-Sim: Learning to Generate Synthetic Datasets", the authors clearly stated that their research goal is to automatically generate large-scale annotated datasets, and this dataset is helpful for downstream tasks (the content distribution in the dataset can meet the target usage scenario). The method proposed by the authors is Meta-Sim, which will learn a generative model about the newly synthesized scene, and can simultaneously obtain the training graphics and the corresponding true label values ​​through a graphics engine. The authors then parameterized the dataset generator with a neural network so that it can learn to modify the properties of the scene structure graph obtained from the scene content distribution probability in order to reduce the difference between the image output by the image engine and the target dataset distribution. If the real dataset to be imitated comes with a small annotated validation set, the author's method can also optimize for an additional meta-objective, that is, it can optimize for the downstream tasks of the current dataset task. Experiments show that compared with the manually designed scene content distribution probability, their proposed method can greatly improve the quality of content generation, which can be qualitatively and quantitatively verified on downstream tasks. For more specific details, please refer to the original paper.

The paper’s authors are from Nvidia, the University of Toronto, the Vector Institute for Artificial Intelligence, and MIT.

For the project homepage, see: https://nv-tlabs.github.io/meta-sim/

Paper address:

https://arxiv.org/abs/1904.11621