README.md

ThreeDWorld (TDW)

A Platform for Interactive Multi-Modal Physical Simulation

ThreeDWorld (TDW) is a platform for interactive multi-modal physical simulation. With TDW, users can simulate high-fidelity sensory data and physical interactions between mobile agents and objects in a wide variety of rich 3D environments.

Getting Started (Read this first!)

Code of Conduct

Changelog

Website

C# Code

License

How to upgrade from TDW v1.7 to v1.8

API

Commands and Output Data

Document	Description
Command API	API for every command a controller can send to the build.
Command API Guide	Overview of how to send commands to the build.
Output Data	API for all output data a controller can receive from the build.

High-Level APIs

API	Description
Image dataset	Generate 1.3M photorealistic images.
tdw_physics	Generate physics datasets.
Magnebot	High-level Magnebot robotics API.
Transport Challenge	A higher-level API that uses the Magnebot API.

Python tdw module

Frontend

Document	Description
tdw module	Overview of the Python tdw module.
Controller	Base class for all controllers.
TDWUtils	Utility class.
AssetBundleCreator	Covert 3D models into TDW-compatible asset bundles.
PyImpact	Generate impact sounds at runtime.
DebugController	Child class of Controller that has useful debug features.
KeyboardController	Child class of Controller that can listen for keyboard input.
FloorplanController	Child class of Controller that creates an interior environment and populates it with objects.
Librarian	"Librarians" hold asset bundle metadata records.
FluidTypes	Access different NVIDIA Flex fluid types.
Object Init Data	Wrapper classes for storing object initialization data.

Backend

Document	Description
Build	Helper functions for downloading the build.
PyPi	Helper functions for checking the version of the tdw module on PyPi.

Audio and Video

Document	Description
Impact Sounds	Generate impact sounds at runtime using PyImpact.
PyImpact	PyImpact API.
Audio/Video Recording	Record audio, video, or audio+video.
Remote rendering	How to render using xpra.

Avatars (Agents)

Document	Description
Observation Data	Different means of determining what an agent is observing.
Depth Maps	How to use depth maps.
Avatar Movement	Different techniques for moving agents.
Magnebot	High-level Magnebot API.

Benchmarks and Speed

Document	Description
Benchmarks	Performance benchmarks.
Performance Optimizations	Increase simulation speed.

Examples

Document	Description
Example Controllers	Index of all example controllers in the repo.
Use Cases (see below)	The "use cases" section showcases "advanced" usage of TDW.

Misc.

Document	Description
VR	VR in TDW.
Humanoids	Add "humanoids" and play animations.

Physics (PhysX and Flex)

Document	Description
Physics	Common physics problems and solutions.
NVIDIA Flex	Add soft bodies, cloth, and fluids to TDW.
FluidTypes	Access different NVIDIA Flex fluid types.
Physics Determinism	Benchmark of PhysX physics determinism.
tdw_physics	Generate a physics dataset.
Rube Goldberg (demo)	Demo of complex physical interactions between objects, with PyImpact generation of impact sounds, set in a photorealistic scene.

Releases

Document	Description
C# code	Access to C# backend source code
Releases	Release versioning in TDW.
Freezing your code	"Freeze" your controller into a compiled executable.
v1.6 to v1.7	How to upgrade from TDW v1.6 to TDW v1.7

Remote Server

Document	Description
Docker	Create a Docker container for TDW.
Remote rendering	How to render using xpra.
BinaryManager	Manage multiple instances of TDW builds on a remote server.
bash scripts	Useful bash scripts for Linux.

Rendering and Photorealism

Document	Description
Asset Bundle Librarians	Overview of what asset bundles are, how to add objects, scenes, materials, HDRI skyboxes, and humanoids, and how to access each asset bundle's metadata.
Model Screenshotter	Generate images of every model in TDW.
Material Screenshotter	Generate images of every material in TDW.
Materials, textures, and colors	Defines materials, textures, and colors.
Depth of Field	Prevent blurry images and increase realism.
Depth Maps	How to use depth maps.
Remote rendering	How to render using xpra.
Observation Data	Different means of determining what an agent is observing.

Robotics

Document	Description
Robots	Overview of robotics and the Magnebot in TDW.
Robot Librarian	API for accessing robot metadata.
Magnebot	High-level Magnebot API.

Scene Setup

Document	Description
Scene Setup	Overview of how to set up a scene.
Asset Bundle Librarians	Overview of what asset bundles are, how to add objects, scenes, materials, HDRI skyboxes, and humanoids, and how to access each asset bundle's metadata.
Model Librarian	Overview of how to add objects and access metadata.
Rotation	Different means of rotating objects and agents in a scene.
Scene Reset	How to reset a scene.
FloorplanController	Child class of Controller that creates an interior environment and populates it with objects.

TDW and 3D Objects

Document	Description
Model Librarian	Overview of how to add objects and access metadata.
Model Screenshotter	Generate images of every model in TDW.
Non-free models	Access the TDW "full model library".
Local 3D models	Add your own objects to TDW.
ShapeNet models	Convert ShapeNET models into TDW objects.
Composite Objects Creating Composite Objects	Use and create "composite objects".
AssetBundleCreator	API for the AssetBundleCreator class (used to convert 3D models into TDW-compatible asset bundles).
Rotation	Different means of rotating objects and agents in a scene.

Troubleshooting TDW

Document	Description
Debug TDW	Several strategies for debugging errors in your code.
DebugController	API for the DebugController class.
Depth of Field	Prevent blurry images and increase realism.
Performance Optimizations	Increase simulation speed.
OS X	Common OS X problems and solutions.
Physics	Common physics problems and solutions.

Use Cases

Use Case	Description
Image dataset	Generate 1.3M photorealistic images.
IntPhys (demo)	Demo of how to simulate IntPhys in TDW.
Humanoid videos	Generate a dataset of humanoid animations.
tdw_sound20k	Generate an audio dataset.
tdw_physics	Generate a physics dataset.
Rube Goldberg (demo)	Demo of complex physical interactions between objects, with PyImpact generation of impact sounds, set in a photorealistic scene.
Magnebot	High-level Magnebot API.