Skip to Main Content
HBS Home
  • About
  • Academic Programs
  • Alumni
  • Faculty & Research
  • Baker Library
  • Giving
  • Harvard Business Review
  • Initiatives
  • News
  • Recruit
  • Map / Directions
Faculty & Research
  • Faculty
  • Research
  • Featured Topics
  • Academic Units
  • …→
  • Harvard Business School→
  • Faculty & Research→
Publications
Publications
  • 2021
  • Article
  • Conference on Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track

ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation

By: Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Michael Lingelbach, Aidan Curtis, Kevin Feigelis, Daniel M. Bear, Dan Gutfreund, David Cox, Antonio Torralba, James J. DiCarlo, Joshua B. Tenenbaum, Josh H. McDermott and Daniel L.K. Yamins
  • Format:Print
ShareBar

Abstract

We introduce ThreeDWorld (TDW), a platform for interactive multi-modal physical simulation. TDW enables simulation of high-fidelity sensory data and physical interactions between mobile agents and objects in rich 3D environments. Unique properties include: real-time near-photo-realistic image rendering; a library of objects and environments, and routines for their customization; generative procedures for efficiently building classes of new environments; high-fidelity audio rendering; realistic physical interactions for a variety of material types, including cloths, liquid, and deformable objects; customizable “avatars” that embody AI agents; and support for human interactions with VR devices. TDW’s API enables multiple agents to interact within a simulation and returns a range of sensor and physics data representing the state of the world. We present initial experiments enabled by TDW in emerging research directions in computer vision, machine learning, and cognitive science, including multi-modal physical scene understanding, physical dynamics predictions, multi-agent interactions, models that ‘learn like a child’, and attention studies in humans and neural networks.

Keywords

Artificial Intelligence; Platform; Interactive Physical Simulation; Virtual Environment; Multi-modal; AI and Machine Learning

Citation

Gan, Chuang, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Michael Lingelbach, Aidan Curtis, Kevin Feigelis, Daniel M. Bear, Dan Gutfreund, David Cox, Antonio Torralba, James J. DiCarlo, Joshua B. Tenenbaum, Josh H. McDermott, and Daniel L.K. Yamins. "ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation." Conference on Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track 35th (2021).
  • Read Now

About The Author

Julian De Freitas

Marketing
→More Publications

More from the Authors

    • 2022
    • Faculty Research

    Moral Thin-Slicing: How Snap Judgments Affect Online Sharing of Moral Content

    By: Julian De Freitas and Alon Hafri
    • July 2022
    • Faculty Research

    Hometown Foods

    By: Julian De Freitas, Jeremy Yang and Das Narayandas
    • 2022
    • Faculty Research

    Moral Deteriorations Sever Firm Identity

    By: Julian De Freitas, Zarema Khon, Pechthida Kim and Samuel G.B. Johnson
More from the Authors
  • Moral Thin-Slicing: How Snap Judgments Affect Online Sharing of Moral Content By: Julian De Freitas and Alon Hafri
  • Hometown Foods By: Julian De Freitas, Jeremy Yang and Das Narayandas
  • Moral Deteriorations Sever Firm Identity By: Julian De Freitas, Zarema Khon, Pechthida Kim and Samuel G.B. Johnson
ǁ
Campus Map
Harvard Business School
Soldiers Field
Boston, MA 02163
→Map & Directions
→More Contact Information
  • Make a Gift
  • Site Map
  • Jobs
  • Harvard University
  • Trademarks
  • Policies
  • Accessibility
  • Digital Accessibility
Copyright © President & Fellows of Harvard College