VR Simulation for NEDO's Molecular Artificial Muscle Project (2016–2020)
From 2016 to 2020 I worked on NEDO’s Molecular Artificial Muscle project (分子人工筋肉) in Prof. Akihiko Konagaya’s lab at Tokyo Institute of Technology. NEDO, the New Energy and Industrial Technology Development Organization, is Japan’s national research and development agency under the Ministry of Economy, Trade and Industry, with a budget of about 505 billion yen in fiscal 2026 (NEDO profile). Its projects are chosen through public calls and must pass stage-gate reviews to keep their funding. This one passed its review in 2018 and ran its full term.
The project was part of NEDO’s Next-Generation AI and Robot Core Technology program (次世代人工知能・ロボット中核技術開発). Hokkaido University worked on the motor proteins and Kansai University on the DNA origami. Besides managing the project, our lab built a VR simulation system for looking at and handling molecular structures in real time.
I designed and wrote the simulation, rendering, VR interaction and networking code.
What I built
- 2016–2018: multi-GPU simulation. The particle simulation grew from two GPUs to four, then to up to 32 GPUs on one node, with the workload scheduling computed on the GPUs. That version was never published.
- 2017: the DirectX 12 version. I started a new version built on DirectX 12 and CUDA, with the simulation and VR rendering in one program. It became the main system for the rest of the project.
- October 2017: NEDO exhibition. I ran the real-time VR simulation on a four-GPU workstation at our NEDO booth.
- 2017–2018: networked VR and molecules. I split the system into a VR client and a remote simulation server connected over a 10 Gbit network, and in 2018 added a spring model for representing molecules.
- 2018–2019: predictive simulation. Network latency makes a remote simulation feel laggy in VR, so I predicted the user’s hand motion and let the server compute ahead. See Predictive Simulation: arXiv 2019 and High-Performance Graphics 2019.
- 2019–2020: live-trained AI prediction. The hand prediction moved to an artificial neural network trained live while the user interacts. I demonstrated it at the NEDO booth at Nano Tech 2020 (post).
Arif Pramudwiatmoko added haptic feedback to the simulation using a device developed by Fuji Xerox, and Ryuzo Azuma provided the DNA origami models.
Besides the NEDO exhibitions above, we showed the work at nano tech and BioJapan nearly every year of the project.
Where it ended up
By February 2020, with a VR client and a simulation server:
| DNA origami model | 450,000 atoms and 49 million springs, about 90 simulation steps per second |
| Rendering | 160,000 particles on a desktop, 90,000 in VR |
| Network | 300 to 1,200 Mbit/s in normal VR use, up to 7 Gbit/s for large-scale viewing |
| Code I wrote | 22,570 lines for VR and local simulation, 14,025 lines for the server (C and C++, CUDA, DirectX 12, Leap Motion SDK) |
Papers
- Gutmann, G., & Konagaya, A. (2019). Predictive Simulation: Using Regression and Artificial Neural Networks to Negate Latency in Networked Interactive Virtual Reality. arXiv:1910.04703.
- Gutmann, G., & Konagaya, A. (2020). Real-time Inferencing and Training of Artificial Neural Network for Adaptive Latency Negation in Distributed Virtual Environments. 2020 International Congress on Human-Computer Interaction, Optimization and Robotic Applications (HORA).
Public records
- Tokyo Tech, May 2019: 世界初！DNAオリガミを融合した分子人工筋肉を開発, the project’s artificial muscle result.
- Tokyo Tech, January 2020: ナノスケール構造体を手で操作できるネットワーク型VR環境を開発, the networked VR shown at Nano Tech 2020.
The work continued in a second NEDO project from 2020: Cloud VR Co-creation Environment for Molecular Robots.