Cloud VR Co-creation Environment for Molecular Robots (NEDO, 2020–2025)

Simulation 4 min read

Contents

From late 2020 to March 2025 I worked on a NEDO project run by Molecular Robotics Research Institute (分子ロボット総合研究所), the company Prof. Akihiko Konagaya founded in 2020. NEDO, the New Energy and Industrial Technology Development Organization, is Japan’s national research and development agency under the Ministry of Economy, Trade and Industry, with a budget of about 505 billion yen in fiscal 2026 (NEDO profile). Its projects are chosen through public calls and must pass stage-gate reviews to keep their funding. This one passed its review in early 2022 and ran its full term.

The theme was a co-creation environment for designing molecular robots using AI and VR (AIとVRを活用した分子ロボット共創環境の研究開発), under NEDO’s program for next-generation AI that evolves with people (人と共に進化する次世代人工知能に関する技術開発事業). Kansai University (Kuzuya lab) and Hokkaido University, later Kyoto University (Kakugo lab), were the partners.

The goal was a shared VR space where researchers in different places can view and handle molecular models together. It built on the networked VR from the previous NEDO project. I developed the simulation, rendering, user interaction and networking code, and from 2023 I was the VR Co-creation Environment Research Team Leader.

Simulation in the cloud

  • 2020–2021: a CUDA simulation for the cloud. I wrote a cloud version of the simulation and had it running on a remote cloud server by April 2021.
  • February 2022: Sakura Cloud. I moved the simulation server to a GPU server on Sakura Cloud. By February 24 four VR users were connected to it at once.
  • TCP and UDP. I compared TCP and UDP bandwidth between a cloud server and a remote client. The results are Figure 5 of the paper below.

Multi-user VR

By December 2020 three users could share the same simulation, one in VR and two on desktops.

Three users in one shared simulation: one in VR (pink hands, stereo view) and two on desktops (yellow and orange hands)
Multi-user demo, December 2020. User one is in VR (top, stereo view); users two and three are on desktops.

The same code ran either locally or on a cloud server. In a January 2022 cloud test, two users shook hands and held a DNA model together in VR, and four users followed in February on Sakura Cloud. To hide the network delay, the environment used the real-time AI hand prediction from the earlier project, as described in the paper below.

A VR remote controller

For VR on a phone, I designed a controller that connects over Wi-Fi, so it can work with any device, and that also lets you tune simulation parameters live. I built the first versions with an ESP32 microcontroller and a 3D-printed case in 2021 and 2022, and an electronics company then built a prototype from the design.

We were also working toward having the controller track itself with low-cost sensors: an IMU, plus a time-of-flight (ToF) sensor measuring the distance to the user at different orientations.

Molecular models and DNA hybridization

A large part of the work was the simulation model itself. My simulation was a hybrid of MD-like methods and others that allowed larger scales at high simulation rates. Over the project I added atom-level forces such as Lennard-Jones, then hydrogen bonds, water and salt, with the goal of reproducing DNA hybridization. It was a major effort, but hybridization never fully worked. By the end we were working toward a more detailed, MD-like model that also included chemical bonding information.

Atom charges were precomputed with methods such as the electronegativity equalization method (EEM). The next step was to update the charges during the simulation. EEM finds an equilibrium charge distribution, and in our simulation repeatedly re-equalizing the charges settled the molecules into an artificial ground state that suppressed the interactions we wanted to study. We were working on an EEM-like method that allows non-equilibrium charge states, so those interactions can still occur.

The same model could also run exploratory demos, such as RNA condensing around SARS-CoV-2 nucleocapsid proteins, pulled together by their opposite charges.

An RNA strand winding between and onto three nucleocapsid proteins
Exploratory demo: RNA condensing around SARS-CoV-2 nucleocapsid proteins in a simplified model, shown at 80x speed in the video.

5G

We also tested the cloud VR over 5G, with the VR side on a 5G connection to the cloud server, and compared how it performed in different areas. In a smaller town, 5G in those early days performed about like 4G, with high latency. In more populated areas it was noticeably better, but still short of what the carriers advertised.

Paper

Konagaya, A., Gutmann, G., & Zhang, Y. (2022). Co-creation environment with cloud virtual reality and real-time artificial intelligence toward the design of molecular robots. Journal of Integrative Bioinformatics, 20(1). doi:10.1515/jib-2022-0017

Exhibitions and award

We showed the work at nano tech and BioJapan nearly every year of the project, mostly at NEDO’s booth, and at other events such as NEDO AI NEXT FORUM and iREX.

A visitor wearing a VR headset tries the VR co-creation environment at NEDO's booth while two others watch
The VR co-creation environment at NEDO’s booth, iREX 2023 (International Robot Exhibition), November 2023.

In September 2023 I presented the research group’s work for the company at Leave a Nest’s Bio-Tech Grand Prix 2023, where it won the Mitsui Chemicals Award (report).

Public records