Network-based VR with Live ANN Training for Interacting at the Nano-scale

Author: Greg Gutmann
Affiliation: Tokyo Institute of Technology

I recently demonstrated my network-based VR technology at the Tokyo Nano Tech conference on January 29th to 31st with the other members of the Artificial Molecular Muscle Project. The network VR mentioned is a continuation of my work from my last paper titled Predictive Simulation: Using Regression and Artificial Neural Networks to Negate Latency in Networked Interactive Virtual Reality. In that paper, I had begun the initial work testing deep leaning as a solution to user hand motion prediction in VR. Since then I have added the ability to train the artificial neural network (ANN) live as the user is interacting to fine-tune the prediction; for such things as individual user motion patterns and fluctuating latency.

The application of our network-based VR is to create a virtual environment where researchers can run interactive coarse-grain simulations, or brainstorm for new ideas while manipulating nano-scale structures with their hands in VR.

Grabbing DNA Oragami
Pulling Closer in VR
In VR with a stereoscopic view (3D) it is easy to see the depth of each particle creating a more impressive view
Photo of Greg Gutmann at
Nano Tech 2020 with 47,692 visitors


More detailed information can be found in the links below. Google Translate may be needed.

Information in Japanese can be found here:

Information in English: TBA

Nano Tech website:

Video: TBA

Credit for the work on the Networked VR goes to:

Greg Gutmann: Live trained ANN for motion prediction, simulation system, VR, haptic integration, network protocol

Akihiko Konagaya: Organization of the NEDO project, suggestions/guidance during the research & development

Ryuzo Azuma: DNA origami structure

Fuji Xerox: Development of haptic hardware

Tokyo Institute of Technology: Research environment

NEDO: Artificial Molecular Muscle Project funding

Published: February 2nd 2020

Leave a Reply

Close Menu