Network-based VR with Live ANN Training for Interacting at the Nano-scale

Author: Greg Gutmann
Affiliation: Tokyo Institute of Technology

I demonstrated my network-based VR technology at the Tokyo Nano Tech conference on January 29th to 31st with the other members of the Artificial Molecular Muscle Project. The network VR mentioned is a continuation of my work from my last paper titled Predictive Simulation: Using Regression and Artificial Neural Networks to Negate Latency in Networked Interactive Virtual Reality. In that paper, I had begun the initial work testing deep leaning as a solution to user hand motion prediction in VR. Since then I have added the ability to train the artificial neural network (ANN) live as the user is interacting to fine-tune the prediction; for such things as individual user motion patterns and fluctuating latency.

The application of our network-based VR is to create a virtual environment where researchers can run interactive coarse-grain simulations, or brainstorm for new ideas while manipulating nano-scale structures with their hands in VR.

Grabbing DNA Oragami
Pulling Closer in VR
In VR with a stereoscopic view (3D) it is easy to see the depth of each particle creating a more impressive view
Photo of Greg Gutmann at
Nano Tech 2020 with 47,692 visitors


Gutmann, G., & Konagaya, A. (2020). Real-time Inferencing and Training of Artificial Neural Network for Adaptive Latency Negation in Distributed Virtual Environments2020 International Congress on Human-Computer Interaction, Optimization and Robotic Applications (HORA). doi:10.1109/hora49412.2020.9152833


More detailed information can be found in the links below. Google Translate may be needed.

Information in Japanese can be found here:

Nano Tech website:


Credit for the work on the Networked VR goes to:

Greg Gutmann: Live trained ANN for motion prediction, simulation system, VR, haptic integration, network protocol

Akihiko Konagaya: Organization of the NEDO project, suggestions/guidance during the research & development

Ryuzo Azuma: DNA origami structure

Fuji Xerox: Development of haptic hardware

Tokyo Institute of Technology: Research environment

NEDO: Artificial Molecular Muscle Project funding

Published: February 2nd 2020

Leave a Reply

Close Menu