deepsense.ai and Volkswagen deliver breakthrough in autonomous car research
deepsense.ai and Volkswagen show that an autonomous vehicle trained entirely in simulation can drive in the real world. The team transferred its neural network into a real car, providing it with over 100 years’ driving experience before the real-world engine had ever started.
Researchers from deepsense.ai recently presented their work on training an autonomous vehicle in a simulated environment and then transferring the model into a real-world vehicle. The team presented their paper during a workshop at the 2019 Neural Information Processing System conference in Vancouver, BC.
“Moving the neural network policy from the simulator to reality was a breakthrough and the heart of the experiment. We have run multiple training sessions in the simulated environment, but even the most sophisticated simulator delivers different experiences than what a car encounters in the real world. This is known as the sim-to-real gap. Our algorithm has learned to bridge this gap,” explains Piotr Miłoś, deepsense.ai researcher and a professor at the Polish Academy of Sciences. „That we actually rode in a car controlled by a neural network proves that reinforcement learning-powered training is a promising direction for autonomous vehicle research overall.”
The experiment was conducted by a team of researchers including deepsense.ai’s Błażej Osiński, Adam Jakubowski, Piotr Miłoś, Paweł Zięcina and Christopher Galias, Volkswagen researcher Silviu Homoceanu and University of Warsaw professor Henryk Michalewski. The team not only trained a model that controls a car in a simulated environment but also executed test drives in a real car at Volkswagen’s testing facility. The car navigated through real streets and crossroads, performing all sorts of real-world driving maneuvers.
Over 100 years of driving experience
The technique the deepsense.ai-Volkswagen team used was innovative for two reasons. First, the neural network was trained in a simulated environment and later transferred to a real car. Second, the neural network was trained using a reinforcement learning paradigm.
Being able to use simulation offers crucial advantages. First, it is cheaper. In their combined experiments, the team generated some 100 years of simulated driving experience. Racking up so much experience in a real car isn’t feasible. Training in a simulator can also be done much faster. The agent can experience all manner of danger, from simple rain to deadly extreme weather, accidents and full-blown crashes, and learn to navigate or avoid them. Subjecting an actual driver to such hazards would be prohibitively complicated, time-consuming and ethically unacceptable. What’s more, extreme scenarios in the real world are relatively uncommon but can be brought on in a simulator at the snap of the fingers.
“It was exciting to see that our novel approach worked so well. By further exploring this path we can deliver more reliable and flexible models to control autonomous vehicles,” explains Błażej Osiński. “We tested our technique on cars, but it can be further explored for other applications.”
Shaping the behavior
Reinforcement learning allows a model to shape its own behavior by interacting with the environment and receiving rewards and penalties as it goes. The model’s goal is, ultimately, to maximize the rewards it receives while avoiding penalties. An autonomous car receives points for driving safely and complying with the traffic laws.
Reinforcement learning delivers a more flexible model that can adjust itself to a changing environment and react accordingly in new situations. Simulating every road condition that can occur is impossible, but shaping a set of guidelines like “avoid hitting objects” or “protect passenger from any and all harm” is not. Ensuring the model sticks to the guidelines makes it more reliable, including in less common situations it will eventually face on the road.
Technical details of the research are described in the team’s white paper, available on arxiv. At the 2019 Neural Information Processing Systems conference in Vancouver, BC, over ten thousand participants took part in sessions and workshops on artificial intelligence. The team presented its paper during the workshop Machine Learning for Autonomous Driving (https://ml4ad.github.io/) organized by researchers from UC Berkeley, Carnegie Mellon University, Columbia University, and Waymo.
