Sensor Fusion is the Heart of AV Perception
March 31, 2021—Think about all the different decisions you make, actively or passively, as you drive. Your brain takes in visual, tactile, and sound cues, all providing input for your brain to analyze and make a driving decision.
How quickly can you merge? Is it safe to make this turn on a crowded block? What lane do I need to be in for an exit? The individual data points that contribute to these decisions can be complex.
That's a difficult task for computers to replicate in autonomous vehicles. Moreover, it's not the goal of autonomous technology developers to just replicate human perception. The key to a successful autonomous vehicle environment is to make them better than human perception—flawless in decisionmaking.
These challenges were covered as part of a recent webinar hosted by Partners for Automated Vehicle Education. The goal, ultimately, is safe driving no matter what the road throws at a vehicle.
“You want to handle these harsh cases and be able to not only infuse redundancy but ensure that you’re building a superhuman driver that goes much beyond the capabilities of what we have today," said Felix Heide, chief technology officer and cofounder of Algolux, an AI software company for automotive solutions.
Perception and Sensor Fusion
Like the different sensory inputs that a human experiences while driving, full autonomy is going to rely on a suite of sensors that can provide redundancy, work in multiple conditions and take in different kinds of information.
That means visible light cameras that can see, as well as LiDAR units that provide range and target information, radar that can back up spatial sensing, thermal cameras that can see in fog or at night, and much more. Working together, the system makes up what is called "sensor fusion."
That's just the hardware side. The software side also needs to be incredibly advanced, able to take information from all those sensors and make sense of the data in a single environment. The whole system is a perception engine.
“The job of the perception engine is to take the various inputs from those sensors and to fuse them into an understanding of those surroundings," said Hod Finkelstein, chief technology officer of Sense Photonics, a technology company specializing in LiDAR.
Challenges and Breakthroughs
There are many challenges in development, in part because autonomous technology is so new. Chris Posch is the engineering director for automotive at FLIR Systems, which develops thermal imaging cameras. He says that as they were developing a perception engine for thermal cameras, they needed raw data to introduce the system to how different objects appear thermally. Such a dataset didn't exist years ago.
“In order to do that, you need to have a whole bunch of videos all around the world of various objects—bikes, cars. And then you need to annotate that,” Posch said.
That construction of basic information is a long process that's still happening for different kinds of sensors.
Of course, cost is going to be a major consideration, especially as technology gets closer to being deployed in the marketplace.
“The more sensors you add, in theory, you add more cost and complexity," Posch said. "But you should be adding more capability.”
More capabilities might mean more safety. Posch said that governments might have a role in regulating a certain benchmark that could affect the makeup of autonomous vehicle sensor and computing technology.
Finkelstein says that he sees the cost coming down without sacrificing performance, similar to what has happened in other technological products. LiDAR is currently one of the more pricey sensors, but a successful market debut will need to mean fewer tradeoffs between cost and performance.
“I think that we will be able to integrate LiDARs that provide a high resolution, offer both long range and short range, a wide field of view, active illumination so that they can offer equally high fidelity output in daylight and in nighttime,” he said.