Marker-less Head Tracking with LPVISION
For our LPVIZ in-vehicle AR system we usually rely on outside-in tracking systems like the ART Smarttrack 3. These camera systems work very well and deliver almost optimum performance for the use-case, especially if they have an interference filter built-in to protect from the influence sunlight. At the same time they are relatively large and require some effort to install.
Therefore for quick and easy prototyping, we started exploring markerless head tracking as alternative to marker-based outside-in tracking. The LPVISION head pose tracker is the result: an advanced 6DOF system that uses a standard RGB camera and machine learning alorithms for accurately following the head pose of a user wearing an AR headset.
The diagram above shows the difference between marker-based (left) and marker-less (right) head tracking setups. Note that for the markerless tracking setup only a single camera and no markers attached to the headset are required.
Key Advantages
Markerless operation via facial landmark detection: The system utilizes AI-powered facial landmark detection and other computer vision techniques to track head orientation and position continuously. Because of this, there are no rigid bodies, no reflective markers, and no physical contact with tracking gear required. Ultimately, this creates a non-intrusive experience.
Single RGB camera replaces multi-camera setups: We have replaced multi-camera IR setups with a single, compact RGB camera – such as the Stereolabs ZED camera line (note that even though the ZED camera is a stereo camera we use only one of the the built-in cameras). Therefore, this dramatically reduces hardware cost and physical setup complexity, while maintaining the precision required for industrial and spatial computing applications.
Zero-contact setup and instant deployment: You can go from zero to tracking instantly. In fact, there is no room-scale calibration, no physical mounting of tracking gear on the user, and no warm-up procedure needed. Consequently, the natural user experience means deployment in the field is as simple as launching the software.
Optimal for vehicle-in-the-loop and cockpit environments: Unlike stationary IR camera arrays, this tracker is engineered for dynamic environments. Particularly, it is optimal for vehicle-in-the-loop (ViL) testing and flight cockpit applications – scenarios where traditional IR-based systems are difficult to setup due to limited space and dynamic lighting conditions.
Integration with the LPVIZ In-car AR Solution
The LPVISION head tracker is designed to act as a drop-in replacement for complex IR-based tracking setups. For users of our LPVIZ platform it is a light-weight, easy-to-install alternative to our regular proven Smarttrack 3 setup. The video below shows a test drive with the system. The tracking camera itself isn’t visible in the video. It shows the view of the camera including the detected coordinate frame on the left. The window on the right shows a simple demo scene that’s being displayed to the user wearing the headset. As the car moves foward the viewpoint in the scene moves in the same way. Direction and offset to the location of the car is controlled by the output of the markerless head tracking.
Conclusion
We successfully implemented a markerless head tracker that works with users wearing an augmented reality HMD like the Xreal Air 2 Ultra. While the solution is easy to set up and use, we were not able to reach the levels of accuracy achieved by marker-based outside-in tracking. Unsolved problems like detection failures when more than one person is in the field of view of the tracking camera remain.
In conclusion, the LPVISION head tracker system is great for prototyping and quick installations, but for applications where accuracy and reliability is essential we found that a traditional outside-in system like an ART Smarttrack 3 is the preferred solution.
We will add a quantitative comparison chart between LPVISION markerless head tracking and marker based ART tracking soon!
