Marker-less Head Tracking with LPVISION

10/07/2026
Blog, Product, Projects

For our LPVIZ in-vehicle AR system we usually rely on outside-in tracking systems like the ART Smarttrack 3. These camera systems work very well and deliver almost optimum performance for the use-case, especially if they have an interference filter built-in to protect from the influence sunlight. At the same time they are relatively large and require some effort to install.

Therefore for quick and easy prototyping, we started exploring markerless head tracking as alternative to marker-based outside-in tracking. The LPVISION head pose tracker is the result: an advanced 6DOF system that uses a standard RGB camera and machine learning alorithms for accurately following the head pose of a user wearing an AR headset.

The diagram above shows the difference between marker-based (left) and marker-less (right) head tracking setups. Note that for the markerless tracking setup only a single camera and no markers attached to the headset are required.

Key Advantages

Markerless operation via facial landmark detection: The system utilizes AI-powered facial landmark detection and other computer vision techniques to track head orientation and position continuously. Because of this, there are no rigid bodies, no reflective markers, and no physical contact with tracking gear required. Ultimately, this creates a non-intrusive experience.

Single RGB camera replaces multi-camera setups: We have replaced multi-camera IR setups with a single, compact RGB camera – such as the Stereolabs ZED camera line (note that even though the ZED camera is a stereo camera we use only one of the the built-in cameras). Therefore, this dramatically reduces hardware cost and physical setup complexity, while maintaining the precision required for industrial and spatial computing applications.

Zero-contact setup and instant deployment: You can go from zero to tracking instantly. In fact, there is no room-scale calibration, no physical mounting of tracking gear on the user, and no warm-up procedure needed. Consequently, the natural user experience means deployment in the field is as simple as launching the software.

Optimal for vehicle-in-the-loop and cockpit environments: Unlike stationary IR camera arrays, this tracker is engineered for dynamic environments. Particularly, it is optimal for vehicle-in-the-loop (ViL) testing and flight cockpit applications – scenarios where traditional IR-based systems are difficult to setup due to limited space and dynamic lighting conditions.

Integration with the LPVIZ In-car AR Solution

The LPVISION head tracker is designed to act as a drop-in replacement for complex IR-based tracking setups. For users of our LPVIZ platform it is a light-weight, easy-to-install alternative to our regular proven Smarttrack 3 setup. The video below shows a test drive with the system. The tracking camera itself isn’t visible in the video. It shows the view of the camera including the detected coordinate frame on the left. The window on the right shows a simple demo scene that’s being displayed to the user wearing the headset. As the car moves foward the viewpoint in the scene moves in the same way. Direction and offset to the location of the car is controlled by the output of the markerless head tracking.

Conclusion

We successfully implemented a markerless head tracker that works with users wearing an augmented reality HMD like the Xreal Air 2 Ultra. While the solution is easy to set up and use, we were not able to reach the levels of accuracy achieved by marker-based outside-in tracking. Unsolved problems like detection failures when more than one person is in the field of view of the tracking camera remain.

In conclusion, the LPVISION head tracker system is great for prototyping and quick installations, but for applications where accuracy and reliability is essential we found that a traditional outside-in system like an ART Smarttrack 3 is the preferred solution.

We will add a quantitative comparison chart between LPVISION markerless head tracking and marker based ART tracking soon!

Wireless Mixed Reality with LPVR-AIR 3.3 and Meta Quest

16/02/2025
Blog, Product

Achieving Accurate Mixed Reality Overlays

In a previous blog post we’ve shown the difficulties of precisely aligning virtual and real content using the Varjo XR-3 mixed reality headset. In spite of the Varjo XR-3 being a high quality headset and accurate tracking using LPVR-CAD we had difficulties reaching correct alignment for different angles and distances from an object. We concluded that the relatively wide distance between video passthrough cameras and the displays of the HMD causes distortions that are hard to be corrected by the Varjo HMD’s software.

Consumer virtual reality headsets like the Meta Quest 3 have only recently become equipped with video passthrough cameras and displays that operate at similar image quality as the Varjo headsets. We have therefore started to extend our LPVR-AIR wireless VR software with mixed reality capabilities. This allows us to create similar augmented reality scenarios with the Quest 3 as with the Varjo XR series HMDs.

Full MR Solution with LPVR-AIR and Meta Quest

The Quest 3 is using pancake optics that allow for a much closer distance between passthrough cameras and displays. Therefore the correction of the camera images the HMD has to apply to align virtual and real content accurately is reduced. We show this in the video above. We’re tracking the HMD using our LPVR-AIR sensor fusion and an ART Smarttrack 3 outside-in tracking system. Even though the tracking accuracy we can reach with the tracking camera placed relatively far away from the HMD is limited, we achieve a very good alignment between the virtual cube and real cardboard box, even with varying distances from the object.

This shows that using a consumer grade HMD like the Meta Quest 3 with a cost-efficient outside-in tracking solution a state-of-the-art mixed reality setup can be achieved. The fact that the Quest 3 is wirelessly connected to the rendering server adds to the ease-of-use of this solution.

The overlay accuracy of this solution is superior to all other solutions on the market that we’ve tried. Marker-based outside-in tracking guarantees long-term accuracy and repeatability, which is usually an inssue with inside-out or Lighthouse-based tracking. This functionality is supported from LPVR-AIR version 3.3.

Controller and Optical Marker Tracking

In addition to delivering high-quality mixed reality and precise wireless headset tracking, LPVR-AIR seamlessly integrates controllers tracked by the HMD’s inside-out system with objects tracked via optical targets in the outside-in tracking frame, all within a unified global frame. The video above shows this unique capability in action.

When combined with our LPVR-CAD software, LPVR-AIR enables the tracking of any number of rigid bodies within the outside-in tracking volume. This provides an intuitive solution for tracking objects such as vehicle doors, steering wheels, or other cockpit components. Outside-in optical markers are lightweight, cost-effective, and require no power supply. With camera-based outside-in tracking, all objects within the tracking volume remain continuously tracked, regardless of whether the user is looking at them. They can be positioned with millimeter accuracy and function reliably under any lighting conditions, from bright daylight to dark studio environments.

In-Car Head Tracking with LPVR-AIR

After confirming the capability of LPVR-AIR to work well with large room scale mixed reality setups, we started developing the system’s functionality to do accurate head tracking in a moving vehicle or on a simulator motion platform. For this purpose we ported the algorithm used by our LPVR-DUO solution to LPVR-AIR. With some adjustments we were able to reach a very similar performance to LPVR-DUO, this time with a wireless setup.

Whereas the video pass-through quality of the Quest and Varjo HMDs are comparable in day and night-time scenarios, the lightness and comfort of a wireless solution is a big advantage. Compatibility with all OpenVR or OpenXR compatible applications on the rendering server makes this solution a unique state-of-the art simulation and prototyping tool for autmotive and aerospace product development.

Release notes

See the release notes for LPVR-AIR 3.3 here.

LPVR New Release 4.9.2 – Varjo XR-4 Controller Integration and Key Improvements

12/11/2024
Blog, Product

New release LPVR-CAD and LPVR-DUO 4.9.2

As it is with software, our LPVR-CAD and LPVR-DUO products for high-fidelity VR and AR need maintenance updates. Keeping up-to-date with the wide range of supported hardware as well as fixing issues that are discovered necessitates a release every now and then. Our latest release, LPVR-CAD 4.9.2 and LPVR-DUO 4.9.2 is no different. This blog post summarizes the changes in the latest version, LPVR-CAD 4.9.2 and LPVR-DUO-4.9.2.

Support for Varjo XR-4 Controllers

The feature with the highest visibility is support for the hand controllers that Varjo ships with the Varjo XR-4 headset. These controllers are tracked by the headset itself, and Varjo Base 4.4 adds an opt-in way of supporting them with LPVR-CAD. Varjo does not enable the controllers by default because the increased USB traffic can negatively affect performance on some systems, and so an LPVR user has to decide whether the added support is worth it on their system. Of course, we also continue supporting the SteamVR controllers together with LPVR-CAD. We detailed their use with the XR-4 in our documentation.

To enable the Varjo controllers in LPVR-CAD, first open Varjo Base. Then navigate to the System tab in Varjo Base. When LPVR-CAD is configured you will find a new input field, depicted below.

Setting its value to “true” will enable controller support, and “false” will disable it. After changing the value, scroll down to the Submit button and click it to effect the change. Varjo also recommends restarting Base after making this change.

Please note that this input is handled by Varjo Base itself, and so this button will also appear in older versions of Varjo Base, for reasons that are too broad to go into here. Providing this support quickly had higher priority to Varjo and us than polish. One issue that can cause confusion is that the Varjo Home screen will not display the controllers, at least in Varjo Base 4.4.0. Unity applications will have to be updated to a recent version of the Varjo plugin. Varjo is working on improving these issues.

 

Updated Support for JVC HMD-VS1W

An interesting AR headset of see-through type is JVC’s HMD-VS1W. It is a niche product which is typically used in the aeronautical sector. This is a headset which uses Valve tracking with a few custom twists. With a recent software update on their side (version 1.5.0) compatibility with LPVR was broken, but it was easy enough to restore and we have recovered full compatibility.

 

Various other changes

One of the key points when creating an immersive VR and AR experience is that the motion should appear as smooth as possible. We are therefore constantly refining our algorithms to meet that goal. This release significantly improves the smoothness of rotations, especially for Varjo’s third-generation headsets such as the Varjo Aero and the Varjo XR-3.

We fixed a condition where under some circumstances LPVR-DUO would crash after calibrating the platform IMU. This was related to a multi-threading issue which caused a so-called deadlock in the driver.

We also added support for a global configuration of our SteamVR driver which can be overridden by local users. Since automatic support for this requires major changes to our installers and uninstallers, we decided to postpone enabling this feature by default. Please get in touch if that is something you want to use already.

We often recommended the so-called “freeGravity” feature to our users to improve visual performance in most circumstances. We changed the default for this setting to match the needs for the most common use cases.

 

*Important note for LPVR-CAD, LPVR-DUO users with Varjo headsets:

Customers who initially purchased LPVR-CAD, LPVR-DUO for the Varjo XR-3 and wish to upgrade to the XR-4 must purchase a separate license upgrade to ensure compatibility. Orders placed before 2024 only cover up to Varjo XR-3 HMD, and orders made from 2024 will cover up to Varjo XR-4 HMD.

We recommend reviewing your maintenance coverage and hardware plans before making upgrades or deploying LPVR across multiple locations. For questions, feel free to contact our support team.

FusionHub – Sensor Fusion Operating System

17/04/2024
Blog, Product

In the past years our team at LP-RESEARCH has worked on a number of customer projects that required us to customize our sensor fusion algorithms for a specific application.

Such customization usually doesn’t include significant changes to our existing filter algorithms, but in most cases only warrants changes to input/output interfacing, specialized extensions to the core functionality.

Our own products such as the LPMS inertial measurement units, the LPVR virtual and augmented reality tracking system series and LPVIZ are based to a large extent on the same fundamental algorithms, but with different sensor inputs and different data output interfaces.

For this reason we decided to develop a modular platform that would allow us to create a graph of nodes, each node with a specific functionality as either:

  • Sensor data source (Gyroscope source, GPS etc.)
  • Filter algorithm (IMU-optical filter, odometry-GPS filter etc.)
  • Output sinks (File writer, websocket output, VRPN output etc.)

Modular Platform

This modular platform we call the FusionHub or sensor fusion operating system (SFOS) as it creates an end-to-end solution for applications that need a high performance and flexible sensor input/output and filtering system. FusionHub is platform independent and can run on Windows, Linux and Android. We are also working on a port to Apple’s VisionOS.

One might argue that a very simlar solution exists in the form of the Robot Operating System (ROS). ROS is a powerful platform, but it is relatively large in the amount of base components that it needs to run.

With FusionHub, we rely on the ZeroMQ protocol and Google’s Protocol Buffers for the communication of our nodes. This keeps the application light weight and independent from any other software stack or platform.

Apart from proprietary external libraries that might be needed to connect to certain sensor systems, FusionHub is a standalone application. It easily runs even on embedded hardware systems such as the Meta Quest virtual reality display series.

Easy Configuration

The components of a FusionHub graph are configured using a simple JSON script. For example a sensor input from one of our LPMS-IG1 IMUs is configured as shown below:

"imu": {
	"outEndpoint": "inproc://imu_data_source",
	"settings": {
		"autodetectType": "ig1"
	},
	"type": "OpenZen"
}

An IMU-optical sensor fusion node would be configured similar to the below script:

"fusion": {
	"dataEndpoint": "inproc://fusion_output",
	"inputEndpoints": [
		"inproc://optical_data_source",
		"inproc://imu_data_source"
	],
	"settings": {
		"SensorFusion": {
			"alignment": {
				"w": 0.464394368797396,
				"x": -0.545272191516426,
				"y": 0.5236543865322006,
				"z": 0.46130487841960544
			},
			"orientationWeight": 0.005,
			"predictionInterval": 0.01,
			"sggPointsEachSide": 5,
			"sggPolynomialOrder": 5,
			"tiltCorrection": null,
			"yawWeight": 0.01
		}
	},
	"type": "ImuOpticalFusion"
}

Each node contains input and output endpoints, or one of the two for source and sink nodes, that allow other nodes to connect to them. Connections can be limited to inside the application (“inproc://..”) or also be accessible from outside the application (“tcp://..”). This allows for constructing node networks that run on distributed machines. For example one computer could acquire data from a specific sensor and another computer could apply a sensor fusion algorithm to the incoming data.

This is also a simple way how additional nodes developed by 3rd parties can communicate with or via the FusionHub.

We have so far created the follwing node components:

Inputs

  • LPMS input node
  • Optical tracking system input node (ART, Optitrack, VICON, Antilatency)
  • (RTK-)GPS input nodes (NMEA, RTCM)
  • Car odometry via CAN bus input node

Outputs

  • File writer node
  • VRPN output node
  • Websocket output node

Filters

  • IMU-optical fusion node (for LPVR systems)
  • GPS-odometry fusion node (for car navigation)
  • RTK-GPS-odoemtry-IMU fusion node (for car navigation)

Future Outlook

With great success we’ve deployed FusionHub for our own products as well as several customer projects.

While our collection of node components and the range of applications for FusionHub is growing, we spend significant development resources on creating testing frameworks that guarantee the performance and robustness of FusionHub. As the application of FusionHub in our LPVIZ driving assistance system is mission critical, reliability and redundancy of our core framework is essential.

In the future we’re looking to connect FusionHub with our internal IoT solution LPIOT to open up access to our core algorithm to large numbers of sensor devices in parallel. Furthermore we are working with machine learning-focused company Archetype AI to give deeper inteligence to this solution.

See further information about how to use FusionHub in the FusionHub documentation.

LPVR-AIR for Immersive Collaborative Industrial Design

19/03/2024
Blog, Product

Wireless Content Streaming

LPVR-AIR is LP-Research’s wireless VR streaming solution. Content is generated on a rendering computer and wirelessly streamed to a VR headset to be displayed. At the same time the pose, orientation and position, of the headset is calculated from tracking data from a camera system and inertial measurements on the headset itself.

The core tracking algorithm of LPVR-AIR is similar to our LPVR-CAD solution. We are combining this established tracking method with wireless data streaming.

This has a few significant advantages:

  • Rendering detailed VR content is computationally too heavy to do all calculations on embedded hardware on the headset itself. Therefore, content needs to be rendered on an external computer and the result is streamed to the headset. LPVR-AIR allows doing this.
  • Designers in eg. the automotive space have their own preference of applications to create content, such as Autodesk VRED. These applications usually don’t run on a headset’s embedded hardware. With LPVR-AIR dsigners can use any application that normally works with a Windows based PC.

Technical Implementation

See below a block diagram of how the LPVR-AIR system is implemented. While we in principle support any Android based standalone VR headset, we currently focus on the Meta Quest line of HMDs, specifically Meta Quest Pro and Meta Quest 3.

Our solution effectively enables designers to explore a large 3D design space with full high resolution renderings using a lightweight headset. LPVR-AIR even allows for the interaction of several users in a design space. An example of such a use case is shown in the video on the top of this post. Two users in our office in Tokyo, being tracked by LPVR tracking, explore a car design together.

Improved Design Process

This opens new possibilities for automotive, industrial, architecture and many more design applications, leading to increased performance of designers and a higher success rate of their designs. LPVR-AIR is based on the ALVR wireless streaming engine, which we have extended to work with our FusionHub sensor fusion solution.

Long term, the ALVR engine makes it easy for us to support a number of different HMDs, additionally to Meta Quest also the Varjo and eventually the Apple Vision Pro series as shown in the image below. With VRED we have an outstanding rendering solution at the base of LPVR-AIR that allows designers to create photo-realistic content while providing extensive collaboration abilities.

If you would like to move towards immersive and interactive 3D design, don’t hesitate to consult with us and give our LPVR-AIR on-premise collaborative design solution a try!

1 2 3 … 7