How VR and AR Headset Tracking Works: Inside-Out, Outside-In and Sensor Fusion

Every virtual or augmented reality headset has to answer the same question hundreds of times per second: where is the user’s head, and which way is it pointing? The quality of that answer decides whether the virtual world feels solid or swims, whether a mixed reality overlay sits on the real object or floats beside it, and whether the user feels fine after an hour or sick after five minutes. This article explains the tracking methods used in today’s headsets, why professional installations combine them, and where each approach reaches its limits.

Degrees of freedom

A headset pose has six degrees of freedom (6DoF): three for position (x, y, z) and three for orientation (roll, pitch, yaw). Early mobile headsets tracked only orientation (3DoF) with an inertial measurement unit, which is enough to look around but not to lean, crouch or walk. Any application in which the user moves, or in which virtual content must line up with the real world, needs full 6DoF tracking.

The inertial measurement unit

An IMU measures angular rate and acceleration directly on the head. It is fast (hundreds to a thousand samples per second), has almost no latency and works in any lighting. Its weakness is drift: orientation integrated from a gyroscope wanders over time, and position integrated from an accelerometer becomes useless within seconds. Every headset contains an IMU, and every serious tracking system uses it as the high-speed backbone that some other sensor keeps honest.

Inside-out tracking

Consumer headsets such as the Meta Quest 3 carry cameras that watch the room. Software recognizes features in the images, builds a map of the space and locates the headset in it (visual-inertial SLAM). Inside-out tracking is self-contained and needs no installation, which is why it has won the consumer market. Its limits appear in professional use:

• It needs texture and steady light. Featureless walls, glossy surfaces, darkness and bright sunlight degrade it.
• Its coordinate system is private to each headset. Two users in the same room do not share a frame, and the map may shift when the headset is restarted.
• It assumes the environment is static. In a moving vehicle the cameras see a cabin that is stationary relative to the head while the IMU feels the vehicle accelerating, and the two disagree.
• Repeatability is limited to a few centimetres and a fraction of a degree, which is visible as soon as a virtual object must sit on a real one.

Outside-in optical tracking

Professional installations turn the arrangement around: cameras mounted in the room track reflective or active markers attached to the headset. Systems from ART, OptiTrack and VICON locate a marker constellation to sub-millimetre accuracy in volumes of 15 × 15 × 3 m and more, and every tracked object, headset, controller or tool shares one calibrated coordinate system. This is what makes multi-user installations, exact mixed reality overlays and repeatable measurements possible.

Outside-in optical tracking has its own weaknesses. Camera frame rates of 100 to 300 Hz and the image processing behind them add latency of tens of milliseconds, and orientation from a small marker constellation is noisier than its position. Markers can be occluded by the user’s hands or by other people. Used alone, optical tracking gives an accurate but slightly late and slightly jittery pose, which the user perceives as swimming.

Sensor fusion: the best of both

The answer is to fuse the two sources. The IMU delivers orientation and short-term motion at 1000 Hz with no delay; the optical system delivers absolute position and corrects the IMU’s drift whenever a marker measurement arrives. A head motion model predicts where the head will be at the moment the next frame is displayed, so that the rendered image matches the user’s actual pose instead of the pose from a few milliseconds earlier. This is how LPVR-CAD tracks: an LPMS IMU on the headset, markers for the optical system, and a fusion filter that outputs sub-millimetre position and 0.2° orientation at 1000 Hz through a standard SteamVR or OpenXR driver.

The same principle scales to wireless headsets. LPVR-AIR replaces the inside-out tracking of a Meta Quest 3 or similar standalone headset with the fused optical-inertial pose and streams the rendered image from the PC over Wi-Fi, so that several standalone headsets can operate in one large, shared tracking volume.

Latency, prediction and rendering

Tracking accuracy is only half the story. The delay from head motion to the matching pixel on the display, the motion-to-photon latency, should stay under about 20 ms to feel natural. Rendering pipelines reduce it with motion prediction, late latching (updating the pose as late as possible before rendering) and asynchronous timewarp (re-projecting the last frame to the newest pose). LPVR integrates these steps with its tracking so that the pose used for rendering is the predicted pose at display time.

Mixed reality raises the bar

In video passthrough mixed reality, virtual objects are drawn onto live camera images of the real world. Any tracking error becomes visible as an offset between the virtual and the real object, and any drift becomes visible as the object creeping. Our tests with Varjo XR-3 and LPVR-CAD and with Meta Quest 3 and LPVR-AIR show that marker-based outside-in tracking with IMU fusion gives repeatable overlay alignment that inside-out and Lighthouse tracking do not reach.

Tracking on a moving platform

Cars, aircraft and motion simulators break the assumption that the world is static, and neither inside-out nor plain outside-in tracking survives it unchanged. That problem, and the differential IMU approach LPVR-DUO uses to solve it, has its own article: VR and AR in moving vehicles and motion simulators.

Summary

See the LPVR product overview for supported headsets and specifications, or contact us to discuss your installation.