Hand tracking is a technology that helps you operate in a VR or MR environment and interact with it without separate controllers. Instead, your hands turn into a virtual version of themselves, and you can naturally use them to control the virtual environment – just as you would interact with a real-world environment.
Hand tracking enables anyone to engage with virtual or mixed reality contents immediately and intuitively – even if they have no experience with VR or gaming. Users can interact with and move components around just by using their hands. In augmented and mixed reality, users can switch fluidly between interacting with real and virtual objects. Users can do gestures such as pinching, grabbing, or pointing to interact with virtual content. Real-world hand movements and gestures can also be mirrored by virtual avatars, allowing for a new level of immersion in your VR/XR applications and collaborations.
How does hand tracking work?
Optical cameras observe your hands and reconstruct a 3D model of their position, orientation, and finger joints. This joint data is fed into applications via standard runtimes (such as OpenXR), so software can respond to gestures and contact with virtual objects. In mixed reality, the same approach works alongside a live pass‑through view of the real world.
What can you do with hand tracking?
- Direct manipulation: Grab, rotate, and place objects without a controller.
- UI interaction: Tap, push, or pinch to activate buttons, sliders, and menus.
- Precision gestures: Point, pinch, or make custom poses for shortcuts.
- Avatars in collaboration: Mirror hand poses to improve non‑verbal communication.
- Hands‑first onboarding: Lower learning curves for trainees and visitors.
Hand tracking landmarks
Read more: