F01

The Reality-Virtuality Continuum

Extended Reality (XR) is a collective term for immersive technologies that transform or extend human perception by blending physical and digital environments. XR encompasses experiences across the Reality–Virtuality Continuum, a conceptual framework introduced by Paul Milgram and Fumio Kishino (1994). The continuum spans from the completely real environment—perceived without technological mediation—to the completely virtual environment, where all sensory input is generated by a computer. Between these two extremes lies a range of experiences that combine real and virtual elements to varying degrees. XR is not defined by a single device or interface but by its affordances—the new capabilities it offers for perceiving, interacting with, and manipulating environments. These affordances go beyond the constraints of flat screens and 2D interfaces, enabling more natural, spatial, and context-sensitive experiences.

F02

While “mixed reality” is often presented as its own category, in this course it is considered a subset of augmented reality, specifically involving high interactivity and strong spatial alignment between virtual and real elements.

Virtual Reality (VR)

VR is the endpoint of the continuum where the user’s perception is fully immersed in a computer-generated environment. Real-world sensory input is entirely replaced by digital stimuli, typically through head-mounted displays (HMDs), spatial audio, and motion tracking systems. VR is a fully synthetic experience where users are placed inside an interactive, three-dimensional virtual environment that responds in real time to their movements and actions. The core affordance of VR is total immersion—a complete disconnection from the real world, enabling unrestricted control over the virtual environment. Key characteristics of VR include:

Embodied Presence: Users experience a powerful sense of “being there” in the virtual space, often referred to as presence, reinforced by accurate head and body tracking.
Spatial Interaction: Natural gestures, controllers, or full-body tracking allow direct manipulation of virtual objects, creating intuitive and physically consistent interactions.
Environmental Control: Every aspect—lighting, physics, weather, and even gravity—can be designed and modified to suit the intended scenario, unconstrained by real-world limitations.
Perspective Shifting: Users can explore environments from different viewpoints and scales, such as shrinking to explore a microscopic world or enlarging to survey a city from above.
Sensory Integration: VR can incorporate haptic feedback, 3D soundscapes, and even scent to deepen immersion and realism.

VR excels at transporting users into environments and situations that would be impractical, dangerous, or impossible in the real world—ideal for training simulations, entertainment, and scientific visualization. It is ideal for simulation, exploration, and scenarios beyond physical possibility.

Augmented Reality (AR)

AR sits near the real-world end of the continuum, overlaying digital content directly onto the user’s perception of the physical environment. Unlike VR, AR preserves the user’s connection to reality while enriching it with contextually relevant virtual elements. AR is the real-time integration of virtual objects, information, or media into a live view of the real world, where digital elements are spatially registered and interactively responsive. The core affordance of AR is contextual enhancement—adding meaningful, actionable, and interactive information to the real world without removing the user from it. Key characteristics of AR include:

Persistent Physical Awareness: Users remain fully aware of and engaged with their surroundings, avoiding the isolation of immersive VR environments.
Context-Aware Visualization: Virtual overlays are intelligently linked to specific objects, locations, or tasks, ensuring information is timely, relevant, and situationally aware.
Hands-Free, Heads-Up Interaction: Wearable AR devices allow seamless engagement with digital content while maintaining focus on the physical task at hand.
Spatial Anchoring: Digital elements remain fixed to physical locations or objects, maintaining proper position and orientation as the user moves.
Bidirectional Interaction: Physical objects can influence virtual overlays and vice versa, enabling adaptive, real-time information systems.
Environmental Responsiveness: AR can adapt overlays to lighting, geometry, and motion in the environment for more convincing integration.

AR excels at enhancing real-world experiences—whether in industrial maintenance, navigation, education, or cultural tourism—by embedding relevant, dynamic content directly into the user’s field of view. It is ideal for in-situ guidance, visualization, and collaborative real-world tasks.

XR Across Industries

Extended Reality (XR) has evolved far beyond entertainment—today it is a core digital capability reshaping processes across the entire industrial value chain. From concept design to after-sales service, XR adoption is accelerating in manufacturing, healthcare, logistics, retail, and the public sector. Worldwide, the AR & VR market is projected to reach USD 46.6 billion in 2025, growing at a 7.42% CAGR (2025–2029) to approximately USD 62.0 billion by 2029. The largest segment is AR Software with an expected USD 14.0 billion in 2025. The United States will generate the highest revenue, at USD 12.6 billion in 2025. By 2029, the global user base is expected to reach 3.7 billion people (source: Statistica.com).

Training & Knowledge Transfer

Risk-free VR simulations build procedural skills for hazardous, rare, or costly tasks. They allow trainees to learn by doing in a controlled environment, eliminating exposure to danger. Scenario rehearsal in VR trains teams for emergency responses without real-world risk. Repeated practice in realistic simulations builds confidence and quick reaction skills. On-the-job AR coaching delivers step-by-step guidance and preserves expert knowledge as digital SOPs. This ensures that even novice workers can perform complex tasks to expert standards in real time. AR safety overlays identify hazards such as exclusion zones or high-voltage areas. They serve as real-time visual warnings, improving workplace safety compliance. Studies show up to 40–60% reduction in training time and improved knowledge retention compared to traditional methods. This efficiency translates directly into faster workforce readiness and reduced training costs. Here are some examples:

U.S. Air Force—Maintenance Training: Taqtile has transformed training and maintenance operations with its XR platform Manifest, built on Unity, empowering frontline workers across defense, manufacturing, and transportation to execute complex tasks with precision. Within the U.S. Air Force, Manifest has revolutionized aircraft maintenance training by reducing errors and bridging the skills gap through AR-based, hands-free step-by-step instructions overlaid on real-world equipment using Microsoft HoloLens, Magic Leap One, or iPads. The system enables rapid knowledge capture without programming or 3D modeling skills, supports AR/VR headsets, tablets, and browsers, and deploys quickly thanks to Unity’s flexible development tools. Trials showed Level-1 engineers completing maintenance tasks without assistance or errors when using Manifest, compared to multiple failures with traditional methods. This has allowed less-experienced personnel to confidently perform specialized work, increasing productivity, standardizing procedures, and enabling scalable knowledge transfer for both training and operational use, even in distributed environments.
HR Wallingford—Pilot Training: HR Wallingford uses XR to advance vessel pilot training for ports such as Harwich Haven Authority, which handles 40% of the UK’s container traffic. Their simulation environment uses Unity’s HDRP to model realistic wave, wind, and current conditions, integrating numerical wave and hydrodynamic models for real-time updates. The fully simulated bridge environment supports multi-user interaction with tugs and other vessels, instant playback for debriefing, and customizable scenarios for new vessel types or adverse weather. This approach shortens training cycles from months to weeks, improves pilot proficiency, enhances safety, and eliminates dependency on vessel availability, while also serving as a recruitment and outreach tool.
KLM—VR Cockpit Trainer: KLM Cityhopper developed a VR Cockpit Trainer to modernize pilot training for Embraer E175 and E190 aircraft. The trainer offers an interactive 3D cockpit, 360-degree video perspectives, and virtual walkarounds to familiarize pilots with aircraft systems. Built with Unity, it supports customizable scenarios, hand tracking, multi-platform deployment, and rapid iteration through the Unity Editor. Benefits include reduced development time, flexible training schedules, cost savings by avoiding expensive simulators, improved learning through immersive visuals, scalability for varying trainee numbers, and future multiplayer capabilities to enable collaborative sessions.
VirtaMed—VR Surgical Training: VirtaMed develops VR surgical simulators replicating robotic surgical systems for immersive, risk-free practice. Using Unity’s physics, HDRP, and Shader Graph, these simulators feature high-fidelity anatomical models, haptic feedback, and modular platforms deployable in hospitals, training centers, and conferences. They collect performance metrics for feedback and proficiency tracking. This technology accelerates mastery of robotic systems, reduces reliance on physical labs, allows unlimited practice, improves surgical outcomes, and supports continuous innovation with rapid updates and new procedure modules, extending training access globally through industry partnerships.
Cincinnati Children’s Hospital—Fighting Infant Mortality: Cincinnati Children’s Hospital has pioneered the use of XR through its “Surgeons Without Borders” initiative, leveraging VR to combat infant mortality caused by congenital heart disease. The project centers around a VR Surgical Simulation Suite and a collaborative Surgical Mediverse, enabling global surgical planning and training. Surgeons can collaboratively plan procedures using 3D digital twins of patient hearts, place valves and medical devices, and design complex baffles, all within a Unity-based VR platform. The multiuser system includes real-time multilingual translation and is supported by the Unity for Humanity grant. This technology enhances surgical precision and outcomes in resource-limited settings, connects global experts for knowledge sharing, and reduces preventable infant deaths from CHD. It also offers scalable, cost-effective training without requiring physical presence, democratizing access to advanced medical expertise.

Product & Process Innovation

Immersive design reviews enable global teams to collaborate in full scale before physical builds, shortening iteration cycles. Design flaws can be spotted early, reducing costly changes later in the process. Virtual prototyping & simulation reduce physical mock-ups, cutting material waste and speeding time-to-market. This approach also enables rapid iteration without waiting for manufacturing lead times. Digital twin integration merges live sensor data with 3D models for continuous product optimization. Real-time feedback allows engineers to refine designs and processes dynamically. *Industrial metaverse platforms** combine XR, AI, IoT, and simulation into persistent collaborative environments. These virtual spaces support 24/7 global teamwork without physical location constraints. Examples include:

Audi—Virtual Exhibit XR: Audi partnered with SXCES and Govar Studios to develop Virtual Exhibit XR, an immersive experience for Apple Vision Pro showcasing the Audi Q6 e-tron to journalists and media professionals. Using Unity’s real-time 3D engine with CAD integration, the solution offers high-resolution, photorealistic displays optimized for Vision Pro’s field of view, seamless AR/VR switching, and didactic design to explain complex vehicle technology intuitively. Built with Unity PolySpatial, it integrates existing iOS AR solutions into visionOS for rapid deployment. Beyond product launches, the platform’s immersive communication capabilities can be adapted for training, mission planning, and operational briefings, reducing the need for physical prototypes. This flexibility supports deployment across engineering, marketing, and training, while reinforcing Audi’s leadership in spatial computing and technical storytelling.
Mazda—XR Cockpit: Mazda is integrating XR into automotive cockpit HMI design as part of its 2030 roadmap, embedding Unity’s real-time 3D engine into vehicles for more intuitive, personalized, and safe driver experiences. The platform supports real-time spatial visualization of vehicle data, multiplatform compatibility, and streamlined development workflows, allowing designers and engineers to collaborate efficiently. Applications include VR-based UX testing, digital twins, configurators, and design visualization. This approach reduces cognitive load, improves user experience, speeds time-to-market, and supports Mazda’s vision of delivering human-centric mobility experiences.
Volvo—Collaboration via XR: Volvo Cars employs XR across its product lifecycle, integrating Unity with CAD, AR, and VR platforms to unify workflows from design through sales. Real-time collaboration allows engineers and stakeholders to iterate on virtual models rapidly, while immersive prototyping reduces reliance on physical builds. Volvo uses VR to test in-vehicle UX, train production staff, and engage customers with virtual configurators and AR experiences. In partnership with Varjo, they conduct road tests with photorealistic XR overlays, enabling real-world evaluation of future features. This strategy accelerates development, improves product quality, saves costs, enhances training, and elevates customer engagement.
Vancouver International Airport—XR Digital Twin: Vancouver International Airport (YVR) implemented a real-time XR digital twin integrating live and historical data into interactive 2D/3D visualizations of operations. This situational awareness tool consolidates alerts, monitors GHG emissions from aircraft in real time, and simulates scenarios for planning. YVR partnered with the Musqueam Indian Band to provide 3D technology training, advancing reconciliation and tech career opportunities. Benefits include improved operational efficiency, cost reduction, enhanced safety, sustainability leadership, and scalability for deployment at other airports worldwide.

Operations & Maintenance

Hands-free AR work instructions guide field technicians through complex procedures, lowering error rates and downtime. Workers can keep both hands on their tools while still following clear, visual guidance. Remote expert assistance enables “see-what-I-see” troubleshooting, cutting travel costs and repair times. Experts can resolve issues instantly without needing to be on-site. Real-time performance overlays display live KPIs, pressures, and temperatures directly in the user’s view. This enables immediate corrective action when operational anomalies occur. Some examples are provided below:

ABB—Maintenance Operations: Industrial robotics giant ABB leveraged XR technologies, specifically AR, to revolutionize field maintenance operations by developing the ABB Ability™ Augmented Field Procedures application, transforming traditional paper-based workflows into a fully digital, interactive experience. This multiplatform AR solution—compatible with mobile devices and Microsoft HoloLens—digitizes standard operating procedures into interactive, real-time guidance, synchronizes with control systems for immediate data capture and analysis, and enables remote communication via Microsoft Remote Assist for live collaboration between field and control room operators. Designed for safety and usability, the intuitive interface uses gestures, gaze, and voice commands, while automated workflows minimize manual input. The results include enhanced operator efficiency through guided AR procedures that reduce training dependency, improved safety via clear real-time instructions, elimination of paperwork to lower errors and administrative burden, up-to-date protocols, and seamless field-to-control room communication. Built with feedback from AR novices, the system ensures broad adoption across industrial sites.
HD Hyundai Infracore—AR Guidance: HD Hyundai Infracore has introduced AR Guidance, the construction industry’s first AR maintenance and troubleshooting solution. By overlaying interactive 3D models and real-time sensor data onto physical machinery, AR Guidance allows service staff to quickly diagnose issues and follow detailed repair workflows, including wiring diagrams and component specifications. The mobile-compatible system includes performance test guides and a 3D model viewer for beginners, with future plans for AR glasses and VR integration. Benefits include faster fault resolution, reduced downtime, improved remote support, and more intuitive guidance for new staff—positioning Hyundai as a leader in digital transformation and technical support, with adoption already expanding among European service teams and plans to reach over 80% globally.

Customer & Stakeholder Engagement

Virtual showrooms & configurators allow customers to explore products anywhere, boosting purchase confidence. Shoppers can view, customize, and interact with products before committing to a purchase. In-context AR visualization lets buyers “place” products in their own space before purchasing. This minimizes uncertainty and reduces product returns. Immersive marketing experiences create strong emotional engagement and brand differentiation. They help brands stand out by turning product exploration into a memorable experience. Spatial XR dashboards turn complex datasets into navigable 3D environments for faster, collaborative decisions. Stakeholders can identify trends and outliers more intuitively than with traditional charts. AI-enhanced analytics within XR highlight anomalies, predict maintenance needs, and recommend real-time optimizations. This proactive approach prevents costly downtime and boosts efficiency. Examples:

BMW—AR In-Vehicle Experience: BMW Group is pioneering AR technologies for enhanced in-vehicle experiences, developing AR glasses prototypes that complement head-up displays to improve safety, navigation, and passenger engagement. The system projects driver assistance information such as navigation cues, hazard alerts, and road signs directly into the driver’s line of sight, while passengers can enjoy interactive features like in-car gaming or personal movie theater modes. Proprietary 6-degrees-of-freedom tracking stabilizes AR visuals despite vehicle and head movements, and the expanded field of view surpasses that of traditional head-up displays. Adaptive display management ensures that AR glasses, head-up displays, and vehicle interfaces coordinate without visual clutter. Benefits include reduced cognitive load for drivers, immersive navigation, engaging passenger experiences, and a pathway to standardizing AR integration across manufacturers, with Unity enabling efficient prototyping and stakeholder visualization.
Bosch Rexroth—Virtual Showroom: Bosch Rexroth developed a VR showroom to present large, complex industrial hydraulic products in an interactive 3D environment, overcoming physical display challenges and increasing sales engagement. Using Unity’s real-time 3D capabilities, the showroom allows virtual product teardowns for instant internal inspection, remote access via multiplayer functions, and integration of product documentation within the virtual space. It enhances trade show presence by showcasing multiple large-scale products without transporting them, and complements traditional sales with interactive XR experiences. The approach reduced product demonstration setup from over a year to two months, doubled engagement rates, saved up to 5% in travel costs, eliminated shipping for heavy units, and expanded reach to hundreds of prospects daily—fostering deeper understanding and brand visibility, particularly among younger engineers.
Gucci—Immersive Fashion: Gucci has embraced XR technologies alongside AI to transform customer engagement and redefine luxury retail. Using VR showrooms and fashion shows, customers can explore collections and attend events virtually, while AR-powered “try-before-you-buy” experiences allow shoppers to preview eyewear, sneakers, and accessories via smartphones. Immersive spaces such as Gucci Garden and Gucci Cosmos Land, along with collaborations in Roblox and The Sandbox, extend brand presence into the metaverse. AI-driven personalization offers tailored recommendations, while XR storytelling focuses on artistry and brand connection without overt commercial prompts. The result is deeper emotional engagement, global accessibility to events, personalized shopping, and a seamless blend of physical and digital retail—aligning with Gucci’s innovation leadership and sustainability goals.

Trends in Industrial XR

Industrial XR is evolving rapidly, driven by advances in hardware, AI, connectivity, and the emergence of the industrial metaverse. Lighter, untethered headsets are becoming more comfortable and job-site ready, AI and spatial computing are enabling intelligent, context-aware environments, 5G and cloud XR are removing hardware performance bottlenecks, and persistent digital twins are reshaping collaboration and operations. Together, these innovations are making XR more scalable, interoperable, and integrated into everyday industrial workflows. Organizations are already reporting substantial benefits from XR adoption, including:

Enhanced Design Accuracy & Visualization: Inspect 3D models at true scale, catch design issues before physical builds, and iterate virtually to reduce costly late-stage changes. This early detection prevents expensive rework, minimizes waste, and ensures stakeholders have a shared understanding of the final product.
Improved Collaboration & Communication: Remote teams meet as avatars around shared digital assets, making decisions faster and with greater clarity. This reduces travel costs, eliminates scheduling bottlenecks, and enables globally distributed teams to work as if they were in the same room.
Immersive Training & Skills Development: Realistic simulations allow workers to practice complex or hazardous tasks in a safe environment, boosting knowledge retention and confidence. Training modules can be updated instantly to reflect new standards or procedures, keeping teams aligned and compliant.
Greater Efficiency & Productivity: AR guidance accelerates maintenance, assembly, and inspection processes, while VR prototyping shortens product development cycles. The result is faster task completion, fewer errors, and more consistent quality across operations.
Cost Reduction: Fewer physical prototypes, less downtime, and reduced travel translate into measurable savings and strong ROI. Digital workflows also minimize logistical expenses and can scale without significant increases in overhead.
Increased Customer Engagement & Sales: Interactive “try-before-you-buy” experiences and virtual showrooms let customers explore products in detail before purchase, building trust and brand loyalty. These experiences often lead to higher conversion rates and fewer product returns.

XR Technology Stack

Unity’s XR technology stack connects hardware to software in a way that lets developers build AR and VR apps without having to manually program for each device. You can think of it as a pipeline: data starts at the physical sensors in your XR device (like cameras, gyroscopes, or LiDAR scanners), moves upward through layers of software that interpret and process it, and finally reaches your application where it’s used to create immersive visuals, sounds, and interactions. This stack is designed to abstract away hardware differences so that developers can focus on designing features and user experiences, instead of worrying about whether the code will work on one headset but not another. It also ensures that XR applications can be deployed across multiple platforms with minimal changes—a huge advantage for industries where the same training simulation or visualization tool might need to run on different devices in different environments. Let’s review the stack from the bottom up, just like the data does in a real XR app.

F03

Review this Unity documentation to learn more about the Unity XR architecture.

Provider Implementations

These are Unity’s provider plug-ins, and they act as the direct bridge between Unity and the XR device’s hardware (e.g.,a tablet or a Meta Quest headset). They use the manufacturer’s native libraries (the low-level software provided by the device makers) to get information from sensors, cameras, tracking systems, and other hardware. They’re written and maintained either by Unity, the platform provider (like Google or Apple), or third parties. In practice, they handle all the messy hardware-specific details, so you don’t have to. For example

ARCore XR Plug-in: Connects Unity to Android’s ARCore platform for things like plane detection, light estimation, and motion tracking.
ARKit XR Plug-in: Connects Unity to iOS devices via ARKit, enabling features like face tracking and LiDAR scanning.
Magic Leap XR Plug-in: Gives Unity access to Magic Leap’s spatial computing hardware for immersive mixed reality.
OpenXR Plug-in: An industry standard that works across multiple XR devices and operating systems, so one codebase can run on many headsets.

Without provider plug-ins, Unity would not understand how to get input or send output to a specific device. This layer ensures your application can run on the exact hardware your users have, without rewriting it from scratch for each headset or AR platform. It’s what makes multi-device deployment possible.

XR Plug-in Framework Layer

This is Unity’s universal adapter for XR. It defines subsystems—standard APIs that describe common XR features like camera views, spatial tracking, and environment scanning. All provider plug-ins “speak” through these subsystems so Unity can present a single, consistent interface to developers. This means your code works across devices as long as there’s a provider that implements those subsystems. It is the middleman that normalizes wildly different hardware behaviors. Key subsystems include:

Camera: Manages how the AR or VR camera views are generated, including field of view, position, and orientation. In AR, it also blends the real-world camera feed with virtual overlays.
Display: Handles stereo rendering (one image per eye in VR) or overlay rendering for AR, ensuring visuals appear at the right depth and position for immersion.
Input: Collects user interactions like head movement, controller gestures, button presses, and hand tracking, translating them into Unity’s input system.
Image Tracking: Detects and follows specific 2D images in the real world so virtual content can be anchored to them (e.g., an engineering schematic triggering an interactive model).
Planes: Identifies flat surfaces (like floors, tables, walls) so virtual objects can be placed realistically.
Meshing: Generates detailed 3D meshes of the environment for occlusion (making virtual objects hide behind real ones) or collision detection.

The XR Plug-in Framework allows developers to build once and deploy to many platforms, dramatically reducing time and cost. For engineering applications—like AR-assisted assembly, safety training, or spatial data visualization—these subsystems ensure that key features (e.g., accurate surface detection or precise tracking) work reliably, regardless of the hardware in use.

Developer Tools Layer

These are Unity’s higher-level packages that let developers interact with XR features without directly dealing with the low-level subsystem code. They wrap complex functionality into easy-to-use components, so you can focus on building interactions and content instead of wiring up hardware APIs. Think of them as “developer power tools” that sit on top of the XR SDK. Example include:

XR Interaction Toolkit: Provides ready-made components for grabbing, throwing, teleporting, and interacting with 3D objects in VR/AR. Ideal for quickly prototyping or adding polished interaction systems without reinventing the wheel.
AR Foundation: A single API that works with both ARCore and ARKit, letting you develop AR features like plane detection, anchors, image tracking, and face tracking without platform-specific code.
MARS (Mixed and Augmented Reality Studio): Tools for building AR experiences that respond intelligently to real-world environments, useful for context-aware industrial or field applications.

Developer tools save enormous amounts of time and make complex XR features accessible without deep platform-specific knowledge. In engineering contexts, this means faster prototyping, easier maintenance, and the ability to deliver consistent AR/VR functionality across a variety of devices used on-site or in training facilities.

Applications Layer

This is where all the layers come together—the XR experience itself. Whether it is a VR simulation for safety training, an AR overlay for equipment repair instructions, or a mixed reality design review, your application sits at the top of the stack. It pulls data from the subsystems, uses developer toolkits for interactions, and delivers output to the user’s hardware. Examples:

VR Applications: Fully immersive environments, such as plant operation simulators, virtual walkthroughs of engineering designs, or hazard awareness training.
AR Applications: Overlay digital information on the real world, like annotating a live view of machinery with maintenance steps or visualizing underground utilities before excavation.

Understanding the full stack helps developers troubleshoot, optimize performance, and ensure that critical features—like accurate tracking, realistic rendering, and responsive interactions—function as intended. For engineering, this means the difference between a smooth, reliable training tool and a frustrating, inaccurate simulation.

Real-Time Interaction

All the layers of the XR technology stack — from Provider Implementations at the bottom to Developer Tools and your application at the top — work together to create a seamless, responsive experience. In practice, this means that when you move your head, walk around, or interact with objects, data travels up and down the stack in a matter of milliseconds:

Application Stage (CPU): The hardware sensors (via provider plug-ins) capture changes in your position, orientation, and environment. The CPU processes this input along with game logic, physics, and scene updates, preparing the data that needs to be rendered. (~2–4 ms)
Geometry Stage (GPU): The XR Plug-in Framework passes standardized tracking and scene data to the rendering pipeline, where the GPU transforms 3D geometry, applies lighting, and determines what is visible. (~2–4 ms)
Rasterization & Shading Stage (GPU): The rendering engine converts the processed 3D geometry into pixels (fragments), applies textures, shading, and post-processing effects, and prepares the final frame for output. (~3–6 ms)
Display Stage: The display and output systems send the completed frame to the headset or device displays, synchronized with spatial audio and any haptic feedback so that visuals, sound, and touch cues all align. (~1–2 ms)

This entire cycle — from sensor input to the user seeing and hearing the result — must happen within 8–16 milliseconds per frame, depending on the target frame rate: 8.3 ms for 120 FPS, 11.1 ms for 90 FPS, 13.9 ms for 72 FPS, and 16.7 ms for 60 FPS. Even small delays of just a few milliseconds can disrupt immersion: in VR, higher latency can cause motion sickness due to mismatches between physical movement and visual updates; in AR, lag can break spatial alignment, such as a virtual label attached to a moving machine drifting behind it. Maintaining this real-time performance demands powerful hardware, efficient algorithms, and an optimized graphics pipeline so the XR experience feels immediate, natural, and convincing.

F04

XR Hardware

The XR technology stack relies on specialized hardware to capture inputs and present outputs. Here we introduce the major hardware components in XR systems, from the headsets you wear to the sensors and controllers you use to interact with virtual content.

Headsets

Headsets, also referred to as Head-Mounted Displays (HMDs), are devices worn on the head that place images directly in front of your eyes. They are the primary hardware for most VR and MR systems:

VR Headsets: These headsets completely cover your field of view with screens, fully immersing you in a virtual environment. Examples include the Meta Quest (formerly Oculus Quest), HTC Vive, and Varjo headsets. VR headsets usually have built-in displays (one for each eye) and lenses that focus and enlarge the image. They also include sensors (like accelerometers, gyroscopes, and often external or inside-out cameras) to track head movement so that the view in the virtual world updates as you look around. High-end VR headsets often come with handheld controllers for interaction, and some are tethered to a PC while others (like the Quest) are standalone (all processing is done in the headset itself).
AR Headsets: These are headsets with transparent or see-through displays, allowing you to see the real world while layering digital images on top of it. Examples include the Microsoft HoloLens and Magic Leap. Instead of opaque screens, AR headsets use transparent optics (such as waveguides) to project digital light into your eyes, so virtual objects appear as holograms superimposed on your environment. These devices come with multiple cameras and sensors to track your head and hand movements and to map the real environment (so virtual objects can be placed correctly on surfaces and relative to real objects). They often have on-board processors, since they are self-contained computers. They may also use gesture recognition or voice commands for input, in addition to optional controllers.

F05

Mobile AR

You don’t always need a dedicated headset to experience XR. Mobile devices like smartphones and tablets can act as XR platforms; well, AR to be exact. In mobile AR, the phone or tablet’s screen becomes a “window” through which you see the augmented world. The device’s camera captures the real world and displays it on screen, while the software overlays digital content on that live camera view. A great example is using your phone’s camera to see how a piece of furniture would look in your room via an AR app–the furniture is rendered on the screen as if it’s in your room when you view it through the phone.

Modern smartphones are quite powerful and come packed with sensors ideal for AR. They typically have high-resolution cameras, IMUs (inertial measurement units) for motion tracking, and even LiDAR scanners (Light Detection and Ranging) in some devices (like recent iPhone and iPad Pro models) for depth sensing. Platforms like Apple’s ARKit and Google’s ARCore leverage these sensors to allow developers to create sophisticated AR experiences on mobile devices. Mobile XR is very accessible, since many people already have smartphones, and it doesn’t require any special hardware beyond the device. However, it typically offers a less immersive experience compared to using a headset, because you’re still looking at a screen in your hand rather than seeing the digital content in your full field of view.

Projection Systems

XR can also be experienced through projection-based systems, which use projectors to blend digital content with physical environments. These setups are often used for specialized applications, collaborative experiences, or large-scale immersive environments. Projection systems are typically fixed to specific locations and require controlled environments to function effectively, especially in terms of lighting and surface preparation.

CAVE Systems (Cave Automatic Virtual Environment): A CAVE is a room-sized VR environment where images are projected onto multiple surfaces—walls, floors, and sometimes ceilings—to create an immersive virtual space. Users typically wear 3D glasses to perceive depth, allowing them to experience a shared virtual environment without needing headsets. CAVEs are particularly useful for group visualization, such as architectural walkthroughs, scientific simulations, or industrial design reviews. While not as common as head-mounted displays due to their size and cost, CAVEs offer a unique way to engage multiple users in VR experiences simultaneously.
Projection-Based AR (Projection Mapping): In projector-based AR, digital visuals are projected directly onto real-world surfaces, effectively turning any physical object or space into an interactive display. This technique, known as projection mapping, allows virtual content to conform to the shapes and textures of physical environments. For example, a projection system can overlay dynamic information onto a factory machine to guide maintenance tasks, or transform a blank tabletop into an interactive game board. Unlike AR through glasses or mobile devices, projection-based AR doesn’t require users to wear or hold any hardware—making it ideal for public installations, museums, exhibitions, and collaborative workspaces. Advanced systems may include sensors to detect user interactions, such as hand gestures or object movement, enabling real-time interaction with the projected content.

Sensors and Peripherals

To interact with XR effectively, various sensors and peripheral devices are used in conjunction with the main display. These capture user input and provide additional feedback:

Cameras and Depth Sensors: Cameras are fundamental for AR devices to “see” the world. For example, AR headsets and mobile AR use cameras to detect surfaces and track movement. Depth sensors like LiDAR or time-of-flight cameras measure distances to objects in the environment, creating a depth map. This helps an XR system understand the 3D layout of the real world – crucial for placing virtual objects believably and for safety (knowing where walls and obstacles are). The Microsoft HoloLens, for instance, uses a depth sensor to map rooms and track objects in real time.
Motion Trackers (IMUs): Most XR hardware includes an IMU–a combination of accelerometers and gyroscopes (and sometimes magnetometers)–to track the orientation and acceleration of the device or controller. These sensors update rapidly (hundreds of times per second) to detect movements. High-end VR systems (like Varjo XR-4) also use external tracking cameras or base stations that track infrared markers on the headset and controllers for very precise position tracking in a room-scale space.
Handheld Controllers: VR headsets often come with motion controllers–handheld devices with buttons, triggers, and tracking sensors. They let you grab objects, point, and interact with the virtual world. For example, Meta Quest controllers and HTC Vive wands are tracked in 3D space so the system knows where your hands are. These controllers usually provide basic haptic feedback (a rumble or vibration) to make interactions like hitting an object or firing a laser feel more tangible.
Hand Tracking: In addition to or as an alternative to handheld controllers, some XR systems enable hand tracking, where your bare hands act as the input. Cameras on a headset (or external devices like the Leap Motion sensor) can analyze the movement of your hands and fingers, allowing you to touch and manipulate virtual objects directly with your hands. This increases immersion since you don’t need any gadgets – your natural gestures become the controller. Many modern VR and AR headsets (like Quest 3, HoloLens 2, Magic Leap 2) have built-in hand tracking capabilities through their cameras.
Haptic Feedback: These devices provide tactile sensations to match what’s happening in the XR experience. The simplest haptic feedback comes from vibration motors in controllers (providing a buzz when you collide with something). More advanced haptic devices include gloves that give force feedback on your fingers (to simulate the feeling of touching a solid object) or vests that let you feel impacts on your torso. While still an emerging area, haptics aim to engage the sense of touch, making XR experiences more immersive by adding a layer of physical feedback.

XR Software

On the software side, the XR stack includes the engines, frameworks, and systems that create and run XR content. These software components handle everything from generating 3D graphics to interfacing with the hardware devices.

Rendering Engines

A rendering engine is the software that draws the virtual world and handles the logic of the experience. In XR development, this is typically a game engine–a software framework originally designed for video games, which is also well-suited for interactive simulations in XR. The two most popular engines for XR are Unity and Unreal Engine. Both Unity and Unreal support all major XR hardware through plugins or integration with XR SDKs.

Unity: Unity is a widely used cross-platform game engine known for its accessibility and a rich ecosystem for developers. In XR, Unity plays a critical role by providing the tools to build virtual environments and interactions. With Unity, developers can design scenes (arranging 3D models, lights, and cameras), apply physics (so objects have gravity or collide realistically), and write scripts that define how the world behaves. Unity’s real-time rendering capabilities mean it can generate the graphics for VR/AR at the high frame rates needed for smooth experiences. For example, Unity has built-in support for VR headsets and AR through packages like AR Foundation and the XR Interaction Toolkit, which help manage display, tracking, and user input for various devices. Its engine takes care of rendering separate images for each eye in VR (for stereo vision), handling details like perspective and lens distortion, and it integrates with device SDKs so that the developer’s code can easily use head tracking data or trigger haptic feedback. Because of its user-friendly interface and strong community, Unity is often the first choice for new XR developers.
Unreal Engine: Unreal Engine, developed by Epic Games, is another powerful real-time 3D engine commonly used in XR, especially when high-fidelity graphics are required. Unreal is known for its advanced rendering quality (it’s often used in cutting-edge VR experiences with very realistic visuals). Like Unity, it provides a framework for building 3D worlds, simulating physics, and handling input. Unreal Engine offers a visual scripting system called Blueprints, which can be helpful for designers to create interactions without writing code. While Unity currently has a larger share of the XR market (due to its ease of use and early focus on mobile/standalone VR), Unreal is frequently used for high-end simulators, complex VR training systems, and AR projects that demand top-notch graphics.

XR SDKs and APIs

To build XR applications, developers rely on Software Development Kits (SDKs) and Application Programming Interfaces (APIs) provided by platform creators or standard bodies. These SDKs/APIs make it easier to access device features and sensors, so developers don’t have to start from scratch for each hardware type. Some key SDKs/APIs include:

ARKit: ARKit is Apple’s AR development framework for iOS devices. It provides tools to build augmented reality apps for iPhones and iPads. ARKit handles tasks like motion tracking (using the phone’s IMU and camera to track the device’s movement in space), environment understanding (detecting surfaces like floors and tables, recognizing images or objects), and light estimation (so virtual objects can match the lighting of the real environment). Developers use ARKit through Apple’s APIs (in Swift or Objective-C), and the framework takes care of the heavy lifting of tracking and mapping the world. Many popular AR apps on the iPhone–from measuring tools to games–use ARKit under the hood.
ARCore: ARCore is Google’s platform for building AR applications on Android. Similar to ARKit, ARCore lets developers create AR experiences on common smartphones and tablets. It performs motion tracking (also called visual-inertial odometry, which combines camera input with inertial sensor data to track the device’s position as it moves), detects flat surfaces in the environment (for example, finding a table or floor where virtual objects can be placed), and estimates lighting conditions. Developers can use ARCore via the Android SDK or in engines like Unity (Google provides plugins for Unity and Unreal). ARCore and ARKit have many similar capabilities – in fact, both are moving toward the open standard OpenXR to make AR apps more portable across different devices.
OpenXR: OpenXR is an open, royalty-free standard API developed by the Khronos Group that aims to unify the XR ecosystem. Instead of each VR/AR device having its own proprietary SDK, OpenXR provides a common set of functions for interacting with XR hardware. This means an application built with OpenXR can run on different brands of headsets or glasses without needing separate code for each. For example, an OpenXR app can run on a Magic Leap headset, a Meta Quest, or a Valve Index, as long as each system supports the OpenXR runtime. Unity and Unreal Engine both support developing XR applications using OpenXR, so developers can write code once and deploy it to multiple platforms with minimal changes. This standardization greatly reduces fragmentation and makes life easier for XR developers and companies.

Key Takeaways

This first session has introduced you to the fundamentals of XR, its position along the Reality–Virtuality Continuum, and the core concepts of VR and AR as applied in engineering. You’ve seen how XR is not just a visual novelty but a practical, high-impact tool—shaping design, training, operations, and customer engagement across industries. We explored real-world case studies, examined the XR technology stack from hardware to software, and unpacked how Unity’s framework enables cross-platform development for both immersive and augmented experiences. As you move forward in this course, remember that XR is most powerful when matched with clear engineering goals. It is not about chasing the newest headset—it is about using spatial computing to solve problems more effectively, communicate ideas more clearly, and create experiences that would be impossible in 2D. In the next sessions, we’ll build on this foundation, moving from understanding the ecosystem to creating and optimizing XR solutions that deliver measurable value in engineering contexts.