E1. Fundamentals of AR

---

Learning Outcomes

• Differentiate AR from VR and explain its role in engineering applications. Before class, review how AR uses spatial overlays to enhance real-world tasks and provide in-context information.
• Describe the AR development lifecycle and identify core Unity components for AR scenes. Ahead of the session, read through the “Prototyping and Creation” stage and run a basic AR plane detection scene, making note of AR Session components and object placement logic.
• Identify essential AR software tools and their functions in Unity development. To prepare, familiarize yourself with AR Foundation and XR Plug-in Management, and follow the installation steps that enable AR features in Unity.
• Compare major AR hardware devices and their capabilities. In advance, summarize the differences in tracking, field of view, and ergonomics for devices like HoloLens 2, Magic Leap 2, and Apple Vision Pro.
• Explain why AR can be advantageous over VR in certain engineering scenarios. Reflect on situations where AR offers real-time, contextual assistance that VR cannot, and come ready with one example to discuss.

---

What Is AR?

Augmented Reality (AR) is an XR technology that overlays digital content onto the user’s view of the real world. Unlike VR, which replaces the environment entirely with a computer-generated simulation, AR enriches the physical environment with interactive digital layers. The term “augmented” signifies that the real world is enhanced with additional information, helping users gain deeper insights and a more engaging perspective of their surroundings. In a physical factory, AR can enhance real-world factory operations by overlaying virtual content directly onto physical environments. For instance, at the engine assembly station, users can view holographic engine components precisely aligned with a physical engine platform and its components, along with step-by-step instructions and part labels. This real-time augmentation improves spatial reasoning, guides accurate assembly, and reduces errors—turning AR into an interactive, context-aware assistant that supports hands-on engineering tasks. Key features of AR include:

• Spatial Mapping and Tracking: AR systems must continuously capture and analyze data from the physical environment. This involves using sensors (e.g., cameras, depth sensors, LiDAR) to map surroundings and track the precise location and orientation of both the user and digital objects in real time. When a user navigates through the logistics station, AR devices can detect walls, racks, and forklifts to create a spatial mesh. A drone flying in the warehouse can be visualized following real spatial constraints, weaving between shelves without clipping through them—thanks to accurate spatial tracking.

• Registration and Alignment: Accurate alignment or registration of virtual content with real-world objects is essential. This ensures that digital overlays appear to interact correctly with physical elements, maintaining spatial coherence even as the user moves. Using image markers, user can place a virtual 3D printer on a physical table for simulation and educational purposes. Once anchored, the virtual 3D printer remains perfectly aligned with the table surface, even if the user walks around or looks from different angles.

• Environmental Understanding: AR technology leverages computer vision and machine learning to recognize and interpret objects and surfaces in the environment. This understanding is used to determine occlusion (which objects should appear in front or behind) and to provide realistic integration of virtual content, including proper shadows and lighting. The virtual quadruped robot can walk across the physical floor while being occluded correctly by tables and furniture and stopped when colliding with physical obstacles. This is can be accomplished using environmental mesh scanning that enables believable occlusion and realistic lighting, enhancing visual fidelity and user immersion.

• Real-Time Performance: Because AR experiences require instantaneous integration of digital data with a live feed of the physical world, low-latency processing is crucial. This real-time capability enables smooth interactions and minimizes lag that could disrupt the immersive experience. During AR-based robotic machine tending training and simulation, users can interact with virtual robot arms and their digital twin in sync with real machine cycles. Any noticeable delay would make the overlay useless, so performance optimization is a key learning outcome.

• Sensor Fusion: Combining data from multiple sensors (such as cameras, gyroscopes, accelerometers) enhances the system’s reliability and accuracy. Sensor fusion allows AR applications to operate in varying conditions and environments, improving robustness in tracking and spatial awareness. The virtual mobile robot in the assembly station can drive across the physical floor, avoiding virtual and real obstacles using data fused from the AR headset’s IMU, camera feed, and depth sensors.

• User Interaction and Interface: An intuitive and responsive interface is fundamental. AR applications often utilize gesture recognition, voice commands, or touch inputs, making the interaction smooth and natural. Designing user-friendly interfaces enhances the seamless integration of digital elements into the physical world. Users can use hand gestures to rotate, assemble, or disassemble a virtual V8 engine placed on a physical engine block. With hand tracking, they manipulate parts as if they were real—strengthening spatial reasoning, learning, and ergonomic interface design skills.

• Context Awareness: AR systems need to understand and interpret contextual data, including location, orientation, and the dynamic state of the task, the user, and the environment. This awareness enables them to provide meaningful, context-sensitive overlays and interactions that enhance the user’s experience. When standing near a physical welding station, AR headsets can automatically trigger a safety overlay that visualizes heat zones, danger areas, and robotic motion paths, enabled by context-sensitive UI design and environment-based triggers.

AR is being integrated into manufacturing workflows to enhance design reviews, improve client presentations, streamline remote assistance, and accelerate workforce development.

---

AR Development Lifecycle

AR development in Unity follows a structured lifecycle that seamlessly integrates digital content into the physical world using AR Foundation. From early planning to final deployment, successful AR projects rely on well-defined goals, intuitive user interfaces, and real-time spatial awareness. Leveraging features like plane detection, image tracking, and environmental probes, developers can create context-aware experiences that feel natural, responsive, and tightly aligned with the user’s physical environment.

Conceptual Design

Strong AR projects begin with clear objectives, hardware considerations, and spatial planning that considers both physical context and digital overlays. Defining how digital content enhances real-world tasks from the start ensures a more seamless and purposeful user experience throughout development.

• Ideation and Planning: Define the purpose of the AR experience—educational overlays, guided procedures, or context-sensitive data visualization. In a factory, handheld AR can be used to guide users through robotic welding procedures by overlaying safety zones and machine statuses directly on the physical equipment on the shop floor. Clear goals at this stage also help determine the scope of features, UI complexity, and expected hardware support.

• User Experience (UX): Design interactions appropriate to the AR platform—touch-based for mobile devices and gesture/voice-based for headsets. On a tablet, users can tap virtual UI buttons anchored to real racks in the logistics station. With an AR headset, they use hand gestures to explore a 3D printer’s interface. It is also important to ensure interactions are intuitive and don’t obstruct users’ view of the real-world workspace.

• Device and Environment Planning: Choose between handheld and head-mounted AR based on use case, spatial requirements, and user mobility. Handheld AR is used for scanning QR/image markers on tables to display anchored robots. Head-mounted AR supports free movement and interaction in complex tasks like engine assembly or troubleshooting, enabled by hand tracking. Environmental conditions—like lighting, reflective surfaces, and physical obstacles—should also be considered early.

• Scene Planning and Setup: Begin with a grounded AR session setup. For mobile AR, this includes AR Session, AR Session Origin, and AR Camera. For head-mounted, configure device-specific profiles and calibration. An AR Session Origin can be configured to detect planes in the welding cell where users can place a virtual welding robot in correct alignment with the physical floor. Good spatial setup ensures digital content aligns properly across different user positions and environments.

• Content Anchoring: Determine how digital elements should be registered to the real world—using surface detection, image markers, or persistent anchors. The quadruped robot is spawned using plane detection and remains fixed to the detected floor even as the user walks around, enhancing realism and spatial coherence. Anchoring methods must be tested for stability and robustness, especially when devices are restarted or tracking resets.

Prototyping and Creation

In this phase, developers use quick iterations to validate AR interactions, surface detection, and user flow. Rapid prototyping helps uncover design flaws and spatial interaction issues before committing to high-fidelity content.

• Rapid Prototyping: Use mock assets and simple UI elements to block out interactions and spatial anchors. A prototype scene places virtual CNC parts on a physical table by scanning an image marker—used in XFactory to simulate inspection tasks using various virtual measurement gauges. Early prototypes often skip visuals and focus on validating tracking, placement, and UI flow in real environments.

• AR Plane and Feature Detection: Enable plane detection and raycasting to place objects in the physical world. Use debugging tools like visualizers and gizmos. Users can use handheld AR to place and resize a virtual display on the physical wall using detected vertical planes. It is essential to test detection robustness under varied lighting and surface textures.

• Testing in Context: Simulate in-editor when possible, but always validate behavior on physical AR devices early and often. Developers test placing the engine block in AR over a physical engine platform using both mobile AR and spatial AR, validating position drift and lighting differences between platforms. This step also includes assessing user ergonomics and real-world workflow integration.

Interaction and Iteration

This stage focuses on responsive interactions using AR Foundation’s tracking systems, Unity’s Input System, and gesture or touch input logic. Testing and refining interactions ensures that the AR content feels natural, timely, and anchored in the real world.

• Interaction Design: Use gesture-based (e.g., air taps, hand pinch) or screen-based (e.g., tap, swipe) inputs for object manipulation, menu control, and system feedback. For example, users can tap to spawn or assemble parts of an engine. On an AR headset like Magic Leap or HoloLens, they use hand pinching to grab and rotate engine components in place. Designing fallback or alternative controls is helpful for accessibility and device compatibility.

• AR Foundation Features: Integrate subsystems like plane detection, image tracking, object tracking, and occlusion using AR Foundation components. The mobile robot can be placed and tracked in AR via surface detection. Users can observe how it interacts with occlusion meshes like tables, showcasing depth-aware spatial behavior. Leveraging these features adds realism and improves spatial interaction fidelity.

• Multi-Sensory Feedback: Use sound and visual feedback for actions—snap sounds, highlights, and haptics when supported. When placing a virtual drone simulates box scanning in a physical logistics station, a click sound and an adaptive animation can confirm alignment and distance. Vibrations or audio pings can guide users during complex tasks or when alignment is off.

• Iterative Refinement: Gather user feedback, tweak interaction distances, improve registration accuracy, and streamline UX. For example, users may prefer a larger, radial UI for the mobile AR app to make button presses easier when anchoring robots to narrow surfaces. Ongoing usability testing helps identify friction points and ensures long-term comfort during extended AR sessions.

Optimization and Deployment

Final polish ensures the AR experience is stable, performant, and cross-platform ready. This phase focuses on packaging for target hardware, optimizing runtime efficiency, and preparing the app for distribution or enterprise deployment.

• Performance Tuning: Profile device CPU/GPU usage, reduce draw calls, optimize shaders, and limit unnecessary raycasting or updates. AR assets can be baked with lightweight materials for mobile AR, while head-mounted versions use LOD groups for distant objects. Developers can also adjust update intervals for less-critical background elements to save power and improve responsiveness.

• Session Management: Use ARSession.Reset() strategically to clear tracking or reload environments. Ensure seamless startup and recovery. When the tracking drifts or misaligns during V8 engine assembly, users can reset the session to recalibrate the anchors to the image marker beneath the platform. Include safeguards and UI prompts to inform users when resets are necessary.

• Platform Packaging and Build Targets: Configure build settings per platform using AR Foundation’s multi-platform support. Test builds on Android (ARCore), iOS (ARKit), or HoloLens/Magic Leap. An AR app can be deployed to Android tablets for handheld use and HoloLens 2 for immersive training, using platform-specific settings under Unity’s XR Plug-in Management. Build automation and platform-specific prefabs help manage differences in interaction or resolution.

• Cloud Anchors and Persistence (Advanced): For persistent content across sessions or users, integrate cloud anchor services or save/load local anchors. In a collaborative training session, two users can use AR to jointly assemble an engine, with both devices anchored to the same real-world marker using persistent AR anchors. This enables collaborative workflows, spatial memory, and multi-user state synchronization across sessions and locations.

---

AR Software

Unity offers a powerful platform for developing cross-device AR applications, supported by AR Foundation and extended through native SDKs like ARKit and ARCore. These software components manage real-time sensor data, surface detection, device-specific features, and spatial interaction—enabling engineers to build high-fidelity, context-aware AR experiences.

AR Foundation

AR Foundation is Unity’s abstraction layer for building AR experiences that run on multiple devices, including iOS (ARKit) and Android (ARCore). Core features include:

• Cross-Platform Abstraction: AR Foundation serves as a unified interface that wraps around platform-specific AR SDKs—such as ARKit (iOS), ARCore (Android), and Magic Leap SDK—providing a consistent API for development. This allows developers to build once and deploy everywhere, significantly reducing the complexity of multi-platform AR projects. The abstraction also simplifies maintenance and testing, since logic and features remain centralized in one codebase.

• Feature Modules: AR Foundation is modular, meaning you can selectively enable subsystems based on your project’s needs—like plane detection for placing objects on real-world surfaces, image tracking for marker-based content, face tracking for AR masks or expressions, and occlusion to let digital objects appear realistically behind real-world ones. These modules are component-based in Unity, making them easy to configure and integrate within the Unity Editor.

• Device Independence: AR Foundation dynamically adapts to each device’s capabilities at runtime. For example, if image tracking is not available on a specific Android device, AR Foundation will automatically disable that subsystem to avoid crashes. This makes it easier to support a wide range of hardware without requiring custom logic for every device. Developers can query device support for features and provide alternate interactions or fallbacks as needed.

ARKit (iOS)

ARKit is Apple’s native AR SDK for iOS and iPadOS. It offers robust tracking, scene understanding, and tight integration with Apple’s hardware ecosystem, making it ideal for handheld AR development using iPhones and iPads. Developers benefit from advanced sensor data, efficient camera processing, and system-level support for AR content. Core features include:

• LiDAR Support: Provides precise depth sensing and spatial mesh generation for realistic placement and occlusion of virtual objects on supported devices (e.g., iPad Pro, iPhone Pro models).
• People Occlusion: Allows virtual objects to appear naturally behind people in the camera feed, enhancing realism in mixed scenes.
• Object & Image Recognition: Supports detection of 2D images (like posters or signs) and 3D reference objects for anchored or triggered AR content.
• World & Face Tracking: Delivers smooth world tracking for stable AR placement and enables real-time face tracking for effects, filters, or expression-driven UI.

ARCore (Android)

ARCore is Google’s AR platform for Android, offering powerful AR capabilities across a wide variety of devices. It uses sensor fusion and computer vision to deliver spatial awareness, enabling applications from entertainment to industrial workflows. Core features include:

• Motion Tracking: Combines camera input and IMU data to accurately estimate device movement in 3D space (visual-inertial odometry).
• Environmental HDR: Matches virtual lighting with real-world conditions using real-time light estimation, enhancing photorealism.
• Plane Detection & Anchoring: Identifies horizontal and vertical surfaces, allowing developers to place and persist virtual content in real-world space.
• Depth API: Offers real-time depth data for more immersive interactions—like realistic object occlusion and depth-based visual effects.

Magic Leap SDK

Magic Leap 2 is an advanced spatial computing headset that delivers immersive AR experiences with precise interaction and environmental awareness. Its SDK supports Unity integration and a growing suite of spatial features tailored for enterprise and collaborative use. Core features include:

• Spatial Anchors: Allows content to be fixed in the real world across sessions, enabling persistent workflows and scene continuity.
• Scene Understanding: Automatically classifies surfaces like floors, ceilings, and walls to support intelligent content placement and interaction logic.
• Eye and Hand Tracking: Provides accurate gaze input and fully articulated hand tracking, enabling intuitive, controller-free interactions.
• Persistent Spatial Memory: Stores and reuses environment maps, allowing devices to “remember” spaces for shared AR experiences and synchronized collaboration.

Mixed Reality Toolkit (MRTK)

MRTK is Microsoft’s open-source framework for developing mixed reality applications, primarily on HoloLens 2. It provides building blocks for spatial interaction, cross-platform support, and UI components optimized for hands-free, gaze-first workflows. Core features include:

• Hand and Eye Tracking: Fully supports HoloLens 2’s articulated hand input and eye gaze, enabling natural manipulation, selection, and attention tracking.
• Spatial Mapping and Understanding: Generates 3D meshes of the physical environment and identifies semantic surfaces for better spatial interaction and navigation.
• World Anchors and Sharing: Enables persistent AR content and multi-user synchronization by anchoring objects in shared physical space.
• UX Building Blocks: Offers prebuilt components like hand-friendly buttons, sliders, radial menus, and dialogs optimized for head-mounted AR use with minimal coding.

Feature Support Comparison

Feature	ARKit (iOS)	ARCore (Android)	Magic Leap 2	MRTK (HoloLens 2)
Plane Detection	✅ Horizontal/Vertical	✅ Horizontal/Vertical	✅ Semantic Surfaces	✅ Horizontal/Vertical
Image Tracking	✅	✅	✅	✅
Object Tracking	✅	❌	✅	✅
Spatial Anchors (Persistent)	✅ (limited)	✅ (Cloud Anchors)	✅ Robust + Persistent	✅ Local & Shared Anchors
Scene Understanding	✅ (via LiDAR)	⚠️ (basic depth)	✅ Semantic Mapping	✅ Spatial Mapping
Occlusion Support	✅ People, Mesh	✅ Depth-based	✅ Full Mesh + Physics	✅ Full Mesh + Hand Occlusion
Environmental HDR	❌	✅	✅	❌
LiDAR / Depth Sensing	✅ (iPad Pro, etc.)	⚠️ (Depth API)	✅ Native Hardware	✅ (via Spatial Mapping)
Face Tracking	✅ TrueDepth	❌	✅ Basic Eye Gaze	✅ Eye Tracking
Hand Tracking	❌	❌	✅ Multi-Hand Support	✅ Full Articulated Hands
Eye Tracking	❌	❌	✅	✅
Gesture Input	❌ (Touch only)	✅ (Touch)	✅ (Hands/Gaze)	✅ (Hands/Gaze/Voice)
Persistent Spatial Memory	❌	❌	✅	✅
UX Toolkit	❌	❌	⚠️ Basic SDK UI	✅ MRTK UI Components
Device Type	Handheld	Handheld	Head-Mounted	Head-Mounted

✅ = Full support

⚠️ = Partial/Limited

❌ = Not supported

AR Foundation abstracts many of these features and capabilities, allowing developers to build once and deploy across platforms. For advanced use cases or full access to hardware-specific functionality, direct SDK integration may still be required.

XR Plugin Management

XR Plugin Management is Unity’s system for managing and loading XR plug-ins. It abstracts the platform-specific details, allowing developers to switch between different XR runtime environments (such as AR and VR platforms) without altering much of the core project. With XR Plugin Management, you can build XR apps that work across multiple VR/AR devices by simply swapping plug-ins. It streamlines configuration for multiple XR platforms, reducing overhead in multi-device projects. In summary, its core concepts include:

• Abstraction Layer: Provides a unified interface for handling various XR hardware and platforms.
• Plug-in Ecosystem: Supports multiple providers, enabling support for devices from different manufacturers.
• Configuration and Optimization: Helps manage XR settings, optimizes performance, and ensures that the correct plug-in is active based on the target platform.

In Unity, plugins are external libraries—either managed (C# DLLs) or native (C++, Java, Objective-C)—that extend the engine’s functionality, often enabling platform-specific features or third-party integrations. They are commonly used for tasks like accessing native device APIs, integrating SDKs, or enhancing editor tools.

XR Interaction Toolkit

The XR Interaction Toolkit is a framework provided by Unity for creating and managing user interactions in XR environments. It offers components and tools that enable developers to easily set up common interaction patterns such as grabbing, teleportation, and UI manipulation. With the XR Interaction Toolkit, you can quickly prototype and deploy VR games or AR experiences with intuitive user interactions and create realistic interaction scenarios for simulation training programs. This toolkit provides a rapid development toolset for experimenting with new interaction paradigms in XR. Its core concepts include:

• Interaction Components: Pre-built components for common interactions (e.g., ray interactor, direct interactor, grab interactable).
• Event-Driven Architecture: Utilizes Unity events to trigger interactions, making it easier to extend and customize behavior.
• Modularity: Designed to be modular so that developers can mix and match components based on the needs of their application.

OpenXR

OpenXR is an open standard for XR application development, designed to standardize how VR and AR hardware interact with software. In Unity, this standard is supported through provider plug-ins, which serve as bridges between the OpenXR API and specific XR hardware. Using OpenXR, you can write code that works on multiple XR devices without needing to change the underlying implementation. With a standard API, different hardware manufacturers can collaborate on improving XR experiences. Core concepts include:

• Open Standard: A universal API that reduces fragmentation across various XR devices.
• Provider Plug-ins: Specific implementations that allow Unity to communicate with different hardware ecosystems (such as Meta Quest, Windows Mixed Reality, etc.).
• Interoperability: Ensures that applications built on OpenXR can work with any hardware that supports the standard, fostering a more unified development ecosystem.

Input System Package

The Input System package in Unity is a modern framework for handling user input across a wide variety of devices. It replaces the older input system with a more flexible, action-based architecture, making it especially valuable in XR development where diverse input methods—such as controllers, hand gestures, eye tracking, and voice commands—are common. This package allows developers to seamlessly integrate inputs from devices like Magic Leap 2, including its hand-tracking, eye-tracking, and 6DoF controller support. It enables the creation of sophisticated, context-aware interaction systems ideal for spatial computing applications requiring custom input mappings beyond traditional peripherals. Core features include:

• Action-Based Input: Uses input actions and bindings, allowing developers to map various input types (e.g., pinch, gaze, trigger) to logical in-app behaviors.

• Extensibility: Supports custom devices and gesture inputs through a modular architecture, enabling integration with Magic Leap’s unique interaction models.

• Event-Driven Model: Responds to user input through event callbacks, providing more responsive and decoupled input logic.

Voice commands, hand pinch gestures to grab machine components, and thumbstick input are managed using action maps defined in Unity’s Input System. This modular setup allows input profiles to be toggled or remapped per station or task with minimal effort.

XR Simulation

XR Simulation enables in-Editor testing of AR applications by simulating XR device behavior in virtual environments. It allows developers to move through preconfigured or custom scenes—like living rooms or office spaces—and simulate real-world AR features without deploying to physical devices. It is useful for testing AR features like plane detection, point clouds, or anchors in contextual environments without needing a phone or headset. Using this tool, you can quickly iterate on UI placement, environment probes, or surface interactions within Unity Play Mode using sample or custom scenes. Core features include:

• Environment-Based Simulation: Loads realistic 3D environments where AR Foundation simulates device movement and tracking.
• Simulated AR Subsystems: Supports key AR features like plane detection, image tracking, anchors, and occlusion within Play Mode.
• Keyboard & Mouse Navigation: Developers can navigate XR Simulation environments using familiar desktop inputs to simulate movement and interaction.

AR Feature	Supported
Session	✅
Device Tracking	✅
Camera	✅
Plane Detection	✅
Bounding Box Detection	✅
Image Tracking	✅
Object Tracking	❌
Face Tracking	❌
Body Tracking	❌
Point Clouds	✅
Ray Casts	✅
Anchors	✅
Meshing	✅
Environment Probes	✅
Occlusion	✅
Participants	❌

AR Foundation uses Editor-only simulation providers when in Play Mode. When building to a device, the correct platform-specific provider (ARKit, ARCore, etc.) is automatically used instead. No changes to scenes or build settings are needed to enable XR Simulation in the Editor.

---

AR Hardware

AR hardware spans a broad spectrum from conventional mobile devices to specialized head-mounted displays (HMDs) and smart glasses. Each type of hardware comes with its own strengths and is chosen based on the specific application needs within engineering and other fields. AR devices must provide clear views of the physical world while overlaying digital content seamlessly. This is critical for ensuring that the virtual information enhances rather than obstructs reality. Whether used for a short demonstration or extended work sessions, AR hardware should be lightweight and comfortable. Prolonged use requires careful design to minimize fatigue and maximize user engagement.

• See-Through AR: Common in optical AR devices like Magic Leap or HoloLens, this approach uses transparent lenses to overlay digital content directly onto the user’s view of the real world. It preserves full environmental awareness but may suffer from limited brightness and color contrast in bright conditions.

• Passthrough AR: Used in devices like Apple Vision Pro or Meta Quest 3, this method captures the physical world via external cameras and re-displays it with virtual overlays. It offers more control over visual effects and occlusion but can introduce latency and reduce natural depth perception compared to optical systems.

Sensors and Cameras

At the heart of every AR device are the sensors and cameras that enable environmental awareness and user interaction:

• Environmental Mapping: Sensors capture spatial details to align digital overlays precisely with real-world objects. This process involves real-time data acquisition from cameras, depth sensors, and sometimes LiDAR to create an accurate model of the physical space.
• Gesture and Motion Tracking: These sensors enable intuitive interactions by detecting user movements and gestures, which is crucial for applications that require hands-free control or natural interactions with virtual elements.
• Gaze Tracking: Specialized sensors monitor the user’s eye movements to determine focus and intent, allowing for more responsive interfaces and contextual interactions based on what the user is looking at.
• Audio Input: Built-in microphones capture voice commands and ambient sounds, enabling voice-activated controls and enhancing the immersive experience by integrating auditory context.
• Inertial Measurement Units (IMUs): Provide orientation and movement data that enhance motion tracking and overall responsiveness.
• Ambient Light Sensors: Adjust display brightness based on the surrounding light conditions, ensuring optimal visibility of digital content.
• Magnetometers: Help determine the device’s orientation relative to the Earth’s magnetic field, improving positional accuracy.

Mobile vs. Wearable AR

AR solutions generally fall into two primary categories: handheld AR using mobile devices and wearable AR using head-mounted displays or smart glasses. Each category serves different user needs and deployment contexts, ranging from consumer applications to industrial workflows.

• Mobile (Handheld) AR uses smartphones or tablets to superimpose digital content on live camera feeds. It’s widely used in applications like on-site equipment inspections, product visualization, remote assistance, and retail try-ons. The key benefits are low cost, broad accessibility, and ease of deployment—ideal for scenarios where quick interaction, portability, rapid deployment, and broad compatibility are essential.

• Wearable AR, by contrast, involves devices like HoloLens 2 or Magic Leap 2, which overlay 3D content directly into the user’s field of view. This enables more immersive experiences and hands-free workflows. It’s particularly beneficial for training simulations, real-time collaboration, 3D model walkthroughs, and assembly guidance, where continuous, hands-free interaction with spatially complex digital and physical environments is required.

Feature	Handheld AR	Wearable AR
Device Type	Smartphones, tablets	AR headsets, smart glasses
User Experience	View through a screen with limited field of view	Direct overlays in user’s line of sight with full spatial alignment
Immersion Level	Moderate – relies on camera feed for context	High – integrates digital content into physical space via SLAM and sensor fusion
Interactivity	Touch and screen gestures	Hand tracking, eye tracking, voice, and spatial gestures
Hardware Cost	Low – uses widely available consumer devices	High – requires specialized hardware
Development Tools	ARKit (iOS), ARCore (Android)	AR Foundation, MRTK, Magic Leap SDK, proprietary SDKs
Use Cases	Field inspections, guided procedures, remote support, product visualization	Design reviews, immersive training, assembly assistance, collaborative 3D planning

Choose mobile AR when speed, reach, and simplicity are key. Opt for wearable AR when tasks require spatial precision, persistent overlays, and continuous user engagement without interrupting workflow.

AR Headsets

Magic Leap 2

• A state-of-the-art see-through AR headset designed for enterprise and engineering applications.
• Uses high-resolution sensors and proprietary algorithms to achieve precise environmental mapping.
• Supports natural hand tracking, gesture controls, and gaze tracking, allowing users to interact with digital content in a fluid, natural manner.
• Powered by an external compute pack with an AMD 7nm Zen 2-based processor as its main controller.
• Includes additional sensors such as eye tracking cameras, inertial measurement unit (IMU), depth sensors, and ambient light sensors to enhance spatial awareness and user interaction.
• Particularly suited for design visualization, rapid prototyping, and collaborative projects where accurate spatial context is required.

Microsoft HoloLens 2

• A see-through AR headset that has been widely adopted for both commercial and research applications.
• Provides high-quality holograms with impressive spatial mapping capabilities, ensuring that virtual content is anchored accurately in the real world.
• Designed with comfort in mind, HoloLens 2 features an adjustable, lightweight headset that can be worn for extended periods.
• Incorporates eye tracking, hand tracking, and voice commands to enable natural and seamless user interactions.
• Extensively used in fields such as engineering, healthcare, and manufacturing for design visualization, remote assistance, and collaborative problem-solving.

Apple Vision Pro

• Represents the next generation of pass-through AR devices, integrating advanced display technologies with high-performance processing.
• Provides exceptional visual fidelity and contextual integration, ensuring that digital overlays are both clear and harmoniously blended with the physical world.
• Uses a combination of cameras, infrared sensors, and other tracking technologies to maintain precise alignment of digital content.
• Emphasizes natural interaction through gesture and voice commands, making it highly suitable for professional engineering and creative applications.

Varjo XR-4 Series

• High-end see-through AR headset targeting advanced industrial training, simulation, and design review.
• Dual 4K × 4K per-eye displays with a central foveated “focus zone” deliver near-retina sharpness where the user is looking.
• ~120° diagonal FOV provides immersive mixed-reality coverage.
• Dual 20-MP cameras enable photorealistic video see-through with depth reconstruction for precise occlusion.
• Built-in eye tracking; optional Ultraleap hand tracking; compatible with SteamVR Lighthouse or inside-out tracking for room-scale accuracy.
• Often paired with engineering-grade software (e.g., Autodesk VRED, Unreal/Unity) for flight, automotive, and medical simulation.

---

Why AR and Not Just VR?

VR excels at creating fully immersive environments for design reviews, safety training, and remote collaboration—especially when the real-world context isn’t accessible or needs to be simulated. However, AR offers something VR cannot: real-time insight and digital augmentation within the actual workspace. For engineers, technicians, and field workers, AR’s ability to overlay digital information on real machinery, environments, and people turns it into a critical tool—not a VR replacement, but a vital counterpart.

In-Situ Immersive Augmentation

AR anchors digital content onto real-world objects—offering live, context-sensitive data right where the work is happening. This level of situational awareness is impossible to replicate in VR. What AR delivers that VR can’t:

• Spatially-Aligned Information: Schematics, annotations, or sensor readouts appear directly on the equipment. This allows technicians to interpret and act on data without needing to mentally translate from a separate screen or memory.

• Live Visualizations: Real-time data (e.g., heat, stress, telemetry) overlays components under actual load. This enables condition-based decisions based on true operating parameters, not static models or assumptions.

• Environmental Adaptation: AR responds to real lighting, geometry, and occlusion. Overlays adjust dynamically to ensure visibility and precision, even in cluttered or unpredictable environments.

Real-World Decision-Making

AR enables engineers to make faster, smarter decisions on the shop floor, eliminating guesswork.

• Instant Data Access: Fault alerts, part specifications, or diagnostic readouts appear next to the actual hardware. This minimizes downtime by eliminating the need to consult separate systems or documentation.

• Live Monitoring: See live telemetry on a pump or valve while standing beside it. It empowers users to validate sensor readings and equipment behavior without leaving the workspace.

• Shared Views: Multiple team members view synchronized overlays on the same equipment—reducing errors and improving communication. This fosters alignment during diagnostics or procedures, reducing handoff mistakes and speeding collaboration.

On-the-Job Skill Development

AR combines real tools with digital guidance, helping trainees learn by doing—without compromising safety or efficiency.

• Procedural Guidance: Floating arrows and tooltips direct each step during a repair or calibration task. This reduces cognitive load for trainees and boosts retention by tying instructions directly to real-world actions.

• Real-Time Feedback: Systems alert the user when a bolt is over-torqued or an assembly step is skipped. These just-in-time corrections help reinforce best practices and reduce training errors early in the learning curve.

• Collaborative Training: Senior engineers can mentor side-by-side, both seeing the same digital overlays. It enables natural, face-to-face coaching with shared spatial references—something remote VR sessions can’t fully replicate.

Design & Prototyping “In Context”

Rather than reviewing a CAD model in a virtual room, AR allows engineers to test it on the factory floor—where it will actually be installed.

• Digital-Physical Fit Checks: Snap a virtual bracket to a real engine block to detect clashes or clearance issues. This allows for real-world validation before a single part is machined, catching issues that CAD alone may miss.

• Real-Time Design Adjustments: Tweak dimensions and instantly preview the results in the live workspace. Designers can iterate rapidly while receiving immediate spatial feedback from the actual environment.

• Error Prevention: Spot misalignments or ergonomic conflicts before manufacturing begins—saving time and tooling costs. AR bridges the gap between digital intent and physical execution, reducing late-stage surprises and rework.

Where AR Lags Behind VR

Despite its strengths, AR is still catching up to VR in terms of visual quality, stability, and ecosystem maturity.

AR Limitation	Engineering Impact	VR Advantage
Narrow Field-of-View	Users must turn their heads to view overlays; limits spatial awareness.	VR offers >100° FOV with full environmental control.
Limited Brightness	Holograms may become invisible in sunlight or industrial lighting.	VR scenes are unaffected by real-world light.
Tracking Instability	Reflective, low-texture, or vibrating surfaces cause anchor drift.	VR relies on fully virtual worlds—no drift.
Bulky Optics & Passthrough Constraints	Comfort and wearability suffer during long sessions.	Standalone VR is becoming lighter and more ergonomic.
Lower Compute Power	Mobile AR can struggle with dense meshes or VFX.	PC/console VR can render high-polygon scenes in real-time.
Fragmented Ecosystem	Different AR platforms = more testing and fewer reusable components.	VR has mature cross-platform standards (OpenXR, Unity XR).
High Device Cost & Limited Supply	Optical see-through AR remains expensive and niche.	Consumer VR headsets are affordable and mass-produced.
UX Overload Risks	Overlaying too much data can overwhelm users.	VR lets designers control every element of the scene and pacing.

Use AR for tasks grounded in the physical world—like inspections, assembly, prototyping, and spatial walkthroughs—where real-time data and context are critical. Use VR for immersive training, design visualization, and simulations in dangerous or remote environments where full control and visual fidelity are essential. Bottom line: AR and VR solve different problems. Together, they form a powerful workflow from real-world execution to immersive simulation.

---

Key Takeaways

AR enhances real-world environments by overlaying spatially aligned digital content, offering engineers and developers context-aware insights, real-time data, and intuitive interactions that VR alone cannot provide. Through Unity’s AR Foundation and related tools, creators can design cross-platform AR experiences that integrate features like plane detection, occlusion, and sensor fusion, supported by both handheld and head-mounted hardware such as HoloLens 2, Magic Leap 2, and Apple Vision Pro. A structured AR development lifecycle—from conceptual design to optimization—ensures that digital elements are robustly anchored, responsive, and contextually relevant, while platform-specific SDKs and hardware capabilities enable precise environmental understanding and user interaction. By blending physical and digital domains, AR empowers in-situ training, collaborative workflows, and real-world prototyping, making it a vital complement to VR in engineering and industrial applications.