E3. Mobile AR: Spatial Awareness

---

Learning Outcomes

• Explain how AR applications achieve spatial awareness. Before class, review how components like feature points, planes, meshes, occlusion, and lighting work together, and connect these ideas to real-world examples such as robot navigation.
• Describe the role of point clouds in AR tracking and anchoring. In preparation, read about how feature points form point clouds and see how they enable accurate spatial mapping in dynamic environments.
• Explain the purpose of plane detection in AR. Ahead of the session, focus on how identifying flat surfaces like floors and walls supports virtual object placement, and explore how plane visualization can be implemented.
• Differentiate between plane detection and meshing in AR. To get ready, study surface triangulation and mesh generation, noting how meshing supports obstacle detection and occlusion.
• Describe occlusion and its impact on AR realism. Prior to class, learn how depth textures allow virtual objects to be hidden behind real-world elements, increasing realism.
• Explain the function of environment probes in AR. Before the session, review how probes support dynamic lighting and reflections, and compare manual versus automatic probe placement.

---

What Is Spatial Awareness?

AR relies on accurate perception of the physical environment to convincingly blend virtual content with the real world. Spatial awareness is the set of techniques and data structures that enable an AR app to “see” and interpret its surroundings. By extracting and organizing information about points, surfaces, volumes, and lighting in the scene, spatial awareness enables:

• Stable Tracking and Anchoring: Feature points and point clouds provide the reference markers that keep virtual objects locked in place, even as the camera moves. Point clouds (via the ARPointCloudManager) capture and visualize these feature points, creating the underlying spatial map AR uses for robust pose estimation. Imagine a virtual quadruped robot in AR navigating a physical room. The robot can use feature points to stabilize its initial placement on the floor. You can visualize the point cloud to verify tracking quality under different lighting or surface conditions.

• Contextual Interaction: Plane detection makes it possible to place virtual furniture on a table, drop characters onto the floor, or snap UI elements to walls. The ARPlaneManager detects, tracks, and classifies horizontal and vertical surfaces—exposing lifecycle events and enabling precise tap-based object placement. The robot can navigate along detected horizontal planes (like the floor) and stop near vertical planes (like walls or cabinets), simulating obstacle-aware, autonomous navigation. Users tap to place the robot at a starting location, and plane detection ensures it’s properly grounded.

• Environmental Geometry and Physics: Meshing constructs a 3D representation of the scene’s surfaces, enabling collision detection, pathfinding, and custom physics interactions with real‑world geometry. The ARMeshManager generates and updates a spatial mesh in real time, which your app can use for physics simulation, raycasting, and advanced gameplay logic. Enable mesh reconstruction of the room to help the robot “see” obstacles like chairs or boxes. You can program it to stop or turn when detecting a mesh collision, demonstrating spatial awareness similar to SLAM (Simultaneous Localization and Mapping) in robotics.

• Realistic Rendering and Occlusion: Depth and stencil textures (from occlusion) allow virtual objects to hide correctly behind real-world objects, while environment probes capture lighting so digital content reflects and casts shadows coherently. The AROcclusionManager supplies per‑frame depth and stencil textures for accurate occlusion, while AREnvironmentProbe components inject real‑world illumination and reflection data to harmonize rendering. When the robot moves behind a couch or table in the room, occlusion ensures it disappears correctly from the camera’s view, enhancing realism. This helps users perceive the virtual robot as if it truly shares the space with physical objects.

• Adaptive Lighting and Reflections: Environment probes sample ambient light and reflections so virtual objects match the scene’s brightness, color temperature, and specular highlights. By placing AREnvironmentProbe components manually or automatically, your app gathers cubemap data to drive dynamic material adjustments for lifelike integration. Add environment probes in the room to dynamically light the robot based on its current location—bright by a window, dimmer near a wall. The robot’s reflective parts (like a metallic head or chassis) will reflect environmental lighting in real time.

---

Point Clouds

A point cloud in AR represents a dynamic set of feature points that an AR-enabled device identifies in its environment. These points serve as anchors for the device to understand its position and orientation in the real world. Unlike discrete trackable objects such as planes or images (introduced later), a point cloud is treated as a single trackable entity, although each point within it carries its own unique data. For example, to visualize a quadruped robot’s navigation, your app can visualize the point cloud of the physical room. This way, users can check whether the floor and nearby furniture provide enough stable feature points for reliable tracking—critical for the robot’s odometry and waypoint accuracy. By integrating point-cloud visualization and management into AR, you will directly observe how low-level perception data drives high-level robot behavior—mirroring real-world engineering workflows in mobile robotics, SLAM, and autonomous navigation.

Feature Points

A feature point is a pixel or small region in a camera image that stands out due to high contrast or distinct texture (e.g., the edge of a table, a knot in wood). AR subsystems detect these points because they remain visually stable across multiple frames. By matching feature points frame-to-frame, the AR system calculates how the camera has moved, enabling stable virtual content placement. Each feature point in ARPointCloud provides:

• Position (Vector3): The 3D coordinates of the point in session space relative to the XR Origin (i.e., the device camera). These positions form a sparse point cloud that helps the system understand surface geometry and improve scene reconstruction.

• Identifier (ulong, optional): A persistent ID that lets you track the same physical point across multiple frames. Not all platforms supply IDs. Check ARPointCloudManager.descriptor.supportsTrackableAttachments. Identifiers are useful when attaching content to specific, consistent points in the world—for example, placing a persistent label next to a machine component.

• Confidence (float, optional): A score between 0 and 1 indicating the subsystem’s certainty in the point’s stability. Use ARPointCloud.descriptor.supportsFeaturePointsConfidence to verify availability. In ARKit, confidence values help you filter out noisy or unreliable points when building more robust world meshes or occlusion effects. Filtering out low-confidence points improves performance and prevents unstable virtual object placement in dynamic or low-texture environments.

AR Point Cloud Manager

The ARPointCloudManager is responsible for creating, updating, and destroying ARPointCloud trackables in your Unity scene. It inherits from ARTrackableManager, which handles lifecycle events for AR data types.

• Point Cloud Prefab: This is a prefab that visualizes each detected feature point in the real world—typically as small spheres or particles. You can use this to give users a visual sense of how the device is mapping the environment. Developers often customize the prefab to vary the color or size of the points based on confidence or depth, enhancing debugging and user feedback during development.

• Detect Feature Points: A toggle that enables or disables the generation of point clouds from the camera feed. Disabling this can significantly improve performance, especially on lower-end devices. Expose Detect Feature Points in your in-app settings. If users disable it to boost performance, the robot disables autonomous rerouting and relies on manual waypoint taps instead. This feature is especially important in performance-sensitive scenarios such as real-time navigation or mobile multiplayer apps, where excessive tracking overhead can impact frame rates or battery life.

Implementing Point Clouds

Let’s see how to visualize point clouds in AR Foundation and use them to place an object on the detected floor: XFactory’s quadruped robot. We will use the Spot Animated quadruped robot from the XFactory virtual factory scenario. This robot will be spawned on the physical floor using real-time point cloud data captured by the device.

1. Create and Configure a New Scene:
   • Open the existing AR101 scene from Assets/Scenes.
   • Go to File > Save As and name the new scene World Understanding.
   • Save it in the same folder (Assets/Scenes/World Understanding or similar).
   • Remove any objects unrelated to basic AR setup. Keep only XR Origin, AR Session, and Main Camera.
   • Ensure the XR Origin has AR Camera Manager and Tracked Pose Driver on the Main Camera.
   • Ensure the AR Session has AR Session and AR Input Manager.

   

2. Create a Point Cloud Visualizer Prefab:
   • In Hierarchy, create an empty GameObject named PointCloudVisualizer. Reset its Transform.
   • Right click on PointCloudVisualizer and create a small Sphere (0.01–0.02m scale). Remove its Sphere Collider component.
   • Adjust its material and color.
   • Drag PointCloudVisualizer into Assets/Prefabs to save it as a prefab. Then remove it from the scene.

   

   Alternatively, you can use Unity’s default prefab for point clouds by right clicking in Hierarchy and selecting XR > AR Default Point Cloud.

3. Add ARPointCloudManager:
   • Select XR Origin.
   • Click Add Component > AR Point Cloud Manager.
   • In the Inspector, assign PointCloudVisualizer to the Point Cloud Prefab field.

   

4. Visualize Feature Points in Scene:
   • Create a script named PointCloudDisplay.cs.
   • Attach it to the XR Origin GameObject.
   • This script listens for updates from the AR Point Cloud Manager and visualizes individual feature points in the scene using the assigned prefab.

    using UnityEngine;
    using UnityEngine.XR.ARFoundation;
    using UnityEngine.XR.ARSubsystems;
    using System.Collections.Generic;
   
    [RequireComponent(typeof(ARPointCloudManager))]
    public class PointCloudDisplay : MonoBehaviour
    {
        ARPointCloudManager _manager;
        List<ARPointCloud> _tracked = new List<ARPointCloud>();
   
        void Awake() => _manager = GetComponent<ARPointCloudManager>();
        void OnEnable() => _manager.pointCloudsChanged += OnChanged;
        void OnDisable() => _manager.pointCloudsChanged -= OnChanged;
   
        void OnChanged(ARPointCloudChangedEventArgs evt)
        {
            foreach (var cloud in evt.updated)
                UpdateCloudVisualization(cloud);
        }
   
        void UpdateCloudVisualization(ARPointCloud cloud)
        {
            if (!cloud.positions.HasValue) return;
   
            var positions = cloud.positions.Value;
            var hasConfidence = cloud.confidenceValues.HasValue;
            var confidences = hasConfidence ? cloud.confidenceValues.Value : null;
   
            for (int i = 0; i < positions.Length; i++)
            {
                float confidence = hasConfidence ? confidences[i] : 1f;
                if (confidence < 0.3f) continue;
   
                Instantiate(
                    _manager.pointCloudPrefab,
                    positions[i],
                    Quaternion.identity,
                    transform
                );
            }
        }
    }
   

   Customize filtering behavior in the script. Change the confidence threshold (0.3f) to be more or less strict. Change marker scale or color in the prefab itself.

5. Place Quadruped Robot Using Point Cloud:
   • Create a script named RobotSpawner.cs.
   • Create an empty GameObject named Robot Manager in the scene and attach the script to it.
   • This script allows the user to tap the screen to spawn the Spot Animated robot onto the detected floor using point cloud data.

    using UnityEngine;
    using UnityEngine.XR.ARFoundation;
    using UnityEngine.XR.ARSubsystems;
    using Unity.Collections;
    using System.Linq;
   
    public class RobotSpawner : MonoBehaviour
    {
        public GameObject robotPrefab; // Drag Spot Animated prefab here
        public ARPointCloudManager pointCloudManager;
   
        GameObject spawnedRobot;
   
        void Update()
        {
            if (spawnedRobot == null &&
                Input.touchCount > 0 &&
                Input.GetTouch(0).phase == TouchPhase.Began)
            {
                foreach (var cloud in pointCloudManager.trackables)
                {
                    if (!cloud.positions.HasValue) continue;
   
                    var floorPoints = cloud.positions.Value
                        .Where(p => Mathf.Abs(p.y) < 0.1f) // Filter near y=0 (floor level)
                        .ToList();
   
                    if (floorPoints.Count == 0) continue;
   
                    Vector3 avg = Vector3.zero;
                    foreach (var p in floorPoints) avg += p;
                    avg /= floorPoints.Count;
   
                    spawnedRobot = Instantiate(
                        robotPrefab,
                        avg,
                        Quaternion.identity
                    );
                    break;
                }
            }
        }
    }
   

6. Configure the Script:
   • Drag Spot Animated prefab from Assets > XFactory > Prefabs > Robotics into the Robot Prefab field.
   • Drag XR Origin (which has ARPointCloudManager) into Point Cloud Manager.

   

7. Test on Device:
   • Deploy and run on device.
   • Slowly scan the floor to accumulate points.
   • Once the floor is well-populated with points, tap the screen to place the robot on the average point of the detected floor.

You can later extend this by integrating a navigation controller to make the robot move based on confidence-weighted point density and orientation.

---

Planes

In mathematics, a plane is a large, flat, two-dimensional surface. In AR, we approximate real-world flat surfaces—floors, tables, walls, tabletops—as finite planar patches. Plane detection is crucial because most AR experiences rely on anchoring virtual content to something the user can intuitively understand as a “surface.” For example, to ensure a realistic and immersive experience, our virtual quadruped robot should navigate only on horizontal planes (the floor) and automatically stop or reroute when it detects vertical planes (walls, cabinets, or bulky furniture). Plane detection therefore drives both its locomotion path and obstacle-avoidance logic. Plane detection offers several key benefits:

• Anchor Stability: By placing anchors on surfaces (e.g., the floor), virtual objects remain visually “glued” to the environment. This prevents objects from drifting or floating unrealistically as the user moves around, ensuring spatial consistency across the AR session.

• User Intuition: Tapping on a detected plane to spawn objects feels natural. Because users recognize the surface in both the real and augmented view, it reduces the learning curve and makes interactions more predictable and satisfying.

• Occlusion and Physics: Knowing where real-world surfaces lie helps with correct occlusion (e.g., virtual objects hide behind real tables) and physics interactions. This enables more immersive simulations, such as a ball rolling off a detected table edge or a robot stopping before a real wall it cannot pass through.

Features

• Feature Extraction: The AR runtime scans each camera frame for distinct, high-contrast “feature points” such as edges, corners, or textured regions that are easy to track over time. These points are the foundation for all subsequent spatial mapping—without enough features, plane detection accuracy suffers. Lighting quality, surface texture, and motion blur can all influence how many usable features are found.

• Depth and Motion: By monitoring how these feature points shift between frames, the AR system estimates camera movement and reconstructs a point cloud of the environment in 3D space. This semi-dense mapping is essential for understanding scene geometry, enabling virtual objects to remain in stable, realistic positions as the user moves.

• Plane Fitting: Using clustering and statistical methods such as random sample consensus (RANSAC), the subsystem identifies sets of coplanar points and fits a mathematical plane to them. This process filters out noise and ensures that detected planes correspond to real-world surfaces like floors, tabletops, or walls.

• Boundary Extraction: Once a plane is detected, the system projects all inlier points onto it and computes a 2D polygon representing the surface boundary. These boundaries help define the usable placement area for virtual objects and support features like path planning for AR robots or restricting interaction zones.

Optional Features

Many AR platforms extend plane detection with extra capabilities that improve placement precision and contextual awareness:

• Arbitrary Alignment: Supports detection of angled or sloped surfaces, such as ramps or pitched ceilings, instead of restricting results to purely horizontal or vertical planes. This expands AR use cases to include navigation on inclined walkways or object placement on tilted display stands.

• Classification: Assigns semantic labels to detected planes, such as “floor,” “table,” or “ceiling.” This allows applications to filter placement targets (e.g., only spawn furniture on tables) or enforce safety constraints (e.g., treat “wall” classification as an immovable barrier for navigation).

• Boundary Vertices: Provides the exact 2D polygon coordinates that define the plane’s boundary. These vertices are useful for accurate visual rendering (e.g., highlighting the detected surface) and for collision or navigation meshes that match the real-world footprint.

You can query support for these features at runtime via XRPlaneSubsystemDescriptor or ARPlaneManager.descriptor to adapt your app dynamically to the device’s capabilities. When classification is available, the robot filters planes to use only Floor for navigation and treats Wall classification as a hard stop boundary—even if the wall plane is only partially detected—preventing accidental collisions.

Lifecycle and State

Each plane also carries a trackingState (Tracking, Limited, None) to indicate confidence in its position and orientation (pose). It is essential to check for Tracking before placing or interacting with critical AR content—otherwise, virtual objects may jitter, drift, or appear misaligned. Tracking state changes often occur due to user movement, lighting shifts, or partial occlusion of the surface. AR planes typically pass through three lifecycle phases:

• Added: A new flat area is discovered and reported by the AR subsystem. This is the first time the plane is recognized, often with minimal size and shape information—ideal for triggering initial placement prompts.

• Updated: The plane’s boundaries, alignment, and pose are refined as more of the surface is observed from different angles. Updates can expand plane coverage, improve polygon accuracy, and merge overlapping detections into a single surface.

• Removed: The subsystem determines that the plane is no longer valid—perhaps because it was merged with another plane, the surface is no longer visible, or tracking confidence dropped to zero. Removal events should trigger cleanup of any dependent virtual content or logic tied to that surface.

Review this Unity documentation to learn more about the AR Plane Manager component and AR Plane component.

Implementing AR Planes

Let’s visualize horizontal and vertical AR planes and spawn and control the quadruped robot (Spot Animated) so it walks on horizontal floor planes and stops if it collides with walls. This illustrates how plane classification and plane boundaries can control robot behavior in AR.

1. Create a Plane Visualizer Prefab:
   • In the Hierarchy, right-click and choose XR > AR Default Plane.
   • Rename it to PlaneVisualizer.
   • Inspect its components: ARPlane tracks surface data, ARPlaneMeshVisualizer renders the mesh geometry, and MeshRenderer applies the material.
   • Create two new materials in Assets/Materials. Set FloorMaterial to light blue with ~50% alpha (A=120) and WallMaterial to Transparent red with ~50% alpha.

   

2. Create a Plane Material Setter Script:
   • Create a script named PlaneMaterialSetter.cs.
   • Attach it to the plane prefab.

    using UnityEngine;
    using UnityEngine.XR.ARFoundation;
    using UnityEngine.XR.ARSubsystems;
   
    [RequireComponent(typeof(ARPlane))]
    public class PlaneMaterialSetter : MonoBehaviour
    {
        public Material horizontalMaterial;
        public Material verticalMaterial;
   
        private ARPlane plane;
        private MeshRenderer renderer;
   
        void Awake()
        {
            plane = GetComponent<ARPlane>();
            renderer = GetComponent<MeshRenderer>();
        }
   
        void Start()
        {
            UpdateMaterial();
        }
   
        void UpdateMaterial()
        {
            switch (plane.alignment)
            {
                case PlaneAlignment.HorizontalUp:
                case PlaneAlignment.HorizontalDown:
                    renderer.material = horizontalMaterial;
                    break;
   
                case PlaneAlignment.Vertical:
                    renderer.material = verticalMaterial;
                    break;
   
                default:
                    renderer.material = horizontalMaterial; // fallback
                    break;
            }
        }
    }
   

3. Configure the Script:
   • In the Inspector, drag FloorMaterial to Horizontal Material and WallMaterial to Vertical Material.
   • Drag the PlaneVisualizer object into Assets/Prefabs to save it as a prefab. You can now delete it from the Hierarchy.

   

   With this setup, every detected surface will be colored appropriately: blue for floor planes and red for walls—without needing multiple prefabs.

4. Add AR Plane Detection:
   • Select XR Origin.
   • Add Component > AR Plane Manager.
   • Assign the PlaneVisualizer prefab you just created to the Plane Prefab field of the AR Plane Manager.
   • Set Detection Mode to Everything (horizontal and vertical).
   • Still on XR Origin, add ARRaycastManager. Leave the Raycast Prefab field empty. The ARRaycastManager allows you to cast rays from the screen into the AR scene to detect and interact with tracked features like planes or point clouds.
   • It might be a good idea to disable or remove the ARPointCloudManager (for point cloud visualization) now to reduce clutter. Plane meshes provide more structured visual feedback and will make the robot’s environment interactions easier to interpret.

   

5. Spawn Robot on Floor Only (Plane Classification):
   • Create a script ClassifiedRobotSpawner.cs and attach it to Robot Manager.
   • Disable or remove the previous RobotSpawner.cs component of Robot Manager to avoid duplicate robot spawning or logic conflicts.

    using UnityEngine;
    using UnityEngine.XR.ARFoundation;
    using UnityEngine.XR.ARSubsystems;
    using System.Collections.Generic;
   
    public class ClassifiedRobotSpawner : MonoBehaviour
    {
        public GameObject robotPrefab;
        public ARRaycastManager raycastManager;
        public ARPlaneManager planeManager;
   
        private GameObject spawnedRobot;
        private List<ARRaycastHit> hits = new List<ARRaycastHit>();
   
        void Update()
        {
            // Only process touch when there's no robot and a touch has just begun
            if (spawnedRobot != null ||
                Input.touchCount == 0 ||
                Input.GetTouch(0).phase != TouchPhase.Began)
                return;
   
            // Try raycasting against detected planes
            if (raycastManager.Raycast(
                    Input.GetTouch(0).position,
                    hits,
                    TrackableType.PlaneWithinPolygon))
            {
                var hit = hits[0];
                var plane = planeManager.GetPlane(hit.trackableId);
   
                if (plane == null)
                    return;
   
                // Only allow spawning on horizontal planes (typically the floor or table)
                if (plane.alignment == PlaneAlignment.HorizontalUp)
                {
                    spawnedRobot = Instantiate(
                        robotPrefab,
                        hit.pose.position,
                        Quaternion.identity
                    );
   
                    Debug.Log(
                        $"🤖 Robot spawned at: {hit.pose.position}"
                    );
                }
                else
                {
                    Debug.Log(
                        $"❌ Rejected plane — alignment was " +
                        $"{plane.alignment}, expected HorizontalUp."
                    );
                }
            }
        }
    }
   

6. Configure the Script:
   • Drag Spot Animated into Robot Prefab.
   • Drag XR Origin into both raycastManager and planeManager. This script ensures that the robot only spawns on horizontal planes (e.g., floor).

   

7. Add a Collider to the Plane Prefab:
   • Open your AR Plane Prefab (e.g., PlaneVisualizer) in the Inspector.
   • Disable its Mesh Collider component.
   • Add a Box Collider component.

   

8. Create a Script to Add Trigger Collider to Vertical Planes:
   • Attach the following script to AR Plane Prefab to resize the box.
   • Save the prefab. This will make vertical AR planes physically collidable in the scene.

    using UnityEngine;
    using UnityEngine.XR.ARFoundation;
    using UnityEngine.XR.ARSubsystems;
   
    [RequireComponent(typeof(BoxCollider), typeof(ARPlane))]
    public class VerticalPlaneTriggerCollider : MonoBehaviour
    {
        void Start()
        {
            var plane = GetComponent<ARPlane>();
            var box = GetComponent<BoxCollider>();
   
            if (plane.alignment == PlaneAlignment.Vertical)
            {
                // Collider spans the vertical plane
                box.isTrigger = true;
                box.size = new Vector3(plane.size.x, 2f, 0.02f);
                box.center = new Vector3(0, 1f, 0); // Elevate to match robot's height
                box.enabled = true;
            }
            else
            {
                box.enabled = false;
            }
        }
    }
   

9. Add a Collider to the Robot:
   • Select the Spot Animated prefab.
   • Add a Rigidbody component. Disable Use Gravity and Is Kinematic.
   • Add a BoxCollider, CapsuleCollider, or appropriate collider that fits the robot shape.

   

10. Create a Script to Stop the Robot on Wall Collision:
    • Create a new script called StopOnWallCollision.cs.
    • Attach it to the Spot Animated robot prefab. This script disables walking logic when the robot collides with a vertical AR plane.

    using UnityEngine;
    using UnityEngine.XR.ARFoundation;
    using UnityEngine.XR.ARSubsystems;
    
    public class StopOnWallCollision : MonoBehaviour
    {
        private Animator animator;
        private SpotWalker walker;
    
        void Start()
        {
            animator = GetComponent<Animator>();
            walker = GetComponent<SpotWalker>();
        }
    
        void OnTriggerEnter(Collider other)
        {
            // Check if the object we hit is an ARPlane
            ARPlane plane = other.GetComponent<ARPlane>();
            if (plane != null && plane.alignment == PlaneAlignment.Vertical)
            {
                Debug.Log("🚧 Robot triggered a vertical plane — stopping.");
                if (animator) animator.enabled = false;
                if (walker) walker.enabled = false;
            }
        }
    }
    

11. Test the Behavior:
    • Deploy your AR app to a device.
    • Walk around to scan your environment.
    • Tap to spawn the robot on the floor.
    • Place or walk it toward a detected vertical plane (e.g., wall).
    • When the robot touches the wall, it should stop moving and stop animating.

To improve performance, disable plane detection after initial placement for performance, cull small or off‑screen planes by subscribing to plane.boundaryChanged, and/or throttle mesh vertex count in custom visualizers to save GPU.

---

Meshing

Meshing in AR is the real-time construction of a polygonal surface model from raw depth data captured by a device’s sensors (e.g., LiDAR, depth camera, structured light). Rather than just detecting flat planes, meshing samples depth at many points to create a dense point cloud, triangulates those points into interconnected triangles, forming a continuous surface mesh, and outputs a Unity Mesh object you can render, collide against, or use for occlusion. For example, as you the virtual quadruped robot moves around a cluttered office or lab, the ARMeshManager continuously builds mesh chunks for chairs, desks, and boxes. The robot’s path-planning script can check these chunks every frame and stop the robot when an obstacle mesh collides with it—just like a Roomba avoiding sofa legs. The key benefits of meshing in AR include:

• Precision: Meshing captures curved and irregular surfaces, not just planes. This makes it possible to represent stairs, uneven terrain, and organic shapes with high fidelity, enabling realistic navigation, placement, and collision for virtual content.

• Interaction: Meshes let virtual objects rest naturally on real-world topology (e.g., a ball rolling down a real staircase). This improves immersion by ensuring physics interactions match the actual environment’s contours, whether for entertainment, training, or robotics applications.

• Occlusion and Anchoring: Meshes can hide virtual content behind real geometry or provide robust spatial anchors on complex shapes. This allows, for example, a holographic model to disappear correctly when it moves behind a real desk, or for a spatial marker to remain locked onto an irregular object rather than a flat plane.

Meshing is inherently platform-dependent: each XR SDK (ARKit, ARCore, Windows XR, etc.) has its own meshing pipeline, sensor requirements, and performance trade-offs. On LiDAR-enabled devices, updates can be nearly real-time, while camera-based depth reconstruction may be slower and less detailed. Always consult target device documentation for mesh density settings, refresh rates, and occlusion support.

Features

The ARMeshManager component in AR Foundation orchestrates platform meshing and exposes these tunable properties:

• Mesh Prefab: Every scan volume generates discrete mesh “chunks,” each instantiated from your Mesh Prefab. That prefab must include a MeshFilter component (required) that holds the generated Mesh data, a MeshRenderer + Material (optional) to visualize the scanned surface, and a MeshCollider (optional) to enable physical interactions against the real-world mesh. Choosing the right prefab setup depends on your application—lightweight, invisible colliders work best for navigation or occlusion logic, while visible, textured meshes are ideal for scanning or AR visualization apps.

• Density (0…1): Fraction of maximum tessellation; more triangles capture finer detail. The trade-off is that higher density leads to better surface fidelity but more CPU/GPU cost. Note that some runtimes ignore density and always run at their native resolution. In practice, density should be tuned for your use case—high for precision scanning or architectural apps, low for mobile AR games or robotics where performance is critical.

• Vertex Attributes: During mesh construction the device can compute extra per-vertex data. Disable anything you won’t need. Normals are surface orientation vectors, required for correct lighting and shading. Tangents are vectors orthogonal to normals, used by normal-mapped shaders. Texture Coordinates (UVs) are used for mapping textures onto the mesh surface. Colors can be utilized for per-vertex depth or confidence coloring. Reducing unused attributes can significantly improve performance, especially on mobile devices, while still allowing advanced rendering effects when needed.

• Concurrent Queue Size: To keep your main thread smooth, ARMeshManager offloads converting raw sensor mesh data → Unity Mesh and generating collision meshes if you have a MeshCollider. Increasing this queue size can reduce visible mesh update latency, but may increase memory usage; conversely, smaller queues minimize memory overhead at the expense of slower mesh refresh rates.

Meshing Best Practices

• Optimization: Lower density and disable unused attributes in performance-sensitive apps. This reduces CPU/GPU workload, improves frame rates, and helps maintain tracking stability—especially on mobile devices where thermal throttling can occur during long sessions.

• Chunk Size: Some platforms let you configure how large each mesh section is—smaller chunks mean faster updates but more overhead. Smaller chunks are ideal for dynamic environments or robotics navigation where rapid mesh changes matter, while larger chunks are better for static scenes to reduce draw calls and memory use.

• Material Choice: Wireframe or semi-transparent shaders help debug mesh coverage. These visualization techniques make it easier to spot gaps, overlapping geometry, or poorly scanned areas during development, even if the mesh will be hidden in the final application.

• Persistence: For mapping applications, consider serializing meshes to disk and re-loading them in later sessions. Persistent meshes allow for large-scale mapping over time, enabling use cases like multi-session scanning, AR indoor navigation, or collaborative construction planning.

Implementing AR Meshes

Now, let’s extend the previous quadruped robot navigation example by enabling AR meshing, visualizing scanned surfaces in real time, and stopping the robot when it collides with mesh-reconstructed objects (e.g., furniture, boxes).

1. Create a Mesh Prefab:
   • In the Hierarchy, right-click and select Create Empty. Name it ARMeshPrefab.
   • Select ARMeshPrefab.
   • Add a MeshFilter component. Leave the Mesh field empty. ARMeshManager automatically assigns the mesh data at runtime to the MeshFilter of each mesh chunk it generates.
   • Add a MeshRenderer component for visualization (optional). Assign a semi-transparent material.
   • Add a MeshCollider component, required for physics interaction. Check Convex and Is Trigger.
   • Drag ARMeshPrefab into your Project window to save it as a prefab. Remove it from the Hierarchy. This prefab will be instantiated dynamically for each mesh “chunk” generated by the device in real time.

   

2. Add ARMeshManager to Your Scene:
   • Select your XR Origin, right-click and create an empty GameObject. Name it Meshing.
   • Select Meshing, then click Add Component > ARMeshManager.
   • Go to the Inspector.
   • Assign your ARMeshPrefab to the Mesh Prefab field.
   • Set Density to 0.5 (adjustable based on scan detail/performance).
   • Enable Normals.
   • Optionally, enable Tangents, Texture Coordinates, Colors
   • Leave Concurrent Queue Size at default or set to 4–8.
   • (Optional) You can use a button to toggle the visibility of the mesh in the scene (e.g., meshRenderer.enabled = false) to reduce clutter during robot operation.

   

3. Stop the Robot When It Hits a Mesh:
   • Create a new script called StopOnMeshCollision.cs.
   • Attach it to your robot (i.e., Spot Animated prefab). This script stops the robot when it bumps into an AR mesh chunk representing a real-world obstacle—such as a scanned chair, box, or wall—while ignoring mesh surfaces near the floor.

    using UnityEngine;
   
    public class StopOnMeshCollision : MonoBehaviour
    {
        private Animator animator;
        private SpotWalker walker;
        private Collider robotCollider;
   
        // How much above the robot base a collision must occur to be considered valid
        private const float minCollisionHeight = 0.1f;
   
        void Start()
        {
            animator = GetComponent<Animator>();
            walker = GetComponent<SpotWalker>();
            robotCollider = GetComponent<Collider>();
        }
   
        void OnTriggerEnter(Collider other)
        {
            if (!other.TryGetComponent<MeshCollider>(out var meshCollider) ||
                !other.TryGetComponent<MeshFilter>(out var meshFilter))
                return;
   
            // Get the closest point of contact on the mesh relative to the robot
            Vector3 contactPoint = other.ClosestPoint(transform.position);
   
            // Compare vertical distance to base of robot
            float contactHeight = contactPoint.y;
            float robotBaseHeight = robotCollider.bounds.min.y;
   
            if (contactHeight - robotBaseHeight < minCollisionHeight)
            {
                Debug.Log("🟢 Mesh contact below robot — ignoring.");
                return;
            }
   
            Debug.Log("🟡 Robot collided with mesh above floor — stopping.");
            if (animator) animator.enabled = false;
            if (walker) walker.enabled = false;
        }
    }
   

4. Configure the Script:
   • Ensure the Spot Animated robot has a BoxCollider or CapsuleCollider to the Spot Animated robot, as well as a Rigidbody with Is Kinematic and Use Gravity disabled.
   • Keep StopOnWallCollision.cs if you still want the robot to stop at both vertical planes and scanned mesh obstacles.

   

5. Test the Behavior:
   • Deploy the app.
   • Tap to spawn the robot on a horizontal plane.
   • As the robot walks, ARMeshManager continuously generates 3D surface meshes.
   • If the robot touches one of those mesh chunks (e.g., a chair), it immediately stops.

---

Occlusion

Occlusion ensures virtual content in an AR scene is correctly hidden behind real-world objects based on depth. By comparing the per-pixel distance of real surfaces (from a depth image) against the virtual geometry’s depth, the renderer discards pixels of virtual objects that lie “behind” real ones—making digital elements appear truly embedded in the environment. For example, as the quadruped robot patrols the room, it vanishes when it walks behind an object (e.g., a furniture) and reappears when it emerges—exactly how a physical robot would disappear from view. This seamless hide-and-reveal dramatically increases the believability of the robot’s presence. Key benefits of occlusion in AR include:

• Immersion and Believability: Virtual objects that pass behind chairs, people, or walls reinforce depth cues, making experiences more convincing. This mirrors real-world visual behavior, helping users accept virtual elements as part of their actual surroundings instead of floating overlays detached from reality.

• Visual Comfort: Incorrect layering (“AR always on top”) breaks expectations and can cause eye strain or motion sickness over prolonged use. Proper occlusion aligns virtual depth with real-world depth, reducing cognitive dissonance and making spatial transitions—like an object emerging from behind furniture—feel natural.

• Depth Perception Integrity: Without occlusion, the robot would float in front of the coffee table even when logically behind it, breaking depth perception and confusing users about where the robot actually is. Maintaining accurate occlusion prevents spatial mismatches that could hinder usability in navigation, training, or safety-critical AR workflows.

• Interaction Accuracy: Occlusion ensures that interactions between virtual and real objects appear consistent, such as a virtual ball bouncing behind a couch rather than unrealistically clipping through it. This is essential in mixed-reality simulations, AR gaming, and robotics scenarios where spatial accuracy affects task success.

• Safety and Usability: In industrial or navigation-based AR, occlusion can prevent misleading overlays that might hide real hazards or misrepresent spatial layouts. For example, keeping a machinery warning label visible instead of letting virtual content obscure it supports safer operations.

Review this Unity documentation to learn more about the AR Occlusion Manager component.

Depth and Stencil Pipeline

Modern AR devices provide real-time depth and stencil textures to power occlusion effects. These textures allow Unity to determine whether virtual objects—like your robot—should be visible or hidden behind real-world geometry or people. Each frame, platforms like ARKit (structured light), ARCore (time-of-flight or ML), or XR Simulation (synthetic) generate:

• Environment Depth Texture: A grayscale image where each pixel encodes the distance (in meters) from the camera to the nearest surface. This texture is the backbone of environment occlusion, letting the renderer make per-pixel comparisons between virtual object depth and real-world surfaces. Developers can sample this texture in custom shaders to fine-tune occlusion thresholds or blend virtual objects with physical geometry for advanced effects like depth-aware shadows.

• Depth Confidence Texture: A per-pixel confidence score (range: 0 to 1). High confidence in well-lit, textured scenes; low in dark or flat areas. Useful for diagnosing tracking instability, this texture can also be leveraged to adapt occlusion quality dynamically—reducing visual artifacts by fading or softening occlusion in low-confidence regions instead of producing sharp, incorrect cutouts.

• Human Stencil Texture: A binary mask identifying human silhouettes in the camera view. This mask enables human occlusion, ensuring virtual objects appear behind people but still visible through gaps like arm motions or between fingers. It’s especially valuable in collaborative AR scenarios where accurate layering of people and content maintains spatial believability.

Toggle a debug overlay of the environment depth texture during development to visualize occlusion coverage. Areas with missing depth (e.g., shiny tables or shadows) correlate with moments when occlusion may fail. By pairing this with the depth confidence overlay, you can quickly pinpoint whether failures are due to sensor blind spots or environmental conditions.

Occlusion Pipeline

The occlusion pipeline helps you optimize occlusion accuracy and debug subtle issues in real-world conditions—critical for immersive XR engineering apps. Occlusion in Unity flows through a consistent pipeline:

1. Capture: The AR platform (e.g., ARKit, ARCore) provides synchronized RGB, depth, and stencil data each frame.
2. AROcclusionManager: Attached to your XR Origin, this component interfaces with the underlying XROcclusionSubsystem. It exposes depth/stencil textures via properties:
   • environmentDepthTexture
   • depthConfidenceTexture
   • humanStencilTexture
   • humanDepthTexture (if enabled)
3. Rendering: On mobile, the AR Camera Background component uses built-in shaders to merge color and depth textures and discard fragments behind real-world surfaces.

Implementing AR Occlusion

Now that the quadruped robot can navigate the environment using AR planes and meshes, let’s enhance realism by adding occlusion—making the robot visually disappear behind real-world objects using the device’s depth camera. As the robot walks behind a desk, chair, or your hand, it should visually disappear (fully or partially), just like a real object would. This creates a more immersive AR experience. Here is how to implement this behavior:

1. Add AR Occlusion Manager:
   • Select the Main Camera under XR Origin > Camera Offset.
   • Click Add Component > AR Occlusion Manager. Note that the AR Occlusion Manager must be attached to the Main Camera rather than the XR Origin because occlusion functionality is fundamentally tied to the camera’s view and depth information.
   • Go to the Inspector.
   • Set Environment Depth Mode to Best (automatically falls back if unsupported).
   • Enable Temporal Smoothing to reduce visual jitter.
   • Set Human Segmentation Stencil Mode to Fastest (optional, if needed for people occlusion).
   • Set Human Segmentation Depth Mode to Fastest.
   • Set Occlusion Preference Mode to Prefer Environment Occlusion.

   

2. Hide Plane and Mesh Manager Visuals (Optional):
   • Open your Plane Prefab (e.g., PlaneVisualizer).
   • Disable or remove the MeshRenderer or assign a fully transparent material to the plane mesh.
   • Open your Mesh Prefab (e.g., ARMeshPrefab).
   • Remove or disable the MeshRenderer component, or assign a transparent or invisible material for runtime use.

   

3. Test the Behavior:
   • Build the app to your device. Tap to spawn the robot.
   • The robot disappears partially or fully as it moves behind the object.
   • The effect adjusts dynamically as you move the camera or real-world object.
   • Try switching between Fastest, Medium, and Best depth modes to compare occlusion quality.

Occlusion brings your XR engineering simulation one step closer to real-world realism. It reinforces spatial awareness and provides users with natural visual feedback as the robot navigates the physical space.

---

Environment Probes

Environment probes are specialized “cameras in a box” that sample the real world at a point in space and bake that information into a cubemap—six square textures covering all directions. When you apply this cubemap to virtual materials (e.g. reflective metal or glass), those objects appear to mirror and respond to actual surroundings! For example, the quadruped robot features a polished aluminum shell and glossy sensor domes. By sampling the room with environment probes, those surfaces pick up real reflections—like the nearby window highlights—making the robot look physically present in your workspace. The key benefits of environment probes include:

• Accurate reflections on shiny surfaces: Virtual metals, glass, and water can mirror real-world elements, allowing your AR scene to match environmental realism instead of relying on pre-baked or generic reflection maps.

• Real-time lighting adaptation: As the user moves or lighting conditions change—such as blinds opening or a lamp turning on—the cubemap updates to keep virtual materials consistent with the actual scene illumination.

• Seamless blending between real and virtual: Correct environmental reflections help remove the “sticker” effect, making digital content feel like it shares the same physical space and light as the real environment.

Environment probes can be set to automatic placement by the AR system or manually placed in strategic spots for optimized visual quality. Manually positioning them near high-reflectivity objects (like your robot) can greatly improve realism without excessive probe updates.

Features

• Transform and Coordinate Space: Defines where and how the environment probe samples lighting and reflections in the AR world:
  • Position: Where in the AR world the probe samples, crucial for local lighting accuracy—shadows and highlights shift logically as you move. For reflective robots or objects, place probes at relevant eye or object height to match the expected viewing perspective.
  • Orientation: Which way “forward” is. AR frameworks may sample at arbitrary rotations before re-aligning to world axes.
  • Scale: Typically uniform; scales the virtual cubemap if you want to exaggerate or compress the perceived environment.
• Bounding Volume and Influence: Determines how far a probe’s captured lighting and reflections extend in the scene:
  • Infinite (Global) Volume: Environment texture applies everywhere—ideal for outdoor skies or distant cityscapes.
  • Finite (Local) Volume: Defines a box (e.g., 3 m × 2 m × 3 m) around the probe. Only objects inside this box use that probe’s data. Matching the volume to an area of interest ensures lighting changes feel natural when moving between zones.
• Placement Strategies: Choosing where and how to position probes has a major impact on realism:
  • Manual Placement: Place a probe exactly where a critical virtual object sits. Best for static scenes, architectural previews, or product demonstrations.
  • Automatic Placement: AR framework drops probes at likely “feature-rich” spots—corners, edges, bright areas. Best for rapid prototyping or broad coverage in dynamic spaces.
  • Hybrid Approach: Start with automatic probes, then add manual ones in problem areas—under a table, inside a display case, or alongside a moving object.

• Manager Component and Workflow: The AR Environment Probe Manager is the central hub that tracks, adds, updates, and removes probes. It exposes parameters like Maximum Number of Probes, Detection Interval, and Cubemap Resolution. Debugging often involves attaching a visible mesh (e.g., wireframe cube) to represent each probe, making it easier to see coverage and placement patterns during development.

• Texture Filtering: Controls how cubemap textures are sampled and displayed, affecting the smoothness and accuracy of reflections:
  • FilterMode.Point: Sharp edges, pixel-perfect sampling—rarely used for smooth reflections.
  • FilterMode.Bilinear: Smooth interpolation between faces—good compromise for quality and performance.
  • FilterMode.Trilinear: Adds mipmapping transitions—best for handling reflections at varying distances and minimizing visual artifacts.

Adding AR Environment Probe

1. In the Hierarchy, select your XR Origin GameObject.
2. Click Add Component > AR Environment Probe Manager.
3. In the Inspector, configure the following settings:
   • Automatic Placement: Enable this to let Unity automatically place environment probes throughout the scene based on visible surfaces and lighting.
   • Environment Texture Filter Mode: Choose from Point (fastest, lowest quality), Bilinear (balanced), or Trilinear (smoothest lighting transitions but higher GPU cost).
   • Environment Texture HDR: Enable this for more realistic lighting using high dynamic range cubemaps.
   • Debug Prefab: Optionally, assign a small visual prefab (like a semi-transparent sphere) to see where probes are instantiated during runtime.

   

Let your robot roam near bright windows, reflective floors, or shadowy corners. As it moves, new environment probes are generated dynamically. Reflections and lighting on the robot’s body shift in real time, demonstrating live environment mapping in AR.

---

Key Takeaways

Spatial awareness in mobile AR combines multiple technologies—point clouds, plane detection, meshing, occlusion, and environment probes—to let virtual content interact believably with the real world. Point clouds map stable feature points for accurate tracking and anchoring, while plane detection enables intuitive placement and navigation by identifying flat surfaces. Meshing adds detailed geometry for realistic collisions and pathfinding, and occlusion ensures proper visual layering so virtual objects hide naturally behind real ones. Environment probes complete the illusion by adapting lighting and reflections to match the physical space. Together, these systems create AR experiences where virtual elements not only stay locked in place but also respond to their surroundings as if they truly exist in them.