What Is Spatial Computing?

Spatial computing is an umbrella term for digital experiences that reference objects and locations in the physical world. It includes augmented reality (AR), mixed reality (MR) and virtual reality (VR) recreations of real-world places.

Although the term “spatial computing” has been around for decades, it gained traction following the 2023 announcement of the Apple Vision Pro, a headset with AR, VR and mixed reality capabilities that Apple describes as its first spatial computer.

Creating a device that effectively blends the real environment with virtual elements is a big challenge from a technical perspective. But the use cases — like a four-monitor workspace that packs away in a drawer or an immersive video call that doesn’t fully block out your surroundings — align more closely with how most people want to engage with their devices.

Read MoreEverything You Need to Know About Extended Reality

How Does Spatial Computing Work?

Spatial computing devices use data captured by cameras and other sensors to map the user’s surrounding area. While a headset like the Apple Vision Pro uses 12 cameras, five sensors and a specialized chip to achieve its mixed reality effect, a smartphone might only use its front-facing camera or a combination of a camera and LiDAR for spatial mapping.

That data is then typically fed into an algorithm, or several algorithms, to identify the shape of the objects around the user. More advanced devices may also use image recognition to classify the objects in the field of view.

Once the device has a spatial map and understanding of the objects within a space, it can superimpose virtual objects that blend with the physical environment — like a preview of a new couch in your living room, or a virtual animal racing up and down your hallway.

Examples of Spatial Computing

Spatial computing covers all technologies that fall under extended reality (XR) — an umbrella term that refers to any technology combining elements of the physical and digital world to design an immersive user experience. Below are some examples of how spatial computing is used in different types of XR. 

Augmented Reality

Augmented reality refers to technology that projects digital objects onto a user’s physical environment. This is meant to enhance a user’s interactions with their physical surroundings instead of replacing them with a virtual environment. 

An example of AR is IKEA’s IKEA Place platform. The app allows users to overlay furniture in their home space, generating actual-size models and even capturing details like how light and shadows fall on a virtual object. 

Virtual Reality

Virtual reality creates an interactive simulation with goggles, headsets, gloves and other devices. Users can immerse themselves in this digital environment, which is intended to replace their physical surroundings. 

Launched in 2023, Sony’s PlayStation VR2 is a well-known case of VR in action. The PlayStation VR2 uses eye tracking, 3D audio, haptic feedback and finger touch detection to engage users’ senses as part of the VR experience.

Mixed Reality

Mixed reality combines VR and AR. Rather than replace or enhance one’s physical surroundings, MR lets a user see the real world while still interacting with virtual objects using their body. A user can then simultaneously engage with both the physical and digital world.

The Meta Quest 3S is the result of Meta refining its MR product line. The headset comes with improved color contrast, upgraded spatial audio and precision controllers that enable users to engage with both physical and digital environments.  

Digital Twins

A digital twin is a digital representation of a physical object that changes alongside the object throughout its lifetime. The digital twin uses sensors attached to the object to reflect changes in real time, ensuring it remains an accurate model of that object. 

Ansys offers digital twin builder and AI-based digital twin tools. Both tools are intended to accelerate the development of digital twins, which can be used for conducting predictive maintenance, reducing costs and supporting other business needs.

Spatial Computing Applications

Manufacturing

Spatial computing can be used in a manufacturing environment to provide three-dimensional, step-by-step instructions for a specific process, or allow a worker to call a supervisor who can see what they’re doing and provide guidance through sketches and diagrams. (An early demonstration of the Microsoft HoloLens 2 for The Verge placed the reporter in front of an ATV with a missing bolt, providing holographic instructions for fixing it.)

Spatial computing can also be used to display relevant data about the surrounding environment, like a machine’s temperature or how long an automated process will take to complete. These applications become particularly useful when presented through an AR or mixed reality headset, since this form factor frees up the wearer’s hands. 

Logistics

Spatial computing can help warehouse workers track to-do lists, navigate warehouses and identify items on a shelf. It can also be used for safety purposes, like alerting wearers if multiple people — or forklifts — are approaching the same corner. 

Medical

AR and mixed reality headsets can provide checklists, instructions and other contextual information to healthcare professionals in a hands-free format. VR and AR technologies can also produce 3D renderings of organs, allowing medical professionals to get the training they need before performing surgeries. 

Entertainment

Home theaters have been a popular virtual reality application for a while, with streaming services like Amazon Prime Video and Netflix offering VR versions of their apps. Mixed reality headsets can take this concept to the next level, allowing the user to project a large screen in their living room while still being able to see what’s going on around them. And a sports fan can watch multiple full-screen games at once. 

Games

Games like Pokémon Go and its predecessor, Ingress, use GPS and augmented reality technology to create gaming experiences that are specific to the user’s location and incorporate aspects of the surrounding environment. Given the recent advances in spatial computing, it seems likely that we’ll see a wider range of games that employ the technology in novel ways in years to come. 

Fitness

Fitness applications like Supernatural and BeatSaber have been popular among early VR adopters. We might also soon see mixed reality fitness applications that incorporate free weights and other equipment that’s difficult to handle without looking at them. High-end headsets can also track the wearer’s hand movements, eliminating the need for handheld controllers and making possible a wider range of exercises. 

Productivity

With headsets like the Apple Vision Pro, users can surround themselves with large virtual screens, simulating the multi-monitor battlestations favored by many software developers. Unlike a real four-monitor setup, however, a virtual workspace can be set up instantaneously in a coffee shop or at a dinner table and packed away in a drawer or carry-on case. 

Telepresence

Spatial computing can be used for enhanced video calls that make it feel like the other person is in the room with you.

Read More5 Ways Using VR Technology in Business Can Improve Your Operations

The History of Spatial Computing

Attempts to create VR headsets go as far back as 1968 when Ivan Sutherland led this initial effort. NASA would develop something similar in the form of a virtual workstation in 1988. However, the term “spatial computing” didn’t exist until it was coined in 2003 by MIT graduate researcher Simon Greenwold, whose early work in the field involved a combination of early augmented reality prototypes, input devices that allowed users to control computers through real-world actions, and a cheap 3D scanner.

In the years since, spatial computing has shown up in a number of ways:

• 2005: Google launched a mobile version of Google Maps. Although different from the AR and mixed reality applications that often come to mind when hearing the term, it is perhaps the foremost example of spatial computing: a continuously updated digital model of the physical world that tracks the user’s place within it.
• 2006: Israeli startup PrimeSense showed off a depth-sensing device that would allow users to control video games through gestures without the use of a physical controller. PrimeSense partnered with Microsoft to create Kinect, an Xbox 360 accessory that aimed to capitalize on the popularity of motion-controlled games launched by the Nintendo Wii.
• 2013: Apple acquired PrimeSense, with co-founder and CTO Alexander Shpunt joining the company as a distinguished engineer. Shpunt would later file dozens of patents related to Apple’s emerging work in spatial computing.
• 2015: Microsoft announced the first version of its HoloLens mixed reality headset, which allowed users to place virtual items in a room thanks to spatial mapping technology.
• 2016: Niantic launched Pokémon Go, an augmented reality game that encouraged users to move around in the real world to catch and train virtual monsters. The game earned a world record $206.5 million in its first month since launch and was downloaded 130 million times during that period.
• 2019: Microsoft announced a new version of the HoloLens, with an increased focus on enterprise applications. Key improvements included a larger field of view and brighter, higher-resolution images that make holograms feel more realistic.
• 2020: Apple added LiDAR technology to its 4th generation iPad Pro and iPhone 12 Pro to improve each device’s augmented reality capabilities.
• 2023: Apple unveiled the Apple Vision Pro, marketing the device as its “first spatial computer.”
• 2024: Apple enhanced the Apple Vision Pro with its visionOS 2 update, introducing features like spatial photos, spatial videos and spatial personas. 

Differences Between AR, VR and Spatial Computing

Augmented reality and mixed reality, both of which superimpose digital elements onto the physical world in real time, are common use cases for spatial computing.

Virtual reality, which places the user in an immersive, simulated environment — normally through the use of a virtual reality headset — can employ spatial computing, but doesn’t always. A VR simulation of a real-world place, or of a virtual space that contains real-world objects, would be considered spatial computing under Greenwold’s definition. But an entirely fictional VR world would not.

Read MoreA Guide to the Metaverse and VR Tech: What You Need to Know

The Future of Spatial Computing

Greenwold, whose first augmented reality application predates Pokémon Go by 15 years, said the field has progressed much slower than he and his colleagues at the MIT lab expected when he was working on his thesis 20 years ago.

But he believes we may be approaching an inflection point due to a number of recent technological advancements. 

Higher-Quality Graphics

For one, pixel density is important for any device that’s held up close or worn as a headset, as lower resolutions increase eye strain and break immersion. (The Vision Pro’s postage-stamp-sized displays have a resolution equivalent to a 4k display for each eye.) Improved sensors, more powerful chips and advances in machine learning have also been essential to improving graphical fidelity and making interactions between virtual elements, users and real-world objects feel more natural.

Advancements in Artificial Intelligence 

Greenwold also sees the rise of generative AI as a critical development for the field, as it can dramatically reduce the cost of animating virtual objects and environments, making new experiences cheaper and more ubiquitous. Multimodal AI could also make it easier to harness various types of data, including spatial data, and feed them into visual systems.

Expanded Capabilities of the Apple Vision Pro

Greenwold is also optimistic that the direction Apple is taking with the Vision Pro might push the field forward in a way other devices haven’t — thanks to the decision to make a device that can do all the things people might want to do with a mixed reality headset, rather than make substantial sacrifices to bring the price point low enough for ordinary consumers.

“They’re not going to sell very many of these for $3,500, but it’s not going to be a flop because they’re not expecting to,” Greenwold said. “What they’re going to do is make something that’s amazing and inaccessible, but over time they will bring the cost down at some point to where people can get something they couldn’t have before.”

Frequently Asked Questions