Apple packed a lot of hardware into its new Vision Pro headset, including high-resolution optics, more than a dozen sensors and a pair of Apple-designed processors. Here’s a look at the technology that makes the new augmented reality device tick.

The Vision Pro’s operating system software, called VisionOS, runs on the same M2 processor that Apple uses in its Macs, the company said at its WWDC developer conference Monday. But to handle all the sensor data, Apple designed a companion chip, the R1.

“It processes input from 12 cameras, five sensors and six microphones,” said Mike Rockwell, head of Apple’s Technology Development Group. Handling that rapidly cuts the latency between head movement and the corresponding change in the display’s field of view down to just 12 milliseconds, about an eighth the duration of an eyeblink. Low display latency is essential to avoiding nausea that some VR and AR headset wearers suffer.

The premium electronics come at a cost. The Vision Pro will cost $3,499 (around £2,815 or AU$5,290 converted) when it ships in early 2024, Apple said.

But with the Vision Pro, Apple is betting a premium experience is necessary to overcome the obstacles that have kept VR and AR headsets out of the mainstream so far. Budget headsets from Meta and other competitors so far have failed to attract many people to virtual- and augmented-reality promises like immersive gaming and video, web browsing and word processing with giant virtual computer screens, and videoconferencing with life-size companions.

Watch this: Everything Apple Announced at WWDC 2023

The price tag won’t get you everything, however. Though most of the headset’s electronics are built in, it’s tethered with a cord to an external battery pack.

Vision Pro optics and displays

One major component of the experience is the display. VR headsets with a narrow field of view and obviously pixelated imagery are limited, but Apple is trying to offer something more immersive.

The Vision Pro has two postage-stamp-size displays, one for each eye. But a trio of lenses stretches them to span a wider field of view while still preserving sharpness and color.

“It enables video to be rendered in true 4K resolution with wide color and high dynamic range, all at massive scale,” Rockwell said. “Fine text looks super sharp from any angle.”

The Reality Pro has a conventional camera that will let you take photos. But it’s also got others to track what’s around you for augmented reality views of your home, office or airplane seat. And downward-pointing cameras track your hands for finger gestures that let you select objects and push buttons.

To figure out which parts of the screen you’re focusing on, eye-tracking hardware inside the headset follows your gaze, illuminating your pupils with infrared light. The eye-tracking information isn’t shared with website or app developers who prize such information, since it can reveal what’s most interesting or important on a screen.

Although the Vision Pro is opaque, it’s got a separate outward-pointing display that shows your eyes to people looking at you when you’re wearing the headset. The feature, called EyeSight, also can show a little about what you’re up to, adding a blue tint when you’re using an app or flashing white when you take a photo.

It’s all very elaborate, very expensive, and not likely to sell in anything like the quantity of smart watches, much less smartphones. But in the electronics business, costs often come down as technology expertise and manufacturing volume go up. Apple Chief Executive Tim Cook called the Vision Pro the “beginning of a journey.” Perhaps a possible Apple Vision non-Pro or the Apple Vision 2 Pro will appeal to more buyers.