Ryuichi Sakamoto is hunched over a grand piano, wearing a black suit and his customary tortoiseshell eyeglasses. The room around him is drenched in darkness, with no other textures visible but the freckles on his face and the glossy shine of the Yamaha. And then he lifts his hands and lets them fall back on the keys. They start to sing under his touch.

The performance lasts about 45 minutes. The Oscar-winning composer plays 10 of his own compositions, and at the end of every piece, people clap. Not me at first, but I make up for it with tears, which first build up in my throat, then stream down my cheeks. When it's over, Sakamoto doesn't bow and walk off the stage; he passed away in March this year after a years-long battle with cancer. He just fades into black as I take off the headset I am wearing.

Kagami, as the music performance is called, is a mixed-reality show on view through July 9 at The Shed in Manhattan and at Factory International in Manchester, with runs at the Sydney Opera House and the Big Ears Festival in Knoxville, Tennessee, up next. The show begins with 80 people sitting in a circle around absolutely nothing. After each concertgoer has slid on a Magic Leap 2 headset, a virtual Sakamoto appears in the center of the circle. The musician then performs as you mosey “around” him, being ever so careful not to bump into a piano that doesn’t exist—or your fellow attendees, who do.

Kagami—Japanese for “mirror”—was designed by the production studio Tin Drum, and fuses a 3D model of Sakamoto with the real world, resulting in an experience that feels material and ethereal at once. It is part of an ever-growing cohort of musical experimentations with AR and VR. In the thick of the pandemic in 2020, Billie Eilish performed a livestreamed concert jazzed up with extended reality effects that turned the stage into a gloomy ocean floor and a star-studded sky. In 2022, Gorillaz took over New York’s Times Square, then London’s Piccadilly Circus, as AR technology brought the band’s characters to life on giant screens. The list goes on to include Ariana Grande, Justin Bieber, and Travis Scott, most of whom started to dabble with mixed-reality during the pandemic.

With Kagami, the story began then, too. As Tin Drum founder Todd Eckert remembers it, he and Sakamoto were sitting in Ecker's apartment in early 2020, kicking around the idea of a mixed-reality experiment. At the time, the musician didn't know the cancer he had recovered from years prior had come back. "Maybe he said yes because he had missed two years [of performing] with treatment for his cancer before," says Eckert. Or maybe he said yes because of his fascination with technology.

Fast forward to November 2020, and both men flew from New York City, where they were based, to locked-down Tokyo, one of only two cities in the world with the setup necessary to capture Sakamoto in 3D and generate a virtual model of him using volumetric and motion capture technology. The session lasted three days, during which Sakamoto played his compositions under the scrutiny of 48 cameras, various microphones, and lightboxes.

There was nothing glamorous about it: Every surface around him, piano included, was covered in green—"Kermit land," jokes Eckert. Sakamoto was wearing lentil-sized motion capture markers on his face, with almost 20 of them strategically taped to the back of his hands. His thin silver hair, which normally falls over his forehead, was waxed back and held under a net so it didn’t hide his facial expressions from the cameras. "In service of making something authentic you have to go through impossible odds and processes that seem inauthentic," says Eckert.

It took about six months for Eckert’s team to process the raw data captured in the session. In the meantime, they also had to sculpt Sakamoto's hair from scratch and recreate his iconic glasses. But the most challenging thing to recreate was Sakamoto's face, which was blocked from the cameras for a large part of his performance because he was hunched over.

Maybe that's because the morning before the first session started, Sakamoto had learned the cancer was back, and it was everywhere. "I didn't find out what he had found out until a year and a half later," says Eckert, conjecturing that maybe Sakamoto knew he was going to die soon, so he was leaning over the piano more than usual. "These are the slowest versions of these songs he ever did," says Eckert.

Because the team had inconsistent data of Sakamoto’s face, both in form and texture, they had to make up for the missing data by reconstructing his face using referencing segments of complete data. It took months.

Meanwhile, separate teams were working on a series of 3D artworks that would end up accompanying the performance: gentle fog that curls around everyone’s ankles for the opening piece; a tree that draws itself on top of the piano a bit later. “It was never my intention to simply service people who knew they loved Ryuichi, but to try to expand the audience that would come to understand his work,” says Eckert. Luckily, the artworks elevate the performance instead of detracting from it. While Sakamoto is playing the hit track “Energy Flow,” for example, a window opens high above him, and snowflakes start floating through it. The snow effect references Akira Kurosawa’s 1952 film Ikiru, where the main character—an elderly man who knows he will die soon—spends his final moments sitting on a park swing during a snowfall, singing a romantic ballad.

The entire show feels otherworldly, but there’s only so much technology can achieve. The Magic Leap 2 has a 70-degree field of view, which is twice as big as the previous iteration, but it still means that when you look up at the ceiling to see that window, you can no longer see Sakamoto. The headset is also tricky to get used to, considering you're walking around in a room full of strangers wearing the equivalent of dark sunglasses. (The Magic Leap 2 comes with a dimming feature that allows real objects to slip behind virtual ones, but the illusion was so strong that people kept bumping into each other at test shows, so the team had to disable it.)

Surreal as it was, the performance left me somewhat puzzled. Can a mixed-reality concert ever make up for the sad reality of someone’s absence? I have never seen Sakamoto play live, but I doubt Kagami comes close to the palpable energy of his presence in the room. What do we lose when technology becomes so integral to the most intimate of human experiences—and what do we gain?

Music affects the brain in all kinds of different ways. Hayley Nelson, a neuroscientist and founder of the Academy of Cognitive and Behavioral Neuroscience, explains that our brains pretty much all operate the same way when we're detecting depth, motion, or visual patterns. But when you throw music into the mix, how those inputs affect the experience can differ from person to person. “While some individuals may find visual elements, such as the presence of a live performer or an aesthetically pleasing environment, to enhance their enjoyment of music, others may primarily focus on auditory aspects and derive pleasure solely from the sound itself,” she says.

Indre Viskontas, a neuroscientist and operatic soprano who serves on the faculty at the San Francisco Conservatory of Music, says that knowing who wrote the piece can affect how people experience it too. In 2013, a team of researchers ran a neuroimaging study in which they asked people to lie in a scanner and listen to a piece of music that was the same for everyone. One half was told the music was composed by a human, and the other half was told it was composed by a machine. The first group, Viskontas says, engaged a part of the brain that helps us understand how other people feel, think, and behave. In other words, when people knew the composer was human, they had more empathy. "One of the things we do as humans is try to understand, ‘What is that human being trying to tell me, what are their beliefs, are they friend or foe?’ And we do that by essentially trying to put ourselves in the shoes of that person," Viskontas says.

Viskontas acknowledges that the same study might yield different results if it were performed today, when our culture is steeped in AI-generated music and art. It’s also a bit different considering Sakamoto was human, and was performing his own songs for the recording—it was just his presence that was relayed through a machine. But according to Viskontas, the connection the public forms with him in mixed reality can only work if we buy into the illusion of a performer in the room, playing the piano—and if the technology is advanced enough to help us convince ourselves that it is real.

The latter may take a while. As outlined in a 2016 study, in order for VR technology to truly make us feel like we are present in a room with a performer and an audience, it needs to better translate subtle physical changes in the performer like blushing and sweating, or almost imperceptible things like the raising of an eyebrow. It would also have to give us the option of physical contact with those around us. Replicating the unique energy and atmosphere that stems from the collective presence of performers and the audience being in the venue together is another challenge. This is only partly addressed in Kagami, which may be designed for a small crowd to take in together, but remains a mostly solitary experience.

Still, the show remains a captivating attempt at capturing the impossible and freezing it in time. With no room for improv, Sakamoto will display the exact same emotions every single time the performance runs. Will the magic fade away if I go back a second time?

Perhaps Sakamoto knew the stakes. Reflecting on Kagami in early 2023, shortly before his death, he wrote: "There is, in reality, a virtual me. This virtual me will not age, and will continue to play the piano for years, decades, centuries … Will there be empathy there? Empathy that spans hundreds of thousands of years. Ah, but the batteries won't last that long."