Dry Run for Reality

A floor lamp from the 1970s, the type with a built-in table under the shade, creates a yellow glow in the dark room, where deep-blue matching sofas are spread over the brown parquet floor. It looks a little like a basement sitting room – until you get a glimpse of the dressing table in the background. And the refreshments. And the sculptures placed randomly around along the wall.

We are in an animated green room. The atmosphere is somewhat reminiscent of a scene from Grand Theft Auto. In the Rockstar Games cult series for PlayStation, the protagonist moves quickly around in the room hunting for money or drugs, often holding a weapon. In researcher Johannes Lunde Hatfield’s VR world, it is a different kind of excitement that is about to materialise. From the simulated backstage area you are hoisted through an opening in the floor, like a pop star, and placed in front of a virtual audience waiting to be entertained.

The VR lab on the 6th storey of the Norwegian Academy of Music (NMH) allows music students to do a dry run for authentic concert situations. The project, which is called Practising and Achievement Preparation in the Interactive Space, is associated with CEMPE (the Centre for Excellence in Music Performance Education) and is a continuation of Hatfield’s PhD thesis on mental practising for musicians based on sports psychology. Wearing VR glasses, one can choose between different halls and settings adapted to individual training needs: a chamber music hall or a larger concert hall inspired by the Elbphilharmonie in Hamburg, and a jury is ready to listen to those who want to practise being in an audition situation.

3D sound is provided by 12 small Genelec speakers, eight of which are placed in a circle on the floor, and one from each of the four corners of the ceiling. Composer and sound artist Ulf A. S. Holbrook has mixed and manipulated the environment to evoke realistic sound experiences appropriate to the different spaces, including audience sounds recorded in the Levin Hall at the NMH and in the Oslo Concert Hall.

Hatfield, a teacher and cellist, began working with the VR lab in 2017. So far, the students have tested only the virtual practising tool's audio elements. In November 2019 the first test round with both sound and images was carried out, and the goal is that the VR lab will be able to be used starting in 2020 – although Hatfield is reluctant to name a specific date. It’s not unusual to encounter some bumps in the road when you set out to develop a product that the world has never seen before.

Demanding audience

So far we are watching the virtual practising world from a computer screen on the desk in the VR lab: the VR glasses from China, which just arrived, are not working properly. Pål Hieronimus Aamodt is the COO and project manager of Pointmedia, the company behind the 3D animation in the project. Now he is sitting in front of the computer screen, opening and closing windows, clicking on things, trying to get the large, black glasses to communicate with the software they have developed. Scratching his head.

“We’ll take it up with the producer,” he says, partially to himself and partially to Hatfield, who nods.

The VR lab has presented several challenges along the way, such as how to create the sound of a large concert hall in a room that only measures a few small steps from one end to the other. The biggest problem, though, has been the audience.

“In the midst of the development process, we discovered that the person wasn’t good enough. We have spent 1500 hours creating a new one,” Aamodt explains, bringing up the smallest concert hall on the screen and zooming in on the audience.

They are sitting like a typically well-mannered audience at a classical concert, with their hands in their laps, directing their gaze towards the stage. But the faces are similar; several of the women have bought the same bright green dress, and the men have gone to the same barber. We are seeing an older version of the program, Hatfield assures us. The new people Pointmedia has developed have wrinkles, more texture, different facial expressions.

“But when you animate everything in 3D there are limits to how close their movements can be to reality. If you have face capture and motion capture, you can produce audience movements and facial expressions more easily. But a motion capture suit costs around NOK 100 000, so you need the resources to be able to access it,” says Hatfield.

Coughing, clapping and booing

It is Hatfield himself and Pointmedia that decide how the audience members should look. Their behaviour, on the other hand, can be pre-programmed or triggered by the teacher while the student performs in the virtual concert reality.

Some of the buttons in the program are labelled “Random cough” and “Random clap”, and others “Random sleep” and “Random boo”.

Booing in the practice room – is this meant to toughen up the students?

“Ha-ha… yes, but if the student manages to play despite booing or a lot of hawking and coughing, and can still focus on the music, then he or she will be more prepared to react in the same way in a real-life situation. The brain will be adjusted to dealing with distractions,” Hatfield explains.

The goal of the VR lab is not to programme students to present something that is perfect but to lay the groundwork for enabling them to express what is on their minds when this is important. So far the program is designed for one soloist and one accompanist, but the potential is vast, according to Hatfield and Aamodt. The software can be expanded to involve orchestras and cathedrals, and the VR lab can also be adapted to practice situations within other fields of study. One example is educational psychology, where training could be designed for tackling difficult classroom settings.

“There should be courses in how one thinks as a performer when using simulation. This involves extracting the maximum from a performance by imagining that there is only one take. Maybe students get distracted by the various sounds we add while they’re practising, things that might even seem unrealistic. But the result is that they are more robust when they’re in the actual situation,” says Hatfield.

VR technology is in its start-up phase, he emphasises. VR glasses are large now, but in three or four years small glasses will have the same functions. He does not hide the fact that he would like to have more time and resources to develop the project, and points to RITMO, the Centre for Interdisciplinary Studies in Rhythm, Time and Motion, at the University of Oslo, which operates with a 10-year perspective in its research.

“This project has a two-year framework, and that’s far too short. When they know that we are the first to do this, I wish that the NMH had made a longer-term investment in the project. I don’t think they have truly realised what a complex job it is to work at this level within technology, sound and images. What we are creating now is a beta version of what the VR lab could become. We have built a foundation for something much bigger.”