Spork

Jan van Eyck, The Ghent Altarpiece, Singing Angels (detail)

The Challenge: Innovating the Classical Music Experience

In today’s digital landscape, audiences crave interactive and immersive experiences that traditional classical music performances lack. The industry faces several challenges:

Limited accessibility: Geographic and economic barriers prevent many from attending live concerts in renowned venues.
Static experiences: Conventional recordings and broadcasts fail to provide the engagement modern audiences desire.
Need for innovation: Orchestras and venues require new methods to connect with audiences and enrich the concert-going experience.

Our Solution: Enhancing 4forty.io with Immersive Technologies

Spork is spearheading the development of immersive streaming tools integrated into the 4forty.io platform, which in turn has been integrated into the Sinfon.io iOS app. We have transformed how audiences experience classical music performances. Our innovative approach combines cutting-edge video and audio technologies to deliver an unparalleled, immersive concert experience directly to users’ devices.

Key Developments and Features

Custom 360° Video Player for iOS

Immersive viewing experience: We built a bespoke 360° video player for iOS, allowing users to virtually attend concerts and experience performances as if they were seated in the concert hall.
User-friendly design: Intuitive controls enable effortless navigation of the 360° environment, enhancing engagement and enjoyment.
Multiple viewpoints: Users can switch between different camera angles, exploring various perspectives of the performance.

Integration of POLIMI’s 3DoF Technology

Three Degrees of Freedom (3DoF) audio experience: In collaboration with Politecnico di Milano (POLIMI), we integrated their advanced spatial audio technology into our app. POLIMI developed a JUCE plugin, which we incorporated as an Audio Unit via CoreAudio in our iOS app.
Dynamic audio interaction: This technology allows users to look around the concert space, and the audio changes dynamically with the movement of their device, enhancing the sense of presence and immersion.
Enhanced sensory experience: By synchronising visual and auditory elements, users enjoy a more realistic and engaging concert experience.

Live Streaming Milestone: London Philharmonia Orchestra Concert

One of the most significant achievements of our project was the successful live streaming of the London Philharmonia Orchestra concert from the Royal Festival Hall on November 10th.

Technical Highlights

Multi-camera setup: We collaborated with Metacamera to stream four 4K 360° equirectangular video feeds. Each camera was strategically placed to capture different viewpoints of the concert hall, providing users with a variety of perspectives.
Multiple viewpoints: Users could switch between these four 360° camera angles in real-time, allowing them to choose their preferred view of the performance.
Ambisonic microphones: POLIMI attached an ambisonic microphone to each 360° camera. This setup captured spatial audio corresponding to each visual perspective, enhancing the immersive experience.
High fidelity stereo recording: In addition to spatial audio, we incorporated a full stereo orchestral recording provided by Floating Earth, utilising over 40 microphones placed throughout the orchestra for optimal sound quality.
Customisable audio experience: Users could choose between the immersive 360° audio from the ambisonic microphones or the high-quality stereo recording. This flexibility allowed users to tailor their auditory experience, either immersing themselves in the spatial audio or enjoying the richness of the professionally mixed stereo sound.

360º Streaming Setup Backstage at the Royal Festival Hall

Future Enhancements in Development

6DoF Technology Integration:

Six Degrees of Freedom (6DoF): We are working on incorporating POLIMI’s 6DoF technology, which will allow users to move virtually within the concert hall. Users will be able to explore different locations within the venue, such as the conductor’s podium, the string section, or the balcony, enhancing the sense of physical presence.
Elevated Immersion: This advancement will enable users to virtually “walk” around the venue, offering an unprecedented level of interaction with the concert environment.

Sound Source Separation Technology:

Interactive Audio Experience: Future updates will enable users to focus on specific instruments or sections of the orchestra. By integrating sound source separation technology, users can isolate and highlight individual audio elements, such as the violins, brass, or percussion.
Educational Opportunities: Offers a unique way for students and enthusiasts to study compositions and orchestration. Users can analyse how different instruments contribute to the overall performance.
Personalised Listening: Enhances enjoyment by allowing users to tailor the audio experience to their preferences. Users can create their own mix, emphasising the instruments or sections they are most interested in.

Real-World Applications

Educational Institutions:

Music schools and conservatories can use the app as a teaching tool.
Students can study performances in detail, analyse techniques, and understand spatial acoustics.

Remote Concert Series:

Orchestras can offer virtual concert subscriptions, attracting international audiences.
Enables the hosting of global events without the logistical challenges of touring.

Outcome

Spork’s involvement in the Repertorium Project exemplifies our commitment to bridging technology and culture. Through the development of the Repertorium Workspace and advancements in immersive streaming technologies, we have made significant strides in preserving cultural heritage and innovating the classical music experience.