Digital Sheet Music in Mixed Reality

Virtual Reality (VR): fully immersive (e.g., Meta Quest, Valve Index).
Augmented Reality (AR): overlays digital objects onto the real world (e.g., Pokémon Go, Microsoft HoloLens).
Mixed Reality (MR): merges the two, letting virtual objects exist and interact within real spaces.

Portraits of Change is a mixed-reality jazz show. Learn more at mixedrealitymusician.com/work/portraits-of-change.

A recent update to our software includes HDRP support and a look into digital sheet music, calibrating VFX on a piano using hand tracking (thanks to the Leap Motion), and playing the piano in an AR environment using VR passthrough with the ZED Mini and Valve Index.

In this test, we implimented our PDF reader solution built from the HoloLens 2 demo to test reading sheet music in a VR environment with mixed reality passthrough.

A novel UI was designed to access the visual effects and the sheet music for different instruments.

No VR Detected

Digital Sheet Music in Mixed Reality