Posted on: October 15, 2018
I developed a mobile headphone mounted interface that can track the orientation of the listener’s head and deliver dynamic real-time spatialised audio. Meaning that a sound can be positioned within a 360 degree audio field, and the position of that sound remains static in relation to the position of the listeners head and ears.
This was initially prototyped by mapping the magnetometer readings from an iPhone to OSC (Open Sound Control) messages, using the GyrOSC  mobile app, to control the Envelop  surround sound panner inserted into a single track playing a piece of music in Ableton Live, with the music delivered through a pair of wireless headphones to allow for the listeners 360 degree rotation.
Using the above prototype, the music’s perceived source in physical space could be determined quite accurately by spinning around on an office chair, or by standing up and shuffling around on the spot. This was the case for both the on-ear headphones and the bone-conducting headphones, the later resulting, to varying degrees, in the perception of a virtual audio source within the natural ambience of the physical space.
The failure of the surround sound panner to react to movements of just the head (and the ears), due to it being controlled by the hand-held magnetometer in the iPhone, was quickly made apparent (this had also been experienced previously with other smartphone sensor based mobile sound experiences). It was therefore decided that a head-tracking device would perform much better.
Alongside the above experiment I had been exploring the use of Bluetooth beacons as a way of deploying physical markers in space for the positioning of virtual audio sources, or for tracking the position of listeners.
Other than assessing a beacon’s RSSI (Received Signal Strength Indicator) value from its proximity to a smartphone via a Bluetooth scanning App, most other features, such as determining a range from multiple beacon sources (also known as ‘ranging’) and adjusting the TX, and Major and Minor values of the beacons, proved difficult or impossible to carry out as most beacons are ‘locked’ into their manufacturers API’s via their UUID’s in order to fulfil the IPS (Indoor Positioning Service) that they offer to customers who purchase their beacons.
It also became clear that in order to scan, receive and forward the RSSI values of multiple beacons to a central DAW (Digital Audio Workstation), or some kind of audio engine, would require a BLE (Bluetooth Low Energy) gateway, or GATT (Generic Attribute) Server. Again there are mobile apps available that can do this, but most, if not all, are specific to a beacon manufacturer.
This emerging investigation into BLE networks with both central and peripheral roles for BLE beacon devices lead me to the Puck.js  product, which presented itself as a potentially useful and versatile research tool in this area.
The small, light-weight and mobile Puck.js enabled me to test a headphone/head mountable magnetometer, and also provide further avenues of experimentation such as listener positioning and scanning for the proximity of other beacons, relaying data to and from other BLE capable devices (Figure 1).
Figure 1. The Puck.js mounted on top of a set of Bluetooth headphones.
Figure 2. The progressive web application source code
The Puck.js also enables experimentation with listener mounted beacons, either as central or peripheral devices within a BLE network through Espruino’s GATT server module.
This head-mounted position could also prove useful for indoor positioning and communication between participants headsets as it may eliminate much of the interference generated by human traffic that Bluetooth signals are prone to,especially if it was communicating with other beacons positioned directly overhead, or beacons attached to a ceiling. With this view, it could eventually be embedded in the headband of the headphones, or provided as a clip-on accessory.
Additional prototypes were made using Espruino’s BLE MIDI module and Web Bluetooth capabilities to send the calibrated magnetometer data as MIDI control messages over Bluetooth to a DAW, and to a Bluetooth enabled web application respectively. The later utilised a combination of Web Bluetooth and the Web Audio API to realise a standalone, mobile, dynamic and responsive surround sound experience (Figure 2). This was achieved by mapping the Puck.js’ magnetometer data to the Web Audio API’s PannerNodeparameters. Although this approach was successful in moving the audio source dynamically around the listener, as the Web Audio API provides an AudioListener interface with orientation properties, it would make better sense, both semantically and functionally, to use this going forward.
Figure 3. A sketch of the prototype system.
Potential future studies
To summarise, some of the possible future avenues of study this initial prototyping has revealed include:
All of the above objectives could potentially form part of one carefully designed study.
Things that require attention
Some points that need addressing based on the use and evaluation of the outlined prototypes are:
Refinement and clarification of my research area
The outlined prototyping activities have led me to think about several possible refinements, and some clarifications, of my area of research:
Potential Practice-based studies
This focus has given the practice-based aspect of my research more clarity too. I can now identify potential applications and target collaborative practice with the developing framework around them. These include: