top of page

Sound Moves

3D-Audio-Driven Interactions for Audio Games

Skills I used

UI/UX Design

(3D) Sound Design

Game Design

Sound Moves: 3D audio driven interactions for mobile games 

Sound Moves is an investigation into 3D sound-driven audio gaming, supported by the Royal National Institute of Blind People (RNIB). Sound is traditionally a secondary element in interaction design, used mainly for feedback, notifications, progress tracking, and to support visuals in creating mood, tone, and immersion. As modern interfaces become more diverse and context-sensitive, sound is bound to gain new relevance.

Exploration: how does sound shape the way we move? Series of games and workshops using sound a driver of interaction

custom-made buzzwire game with varied sonifications: here, a gyroscope detects rotation of the handle, which is sonified. how does sound impact our fine motor skills?
Drawings following arcarde design workshop: what concepts would be playable without sight?
arcade game design workshop: what could a sound-driven arcade game be?
study of impact of sound design on location tracking of multiple sources
Could we interact with a sound based interface?
sound Interface: could users use a 3D sound menu?
Final Concept: Interaction with 3D Sound Objects in Audio Mobile Games
In gaming, sound being the primary driver of interaction makes sense. It encourages us to pay attention to the subtleties of a medium ideally suited for storytelling, immersion, and emotional engagement. It also serves the 100 million visually impaired gamers in the UK alone, who are rarely fully included in the design of games. 
​
Interaction with 3D audio is underexplored in current mobile audio games, so I designed a new mode and interaction types, allowing the player to interact with sound objects in an anchored 3D sound space. The concept was notably tested by Ben Breen, Lead Game Accessibility Consultant at the Royal National Institute of Blind People (RNIB).
The Experience 
The Tech
Possibility for interaction with an anchored comprehensive soundscape: Tech ecosystem for embodied mobile interactions: head tracking through CoreMotion framework and gesture recognition through a custom AR app - the phone is used as a remote control/laser pointer. Audio Spatialisation through Steam Audio for Unity. 
The Interaction Toolkit 
Five Interactions
​
  1. Inventory Checking
    Players can access virtual objects by moving their phone around their belt, as if selecting from a holster. Each retrieved object is represented by its characteristic sound.
  2. Weapon Use & Switching
    Players can activate and use weapons with immersive, spatialised auditory feedback. A quick double-tap gesture switches between weapons.
  3. Audio Progress Bar
    To indicate progress during ads or dialogue, the bass component of the audio is spatialised from left (just starting) to right (nearly finished), giving players an intuitive sense of time remaining.
  4. Grabbing a Sound
    Players can “grab” a sound by pointing their phone toward it and long-pressing the screen. Once grabbed, the sound follows their arm movements.
  5. Throwing a Sound
    After grabbing a sound, players can release it by making a throwing motion — thrusting their arm in the desired direction to project the sound away.
Demo: 2 Audio Games 
With the most successful interactions of the toolkit, I designed two games. Here are their recordings! The Audio will not be spatialised properly for you, because it is recorded from the pov of the player. Note that they do never look at a screen and orient themselves using sound only!
GrowlAttack: Classic zombie shooter with 4 different levels 
Shush: RPG where the player needs to find their 'magazines' in their sleeping baby sibling's room without waking it up. The player needs to interact with the 'good' and 'bad' fairies of slumber, who are trying to sabotage the player's mission.
Exhibitions: Sound Moves - Arcade Style 
Sound Moves was exhibited as an arcade-style game. The demo showcased 3D sound source localisation through five hip-level buttons arranged in a circle around the player (one at the front center, two at the front sides, and two at the back sides), which players pressed in response to the directions of a whimsical ‘grandpa fairy’ character guiding the experience.
bottom of page