esther

maltby



︎︎︎ reel 2021

visual
illustration
animation
moving image

set design / installation
macbeth
continuum

murals
чаепитие
I SEE YOU
мечта
dacha

experience
distortion

systems / research
vote now
pouched


 who ??











































distortion



interactive installation / audio experience design / creative coding / AR filters

march 2020
+ A. Munzano // T. Fung

An interactive audio-visual exploration of synesthesia.



The particular aspect of synesthesia this installation looks at is chromesthesia, people with which are able to associate pitch / sound frequencies with specific colours. Using a single sensory input (sound) the installation maps that live into two visual components and one audio.


system overview



Live audio recording processed through MaxMSP to generate distortion effects on live video.



summary of effects


FFT sound and buffer tests
Initially tests were carried out using a buffer to pre-record audio, apply FFT to convert sound from the time domain to a frequency domain; those frequencies can then be used for further processing. This was done using Volker Bohm’s external object vb.FFTWbuf~. The technical side of this exploration was carried out by Aida. In theory the object performed a single FFT to the sound inside the buffer and extract 3 frequency coefficients which were then scaled within the visible light spectrum and used to produce a colour in RGB. This was then converted to HSL values, from which the saturation factor was extracted.

Echo and panning
In order to create a fully immersive the user in experiencing aspects of synesthesia, I wanted to add a ‘whispering echo’ effect to the sound output. This had technical limitations, so was implemented for a computer sound input and 2 channel headphones output - which gave the advantage of somewhat recreating a 3D echoing soundscape. First I applied a reverb effect to the filtered sound to add in an effect of distance / grunge. Then looping through tapin + tapout buffer the sound is ‘panned’ through the left and right ears.