sound frankenstein

track 1

Untitled presentation (3)_Page_01.jpg

track 2

Untitled presentation (3)_Page_11.jpg


  • pick tracks
    • i selected two minimal sounding tracks (with recommendations from Mimi). why minimal? more clear to how they make me feel i.e. easier to tune in.
      • track 1 – Three Voices by Morton Feldman
      • track 2 – Prayer by Huun Huur Tuu
  • tune in to how i feel
    • this was a lot more difficult than expected. even though i am a non-musician, when trying to ‘listen’ to something, i often internalize it logically i.e. in terms of notes, pitch, rhythm etc. perhaps because there are the visuals associated with those i.e. sheet music, spectrograms, itunes visualizers.
    • picking minimal tracks helped a lot with this – more single-minded
  • sketch out on paper
  • look at data from track
    • fft analysis – analyses the amplitude at different pitches, pick number of bins from 16-1024, pick smoothness / noisiness
      • try to visualize this in different ways to see if there are patterns corresponding to my sketch on paper / how i feel
        Screen Shot 2561-11-16 at 15.33.08.png  screen-shot-2561-11-19-at-10-00-18.png
        Screen Shot 2561-12-02 at 15.56.26.png
      • this is the fabric of all things, spotting the right patterns can lead to very satisfying results
    • get energy – take ranges of bins and averages out the values
      • this is a lazy way … although sometimes this works well, it lacks the nuances which sometimes give insights to the tracks … because it ‘averages’ out the data.
    • amplitude – overall volume
      • since it is an ‘overall’ data, this works well when used to set the ‘tone’ of the drawing, equivalent to a canvas’s quality or primer in a painting.
  • start visualizing (coding)
    • this process is full of wonderful surprises // numbers, relationships, numbers, relationships, numbers, relationships, numbers, relationships, numbers, relationships, numbers, relationships, numbers, relationships, numbers, relationships, numbers, relationships, numbers, relationships, numbers, relationships, numbers, relationships, numbers, relationships, numbers, relationships, numbers, relationships, numbers, relationships, numbers, relationships, numbers, relationships, numbers, relationships,
    • time plays an important role in making you feel something (much like in animation) … frame rates, increments, decrements // factor this into numbers and relationships
  • fine-tune visuals
    • fine-tuning takes the most time – changing the transparency by 1% can change the whole drawing. this is a lot of going back and forth between how i feel from listening to the music and what i see
  • know when to end
    • when your ears and eyes feel like they match. i find this process a lot similar to a painting, going too far makes the sketch look ‘overdone’
    • caution: do not get carried away by data – must go back to tuning in process // it is very easy to be lured by the sophisticated visuals (this happens often when headphones are off and i’m just making the sketch ‘look nice’)


  • the best advice i received was to ‘just put in the hours’ // it is like trying to learn illustrator’s black and white mouse tools (so weird)


  • this exploration started because i wanted to hear what musicians were hearing, with a hypothesis that understanding the technical aspects of it and realizing its patterns would help me appreciate and ‘hear’ music more. however, through the exercise i realized it wasn’t about what went in to create the music, rather it’s about listening and realizing what and how exactly a particular sound makes me feel.
  • when listening to music, the feelings i get do not translate to spectrograms, rather some abstract, undefined … stuff
  • these visuals work as a bridge to help me define and express my feelings of a musical piece
  • i hope these visuals stimulate people to ‘listen’ and that they bring people closer towards their feelings (that they might or might not know are there) when they listen to a piece of music.
  • conclusion code: *???* i like arrays because they are like a flexible and well organized box of materials 


project files – tracks, codes, visual thumbnails, other notes //

notes from playtesting //

track 1

track 2
  • track 1
    • play track 1 – how do you feel?
    • play visuals for track 1 – can you see what you feel from what you hear?
  • track 2
    • play track 1 – how do you feel?
    • play visuals for track 1 – can you see what you feel from what you hear?


  • how do you feel?
  • what are the similarities and differences between what you see and what you hear?
  • what if you were to visualise the tracks? what would it be like?



interesting thing about frequency // wine glass breaking

everyday sounds we look pass (or hear pass) // sonic branding

it’s about communicating emotions // comment on jacob’s video –

what actually happens when you create a sound – the visual is more like this than like notes or other metrics // tom thum –

reason to visualise based on feels // feldman site 

reason to visualise based on feels // overtone singing spectrogram

essence of project // pauline oliveros ted talk

if only i had more time – same theme // tuning meditation, pauline oliveros –

if only i had more time – same theme // einstein on the beach, phillip glass –

other references // zero and one, Laurie Anderson – Difficult Listening Hour


ICM questions

presentation question ideas

  • why do you think i’m doing this?
  • what visuals / sounds come to mind?
  • what’s the difference in seeing this vs. spectrogram?
  • what difference would it make if I used ml5 pitch detection and visualized the pitch?
  • what do you think I learnt from doing this?
  • how did it make you feel?

general questions

  • why does pitch detection require machine learning? isn’t it just picking the frequency with highest amplitude?
  • is there a way to input sound into browser mic?
  • how are the bins related to the frequency? what is the minimum vs. maximum frequency the bins are representing? – it’s a lot less nuanced in the lower frequency sounds (really visible on (C5-B7))
  • minimal tracks that are contrasting to three voices?
  • how long do we have?
  • how to store multiple elements in 1 array?
    answer found –
  • is this the correct way to initialize the index for an array?
    let waveL = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
  • what makes something only able to run on local server?

project tension

  • more functions
  • new tracks, tushar’s project, mic
  • hard to make one more sketch in one day without them being similar
  • make it more like a ‘feeling’ supposed to the technical aspect of the track i.e. too influenced by the fft analysis – hard to be authentic to feeling when something is already defined as a measurable thing
  • how to pick the right data to represent something
  • fine-tuning – spend 20% of the time fine-tuning and capturing the nuances // like adding bias?
  • note screen res: 1920:1080

general tensions


Playtest Sound Frankenstein

SOUND – “What do you see?”

Opening, Three Voices – two sidedness, imbalance, space, blur, dynamic movements

bouncing ball, sometimes out of sync, starts with 1 – goes up to 3, pendulum, pinned in the middle, afraid to move because it’ll affect the 2 sides, complexed juggling, mirror of 2 things, stuck between 2 sisters (from The Shining) singing out of balance, unexpected, gradient, no sharp lines/edges/shapes, totally organic, moving from left to right (stereo sound), no discernible shape, all gradients, pingpong ball going back and forth, mix of waves, birds flying, Perlin noise waves moving further and further out, creepy ambience, searching, dessert, a cold pond with 4-5 frogs, multiple layers of sound, mice singing, people approaching closely and singing in the ears, things appearing and disappearing, church, 3-4 girls singing and walking randomly – they won’t look directly, just go round and round, hollow forest, not a single visual, journey changes as song progresses

VISUALS – “What do you hear?”

Route 1 – hollow, space, imbalance, appearing and disappearing

Screen Shot 2561-11-19 at 09.52.42.png

breathing with pneumonia, raspy breathing, out of breath, black hole, deep ocean, screaming, subtle gradient, pulling in, imbalance, chilled out, meditative, taoism, heartbeat, tunnel, water slide, swooshing sound, ambient noise (no form), waiting for random colors to arrive or something to pick up the visual but it never arrives, space and time collapsing, wind blow, cold, orbit sound, pure, inside big snow, slow voices

Route 2 – two sidedness, preciseness, vibrations

Screen Shot 2561-11-19 at 09.54.36.png

high pitched waggling, mosquito buzz (not pure sound), pinned to a particular pitch, 4 note chord, needle, precise, muddy low sounds (flickering), metal on metal, sharp, twins from The Shining, gap between doors, vases, buzzing neon, pulsating lines, deep house, bird zapper, frame – something needs to go inside it, electricity, light with bad connection, sunshine reflection

Route 3 – multiple origins of sound, chords, consonance / dissonance, bling

Screen Shot 2561-11-20 at 11.13.31.png

chorus, a lot of chords, dissonance, consonance, strong middle range, a lot of space between the ranges, 3 narrow bands, few notes, sliding between notes (not jumping), holy songs, organs, instruments & voices, lights from stained glass windows, dark room, nostalgia, light on and off, fireflies, nyc subway artist, city lights, fastness, business, tempo, trip-hop, acid-jazz, verizon wireless commercial, ultrasonic sound, underwater,

Further exploration – “what do!?”

  • Combine sketches according to feedback (Opening, Three Voices)
  • Move on to other songs
  • Use ml5 Pitch Detection to analyze song
    • visualize based on notes / harmony

      Screen Shot 2561-11-22 at 11.07.35.png

  • Work on Tushar’s harmony project
    • a person sings a note and the computer creates a song / harmony


More Updates – Sound


  • listen to sound
  • how do you feel
  • what visuals do you see
  • composition / color / shape
  • show v1 / v2 / v3
  • what do you hear


Song reference:

Song is separated by left and right sounds.

Route 1


Screen Shot 2561-11-19 at 09.52.42.png

Route 2


Screen Shot 2561-11-19 at 09.54.36.png

Route 3


Screen Shot 2561-11-20 at 11.13.31.png

Other tests

Screen Shot 2561-11-19 at 10.00.18.png


Screen Shot 2561-11-14 at 11.24.26.png

Screen Shot 2561-11-14 at 11.26.52.png

Screen Shot 2561-11-19 at 10.03.12.png

Screen Shot 2561-11-19 at 09.58.48.png



Screen Shot 2561-11-11 at 19.45.37Screen Shot 2561-11-11 at 20.59.35Screen Shot 2561-11-11 at 21.22.01Screen Shot 2561-11-11 at 21.37.02Screen Shot 2561-11-12 at 11.47.51


Not just aesthetic, insight to harmony… understand music theory for non musician
Possible to analyze notes?
TeChnical question – split line?
Tools to discover harmony?
Visualization techniques?
Is it possible to save analyze data from console. Log
How’s p5 sound different from p5 tone js
web audio api
mozilla – something beyond fft
ml5 pitch analyse
jason sigal – sound library
wave visualiser
save string / save json
function to round up number
audio analysis
aaron montoya
book office hour with dominic
yotam mann – guy who wrote tone.js
zach voice
p5 speech – text to speech / speech to text


I don’t know a lot about sound and music but they have a lot of influence on me. Last week’s class made me question how sound works, how our brains register different sounds and how they can influence us. After last week’s class on sound I went down a rabbit hole.

First – “Everything You Should Know About Sound”
I feel your vocal cords vibrating

  • “The most important kind of human communication happens when our brains send information to other brains through complex patterns of air pressure waves. Have you ever stopped and thought about how incredible that is?” –
    red dot

Second – Musician Explains One Concept in 5 Levels of Difficulty (Harmony)
How is there so much harmony? What are we really hearing?

Third – Simple Waveforms Combined
Look at those weird patterns

Fourth – Online Tone Generator
Playing different notes that sound unfamiliar

… and so on.

At this point I know I wanted to work with sound.


  • body synth –
    what if you and your movement become the note?
    Screen Shot 2561-11-06 at 10.15.23.png
  • webcam sequencer –
    what is this hidden meaning my face is creating?
    Screen Shot 2561-11-06 at 10.17.49.png
  • slider dance –
    how about switch roles with slides? let the sound turn the slides? and we turn the sound?
    Screen Shot 2561-11-06 at 10.13.56.png
  • self text fill –
    play with lyrics?
    Screen Shot 2561-11-06 at 10.10.47.png
  • petri dish –
    have lyrics bouncing off according to the beat?
    make lyrics explode and rejoin?
    Screen Shot 2561-11-06 at 10.20.25.png
  • bendy grid –
    this connotes the sense of wavy well? this is similar to what I see in my head.
    Screen Shot 2561-11-06 at 10.46.17.png
  • word synth –
    try matching words with pitch?
    Screen Shot 2561-11-06 at 10.16.54.png

Source of materials

Other things I might want to explore

  • Algorithmic patterns that exist in nature… might they be related to sound? … What if a value in a pattern is incremented in the same way a note does?
  • What if we can reach inside a wave and touch a note? Bend a wave? But make the visual stay… like a harp but with visuals?
  • Is there something that is beautifully out of proportion… since our brains just want patterns?
  • When is something hypnotic vs. meditative?

A sketch I made with lyrics


Who’s it for?

Mostly myself … and non-musicians who enjoy music/sound and want to learn how they work.

How will people experience it? Is it interactive?

Visually, sonically.

Body movements.

Is it practical? No. Is it for fun? Yes. Is it emotional? Maybe. Is it to provoke something? Yes… Question the sounds… Seek for nuances in sounds… Observe it more.




For this week’s assignment on video & sound, I decided to work on sound. I do not know a lot about sound but I’ve always loved music. Learning about what sound actually is and how it actually works was mind opening.

“The most important kind of human communication happens when our brains send information to other brains through complex patterns of air pressure waves. Have you ever stopped and thought about how incredible that is?” – Everything You Should Know About Sound – Tim Urban

The musical piece I chose to visualize was “Bloodstream” by Tokio Myers. It is a combination of Clair de Lune, Debussy and Bloodstream Ed Sheeran. I didn’t know either of the songs before I heard Tokio Myer’s edition so they both felt like one song to me. But they’re very different in many ways.

My sketch:

Screen Shot 2561-11-03 at 14.53.14.png

Screen Shot 2561-11-03 at 14.53.35.png

Screen Shot 2561-11-03 at 14.53.39.png

Screen Shot 2561-11-03 at 14.54.07.png



Inspired by

Bands generated from the site have actually been famous!

play here:

Screen Shot 2561-10-23 at 11.02.20.png

Screen Shot 2561-10-23 at 10.55.05.png

I ran through countless problems with this exercise.

  • http vs. https – took me 2 hours to work out that I was missing an ‘s’
  • API rate exceeded – I cannot query data after sometime
    The code would not run
    Screen Shot 2561-10-23 at 10.40.35.pngScreen Shot 2561-10-23 at 10.27.50.png
  • html over canvas – I cannot create the canvas on top of DOM element

Too add:

  • adjectives

The original design:

Screen Shot 2561-10-23 at 10.44.40.png

Screen Shot 2561-10-23 at 10.40.35.png