1234, Tuning-In

Detour (LONG VERSION)

“Music is the pleasure the human mind experiences from counting without being aware that it is counting.” Gottfried Leibniz

Peeling apart the layers between our sensorial experiences and our cognition.

Working with numbers I started being interested in the act of counting. It is one of the most basic forms of structure we take for granted. Did you know children have to learn to count in one to one correspondence? As if it isn’t obvious that counting is one to one.

1234 is an installation which asks people to count together. It is about deep listening.

So how does it work?

There are two groups: L and R, everybody is connected to the same sketch

IMG_0631.jpg

The interface looks like this.

Screen Shot 2563-05-06 at 06.41.38.png

They can’t see each other’s screen. But they can hear each other. One ear Group A, the other ear Group B (optimised for maximum confusion).

Demo video (STEREO):

Last notes on fabrication (MAKE 8)

Milling to every bit of material I had on hand

IMG_7909.JPG

Able to fit in threaded inserts with a little bit of super glue and sanding. I adjusted the holes to be closer to the middle, leaving more room for the edges.

IMG_0397.jpg

Everything came together in the end!

IMG_0464.jpg

I then assigned each participant a specific channel for the experience.

IMG_0500.jpg

Delivered to people by foot (great break, great fun interacting with physical humans, even from a distance).

IMG_0543.jpg

And the time is here!

Screen Shot 2563-05-06 at 06.53.34 copy.png

8PM BEFORE PLAYING

Screen Shot 2563-05-06 at 06.55.48.png

(tease) 0:00 – 1:05 // (end) 2:10 – END

This is after 5 tries:

It is about the interaction, do you choose to lead? Follow? Create a unison? Counterpoint? 

People usually stop when there is a distinct pattern of rhythm.

The response “Visceral, the stereo setup makes it very disorienting, you don’t know whose is whose, it’s restrained which amplifies the satisfaction when a pattern appears.”

8.45 PM AFTER PLAYING

masstuningArtboard 2.JPG

Thank you Ada, Wenjing, Louise, Jake, Matt, Noah, Arnab

More on slides:

First One Done, 8 To Go

A Single-Channel MIDI Instrument

IMG_0250 2.jpg

IMG_0252 2.jpg

C7B605B1-721B-49D0-929D-6863EA9DF05B.JPG

The many steps before that…

Prototyping and fabricating week!

The goal here is to make it as thin as possible in all dimensions, width, height, depth.

I started off by taking apart one of the slides to see what it’s made of and how I can make it thinner.

IMG_0162.jpg

Turns out there is no way to make it thinner except if I make the slide itself a circuit board…

But, that’s not happening! Too risky and no parts.

IMG_0162.jpg

I then tried different ways to stack all the components together to find the best solution.

C73737AB-6A68-44A3-ADAD-3A2CE3B41634.JPG

What’s the best way to package them all?

IMG_9574.jpeg

After trying a few variations, I made a sketch of the ideal scenario.

IMG_9152.JPG

Once I had the idea, I tried out with the cardboard prototype to get a sense of scale (the PCB here is used to represent the board + wires).

IMG_0172 2.jpg

I was quite happy with how thin I got it to be.

IMG_0173 2.jpg

I then tried milling with leftover materials I had to get a sense of the depth of the pocket for the slides and boards.

IMG_0339.jpg

Once I had the pockets I then worked on the wiring.

IMG_0321.jpg

And this is the compilation of everything I tried before finalising the design and parts.

IMG_0267 2.jpg

Finally I put all the best pieces together …

IMG_0260 2.jpg

Once put together, I was satisfied with the outcome.

Only 7 mm (around 5/16″) gap between top and bottom pieces! And the width x height is 10 x 2 cm.

IMG_0250 2.jpg

IMG_0252 2.jpg

IMG_0340.jpg

Yay, this worked:

Sadly, the pictures are not what they seem…

This is the reality

IMG_0311.jpg

Once I decided on the optimal measurements and fixed the files, I started shopping on McMaster-Carr. SO MANY OPTIONS! SO OVERWHELMING! I must have spent around two hours on this website.

Screen Shot 2563-04-21 at 17.58.25.png

Waiting for parts to arrive. These are the expected dates:

  • Soldering paste: April 24 // I have this problem where the very tip of my soldering iron doesn’t heat up (but it’s new and not oxidised, what’s wrong?)
  • Boards: April 24
  • Screws and threaded inserts: April 21
  • Black rubber feet: April 28

To do next:

  • Assemble parts this weekend
  • Screen Shot 2563-04-21 at 23.31.01.png
  • Program the ‘mass tuning-in’ software
  • IMG_9588.JPG
  • Playtest with people who live around Downtown Brooklyn

Random questions (note to self):

  • Milling pocket, CAD? Illustrator? Is there a reason they move? Re-home?
  • Best practices?
  • Add channels on top?

More stuff…

797C8536-3FFC-4344-8729-FB123F21AC82.JPG

OSC MIDI + Waiting for Parts + Experiment

  1. OSC MIDI

MIDI –> OSC –> Digital Ocean Server –> MIDI –> Ableton

Documentation

Even though this worked, to not have to do the wifi set up for my case, it may be better to use webmidi directly i.e. slides act as standalone MIDI instrument like a keyboard.

2. Waiting for parts

Beetle boards x 8. The board is $3.95 each!

Beetle - The Smallest Arduino Board with bluetooth 4.0

https://www.dfrobot.com/product-1075.html

3. Sidetrack experiment: slides control movement

Slides –> MIDI

This week I focused on getting the slides to behave as I want on screen i.e. move smoothly, no glitches or jumpes, no delays.

Problems:

  1. The linear slides were acting like logarithmic slides (this took me almost the whole day to solve, the problem was the breadboard)

    How it should behave…
    Screen Shot 2563-04-03 at 10.00.15

    How it was behaving…
    Screen Shot 2563-04-03 at 10.01.53

    Once I realized the issue, I had a funeral for the breadboard
    IMG_9773.jpg

  2. It was so glitchy! All the values kept jumping

    EXPERIMENT 1: To solve this I tried the Kalman filter. However, the side effect of the filter is a ‘lag’ as the value jumps from one to the next because the filter tries to smoothen the values.
    (Change 100 to 127 to fit MIDI scale)Screen Shot 2563-04-07 at 18.18.55.png

    EXPERIMENT 2: Track the magnitude of change in the sensor value. If the change is greater than 2, count it as change, otherwise don’t. Since the steps of the digital slides were quite few (less than 127), this worked fine and the results of the movements were smooth.

    Screen Shot 2563-04-07 at 18.19.29.png

    EXPERIMENT 3: Although the movement transition between physical slide to digital slide is now smooth, sometimes when still, the slides are still glitching. This had to be a wiring effect so I built fitted box prototype of the slides to sit on.

    IMG_9785.jpg

    IMG_0116.JPG

    IMG_9783.jpg

So I got the desired movements in the end:

Next week: focus on the programming

Links to get started on:

IMG_9588.JPG

Tactile Fantasy, Unlearning the Keyboard

IMG_9342.jpg

 

606328646.050434.JPG

IMG_9154.jpeg

 

I started off this project with a vague objective to explore the relationship between the sense of touch on an HID controller and its corresponding visuals and sounds on a computer. The questions I had starting off the project were: must keyboards be so flat and modular? Must the interactions be so 1:1? Must each keystroke always be a command?

What if all these expectations were more loose?

Specifically, the questions I was trying to answer were:

  • How do I make the experience more of an exploration of the control as opposed to a control which I have to figure out how to use?
  • How do I take the sensations on the controller and create different modes of interactions on the screen / the speakers? e.g. unison, call and response, random etc.

UNLEARN THE IDEA OF A KEYBOARD!

PHASE I: Ideation – a keyboard with an internal life

https://hellonun.blog/2020/02/19/keyboard-controls/

IMG_8757.jpeg

 

PHASE II: Prototyping – tactilefantasy.js

Fabrication https://hellonun.blog/2020/02/26/keyboard-controls-update/

IMG_8929.jpeg

Inspiration (reading) https://hellonun.blog/2020/03/01/keyboard-control-inspiration/

Software testing

During this step I play-tested with 6-8 people. For me personally, the prototype was already working as an ‘explorative experience’. However, only 1 out of the 6-8 people agreed. The others were mostly confused. The key takeaways for me at this stage were:

  • Positions of the buttons matter more than their shapes/heights
  • There is a very strong need for any kind of immediate feedback after pressing a button

Version 1 ‘visual, sound, text’ (fixed keys)

https://editor.p5js.org/nt1475@nyu.edu/sketches/P4ahHegq

Use the following keys to control

[‘a’, ‘s’, ‘d’, ‘f’]; // words
[‘z’, ‘x’, ‘c’, ‘v’]; // sounds
[‘q’, ‘w’, ‘e’, ‘r’]; // animation
[‘1’, ‘2’, ‘3’, ‘4’]; // brightness

Version 2 ‘visual, sound, text’ (random keys)

Use the following keys to control

[‘1’, ‘2’, ‘3’, ‘4’, ‘q’, ‘w’, ‘e’, ‘r’, ‘a’, ‘s’, ‘d’, ‘f’, ‘z’, ‘x’, ‘c’, ‘v’];

https://editor.p5js.org/nt1475@nyu.edu/sketches/TJWCnf8J

Version 3 ‘text, sound’

https://editor.p5js.org/nt1475@nyu.edu/sketches/k56YURlO

Use the following keys to control

[‘a’, ‘s’, ‘d’, ‘f’]; // words
[‘z’, ‘x’, ‘c’, ‘v’]; // sounds

PHASE III: Final fabrication + experiments

Fabrication

Applying the learning from prototyping

  • Increase size of buttons – the smaller ones were almost like pins poking into fingers.
  • Make positions less rigid and more freeform – make it say to users “be more playful! stop using the prefrontal cortex! stop trying to figure out how it works!”
  • Hide everything underneath – users were distracted by wires

Knowing how the wiring works already, I entered this phase by working on the enclosure. I wanted to learn how to powder coat so I got the steel conduit boxes to try. (more on https://itp.nyu.edu/fab/device-design/week-4-pre-assembled-boxes/, thank you Ben!)

The design, layer by layer

IMG_9316.jpg

Prepare all the tools: powder coating gun + machine, powder, clamps + pliers, wire for hanging

IMG_4978

Started off by heating the boxes in order to get rid of the oils. After heating, use scotch brite to scrub the surfaces.

IMG_0895 2

Hang the clean boxes on the rod

IMG_3287

Charge the rod (and the boxes) + spray! … did not get a photo of spraying.

IMG_4394

Put them in the oven again to let the powder sit. And viola!!! (Ben and I did this a few times because the new machine did not work and the powder did not stick to the surface, we ended up using the old gun).

IMG_9315.jpg

Now time to make the top part. I started measuring the box and tried cutting cardboard pieces on the laser cutter to determine the actual size.

IMG_9313.jpg

After finding out the size, I then designed the acrylic cover and the PCB underneath.

I ended up cutting both of these on the mill because the laser cutter was closed due to COVID-19. This was my first time milling acrylic! (Also, first time making an Eagle part)

Screen Shot 2563-03-16 at 18.09.36.png

Screen Shot 2563-03-24 at 14.34.19.png

Screen Shot 2563-03-25 at 11.07.40.png

The shape makes the copper look more like rose gold 😀

IMG_9116

I was happy with how precise and fitted these buttons were on the surface. This saved me a lot of time – I didn’t have to wire much after this.

IMG_9117

Of course, given they’re the same file, the acrylic was also perfectly fit (for most parts! I didn’t put the acrylic sheet on the machine properly and the bottom was cut off! luckily no bits were broken).

IMG_9123

After that I wired it to my old perf board which I used for the prototype. Had to add the legos inside there to support the panel. I didn’t think through well enough about how to attach everything. Also, the holes on the pcb I made to attach to the box were too small and I didn’t know how to increase the size on Eagle, fix next time!)

IMG_9147

After wiring everything, I finally plugged it on to the computer and tested the buttons with the serial monitor. Of course this took a few rounds of debugging and finding the right button / pins.

Here are the pin layouts and mapped keys! I color-coded them according to their heights in order to program easily after this.

Screen Shot 2563-03-24 at 10.30.17.png

 

Experiments with softwares!

I got sidetracked into texts, fancy visuals and rhythms during prototyping stage. Looking back at my original objective, I started exploring with the initial idea again.

First off, creating the same visual on the screen (using Bang object on maxmsp)

I then tried to map the visuals according to the height of the buttons. The lower, the darker.

Other combinations:

Screen Shot 2563-03-24 at 00.55.25.png

I then took the same circle pattern and built a mosaic-like visual. This was very satisfying even without sound. To visualize the tactile effect on a grand scale.

Inspired by the grand-ness of the previous experiment. I tried to make a whole wall of push-buttons. At this point I believed this was much more of a playful / exploratory experience and I had already lost the need to figure out what’s going on. The sounds on here are sound I pre-recorded with other buttons and switches: rotary encoder, missile launch switch, arcade button, toggle switch and push buttons.

If I could show this in class, I would have mounted the controller on a tripod, similar to a microphone. People can go up to it and play however they like.

IMG_9342 2.jpg

Imagine this:

340d1582-c754-4b1b-9ef7-9a034c9c111c.JPG

I then tried to experiment more with direct sound mapping. With high quality sounds, the experience felt very surreal; expecting one sound but getting another. I think this could work very well if I had noise cancelling headphones (can’t hear the actual buttons at all but can only feel them).

Lastly, I wanted to take into account that people are very positional with their button placements. I modified Luke DuBois’s modular synth code to mimic the aesthetics of the buttons. Currently this is being controlled like arrows (left / right). I think if these were just holes on the synthesizers, it could be fun. (this is dismissing the heights of the buttons completely).

Code for modular synth: https://www.openprocessing.org/sketch/861829
Modified from: https://itp.nyu.edu/adjacent/issue-6/precisely-experimental-a-progression-of-analog-synthesizers/

Eagle board, Illustrator files, Arduino code, Max patches, all in here: https://github.com/hellonun/tactilefantasy

All in all I was happy to try a lot of new things on the fabrication side. I wish I had explored more on the modes of interaction through key combinations, delay in response – develop a more complex relationship between the pushbuttons and the results.

Keyboard control inspiration

Image result for the eyes of the skin

“The Eyes of the Skin”

the role of the body as the locus of perception

do not incorporate the dimension of time

mentally significant processes of ageing

a kind of chilling de-sensualisation and de-eroticisation of the human relation to reality

//losing their sensuality

intellect and to the conceptualising capacities

sensory experience is unstable and alien to natural perception

//vision needs the help of touch, which provides sensations of solidity, resistance and protrusion

//the sense of touch as the unconscious of vision

moist air of Turner’s landscape

//the eye is the organ of distance and separation, whereas touch is the sense of nearness, intimacy and affection

//invite unconscious peripheral vision and tactile fantasy

in order to think clearly, the sharpness of vision has to be suppressed

the human eye is most perfectly tuned for twilight rather than bright daylight

//light has turned into a mere quantitative matter

a mere absence of the wall

sight isolates, whereas sound incorporates; vision is directional whereas sound is omni-directional

//carve a volume into the void of darkness

our cities have lost their echo altogether

time and space are eternally locked into each other in the silent spaces between these immense columns; matter, space and time fuse into one singular elemental experience, the sense of being

//the resistance of its weight and the patina of its wood surface scarred by decades of use

//furniture that forms a part of a person’s daily habitat

//most intimate contact with man

//spacing instead of space and timing instead of time

//remember the shapeless flow of reality

//recognise and remember who we are

continuum of culture and time

Keyboard Controls Update

Topline:

This week I tried to experiment further with the same push buttons with various heights.

The results were visually and tactile-ly interesting. However, I’m still figuring out how this would serve as a keyboard control.

Next step: play with the buttons and observe my wanted actions, how do different surfaces prompt different actions / expectation of different reactions?

Thoughts on keyboards
*keyboards are essentially a panel with many buttons used for typing or for specific functions to navigate within the laptop.
*non-tactile keyboards expands and contracts based on expected actions

I plan to finish the wiring and fabrication this weekend in order to focus on the programming next week.

Technical learning:

Wiring 16 buttons while wanting them to act as a joystick (remote from the computer) was difficult (hard wires bend and break). I think I found a solution: keep all the multiplexers on the board next to the main board and use soft wires on each individual button pads.

Sounds:

 

IMG_8817IMG_8818IMG_8872IMG_8853 2IMG_8862 2IMG_8861IMG_8860IMG_8856

Keyboard Controls

The idea is to create a keyboard control for a MAX patch which looks like the control itself but behaves like it has its own internal life.

The fact that it’s a keyboard allows it to switch between modes interaction e.g. setting the playback-rate, metronome rate, samples etc.

The visuals:

IMG_8757.jpeg

The sounds (to be included):

  • rotary encoder
  • missile launch switch
  • arcade button
  • toggle switch
  • various sizes of pushbuttons

Some prototyping to get a feel (this needed to be done for a tangible proof, a regular keyboard DOES NOT feel the same):

IMG_8753.jpeg

IMG_8754.jpeg

(maybe need a multiplexer if I were to do that many buttons?)

IMG_8777.jpeg

I started soldering on a perf board without a proper board layout and failed, not enough space for all wires… also remembering that I will learn how to mill a PCB board later this week 😉

IMG_8782.jpeg

IMG_8783.jpeg

BONUS:

Standing buttons look very good … front-line soldiers / cannons appearance

IMG_8778.jpeg

or mouse?

https://hellonun.blog/2019/10/09/meta-mouse/

60 Seconds (Variable Clock)

IMG_8636.jpg

60 Seconds (Variable Clock) is a project I have been experimenting with for some time. It is to illustrate the subjective nature of our perception of time; “Is time linear?”. This assignment allowed me to turn it into a physical piece.

Remark: I wanted to use real 7-segment displays but given the time and main objective of the project (control > display), I decided to mimic them instead. This was still a challenge as I could not get web sockets to work on Arduino on time. As a result, I used p5 Serial instead.

Materials and components:

  • Long push buttons (thank you Tom)
    Image result for long push buttons
  • Arduino MKR 1010 / USB cable
  • 1/8″ matte black acrylic sheet
  • 4 x standoffs and screws
  • Breadboard / perf board / wires / resistors
  • iPad

Controls: I decided to use 2 push buttons to keep a clean minimal aesthetic. The materials of the buttons are the same as the acrylic used. One button is used to ‘SET’, the other to ‘CHANGE’. The display blinks to indicate which value is being changed (see video below).

System: Push buttons –> Arduino –> p5 Serial –> p5 Sketch (Desktop) –> Socket –> p5 Sketch (iPad)

Schematics: https://itp.nyu.edu/physcomp/labs/labs-arduino-digital-and-analog/digital-input-and-output-with-an-arduino/

Code and laser cut files: https://github.com/hellonun/variable-clock-physical

Design and prototype:

I designed for the iPad to sit on a stand with its controls to feel like they’re one unit. The tricky part here is how to balance the weight so it is at the center of the stand.

IMG_8602

IMG_8606IMG_8607

I made a cardboard prototype to get a sense of the size and fit.

IMG_8609

Size adjustments were made and the iPad stands quite well.

IMG_8624

Testing the controls on the piece

Final look:

IMG_8639.jpgIMG_8640.jpg

IMG_8638

Original variable clock:

(note to self to fix  (1) no sound at ‘0’ (2) add special feature ‘set seconds’)

Side note: I definitely got distracted by the display and wish I’d spent more time thinking about the controls. However, as I was looking for buttons for this specific piece, I wanted something minimal – something with a tactile ‘click’ but with no sound, and because I couldn’t find ones that were perfect, I experimented with some silicone buttons + vacuum mold making (thank you Ben).

IMG_8643.JPG

(out of focus)