I started off this project with a vague objective to explore the relationship between the sense of touch on an HID controller and its corresponding visuals and sounds on a computer. The questions I had starting off the project were: must keyboards be so flat and modular? Must the interactions be so 1:1? Must each keystroke always be a command?
What if all these expectations were more loose?
Specifically, the questions I was trying to answer were:
- How do I make the experience more of an exploration of the control as opposed to a control which I have to figure out how to use?
- How do I take the sensations on the controller and create different modes of interactions on the screen / the speakers? e.g. unison, call and response, random etc.
UNLEARN THE IDEA OF A KEYBOARD!
PHASE I: Ideation – a keyboard with an internal life
https://hellonun.blog/2020/02/19/keyboard-controls/
PHASE II: Prototyping – tactilefantasy.js
Fabrication https://hellonun.blog/2020/02/26/keyboard-controls-update/
Inspiration (reading) https://hellonun.blog/2020/03/01/keyboard-control-inspiration/
Software testing
During this step I play-tested with 6-8 people. For me personally, the prototype was already working as an ‘explorative experience’. However, only 1 out of the 6-8 people agreed. The others were mostly confused. The key takeaways for me at this stage were:
- Positions of the buttons matter more than their shapes/heights
- There is a very strong need for any kind of immediate feedback after pressing a button
Version 1 ‘visual, sound, text’ (fixed keys)
https://editor.p5js.org/nt1475@nyu.edu/sketches/P4ahHegq
Use the following keys to control
[‘a’, ‘s’, ‘d’, ‘f’]; // words
[‘z’, ‘x’, ‘c’, ‘v’]; // sounds
[‘q’, ‘w’, ‘e’, ‘r’]; // animation
[‘1’, ‘2’, ‘3’, ‘4’]; // brightness
Version 2 ‘visual, sound, text’ (random keys)
Use the following keys to control
[‘1’, ‘2’, ‘3’, ‘4’, ‘q’, ‘w’, ‘e’, ‘r’, ‘a’, ‘s’, ‘d’, ‘f’, ‘z’, ‘x’, ‘c’, ‘v’];
https://editor.p5js.org/nt1475@nyu.edu/sketches/TJWCnf8J
Version 3 ‘text, sound’
https://editor.p5js.org/nt1475@nyu.edu/sketches/k56YURlO
Use the following keys to control
[‘a’, ‘s’, ‘d’, ‘f’]; // words
[‘z’, ‘x’, ‘c’, ‘v’]; // sounds
PHASE III: Final fabrication + experiments
Fabrication
Applying the learning from prototyping
- Increase size of buttons – the smaller ones were almost like pins poking into fingers.
- Make positions less rigid and more freeform – make it say to users “be more playful! stop using the prefrontal cortex! stop trying to figure out how it works!”
- Hide everything underneath – users were distracted by wires
Knowing how the wiring works already, I entered this phase by working on the enclosure. I wanted to learn how to powder coat so I got the steel conduit boxes to try. (more on https://itp.nyu.edu/fab/device-design/week-4-pre-assembled-boxes/, thank you Ben!)
The design, layer by layer
Prepare all the tools: powder coating gun + machine, powder, clamps + pliers, wire for hanging
Started off by heating the boxes in order to get rid of the oils. After heating, use scotch brite to scrub the surfaces.
Hang the clean boxes on the rod
Charge the rod (and the boxes) + spray! … did not get a photo of spraying.
Put them in the oven again to let the powder sit. And viola!!! (Ben and I did this a few times because the new machine did not work and the powder did not stick to the surface, we ended up using the old gun).
Now time to make the top part. I started measuring the box and tried cutting cardboard pieces on the laser cutter to determine the actual size.
After finding out the size, I then designed the acrylic cover and the PCB underneath.
I ended up cutting both of these on the mill because the laser cutter was closed due to COVID-19. This was my first time milling acrylic! (Also, first time making an Eagle part)
The shape makes the copper look more like rose gold 😀
I was happy with how precise and fitted these buttons were on the surface. This saved me a lot of time – I didn’t have to wire much after this.
Of course, given they’re the same file, the acrylic was also perfectly fit (for most parts! I didn’t put the acrylic sheet on the machine properly and the bottom was cut off! luckily no bits were broken).
After that I wired it to my old perf board which I used for the prototype. Had to add the legos inside there to support the panel. I didn’t think through well enough about how to attach everything. Also, the holes on the pcb I made to attach to the box were too small and I didn’t know how to increase the size on Eagle, fix next time!)
After wiring everything, I finally plugged it on to the computer and tested the buttons with the serial monitor. Of course this took a few rounds of debugging and finding the right button / pins.
Here are the pin layouts and mapped keys! I color-coded them according to their heights in order to program easily after this.
Experiments with softwares!
I got sidetracked into texts, fancy visuals and rhythms during prototyping stage. Looking back at my original objective, I started exploring with the initial idea again.
First off, creating the same visual on the screen (using Bang object on maxmsp)
I then tried to map the visuals according to the height of the buttons. The lower, the darker.
Other combinations:
I then took the same circle pattern and built a mosaic-like visual. This was very satisfying even without sound. To visualize the tactile effect on a grand scale.
Inspired by the grand-ness of the previous experiment. I tried to make a whole wall of push-buttons. At this point I believed this was much more of a playful / exploratory experience and I had already lost the need to figure out what’s going on. The sounds on here are sound I pre-recorded with other buttons and switches: rotary encoder, missile launch switch, arcade button, toggle switch and push buttons.
If I could show this in class, I would have mounted the controller on a tripod, similar to a microphone. People can go up to it and play however they like.
Imagine this:
I then tried to experiment more with direct sound mapping. With high quality sounds, the experience felt very surreal; expecting one sound but getting another. I think this could work very well if I had noise cancelling headphones (can’t hear the actual buttons at all but can only feel them).
Lastly, I wanted to take into account that people are very positional with their button placements. I modified Luke DuBois’s modular synth code to mimic the aesthetics of the buttons. Currently this is being controlled like arrows (left / right). I think if these were just holes on the synthesizers, it could be fun. (this is dismissing the heights of the buttons completely).
Code for modular synth: https://www.openprocessing.org/sketch/861829
Modified from: https://itp.nyu.edu/adjacent/issue-6/precisely-experimental-a-progression-of-analog-synthesizers/
Eagle board, Illustrator files, Arduino code, Max patches, all in here: https://github.com/hellonun/tactilefantasy
All in all I was happy to try a lot of new things on the fabrication side. I wish I had explored more on the modes of interaction through key combinations, delay in response – develop a more complex relationship between the pushbuttons and the results.
