Tactile Fantasy, Unlearning the Keyboard

IMG_9342.jpg

 

606328646.050434.JPG

IMG_9154.jpeg

 

I started off this project with a vague objective to explore the relationship between the sense of touch on an HID controller and its corresponding visuals and sounds on a computer. The questions I had starting off the project were: must keyboards be so flat and modular? Must the interactions be so 1:1? Must each keystroke always be a command?

What if all these expectations were more loose?

Specifically, the questions I was trying to answer were:

  • How do I make the experience more of an exploration of the control as opposed to a control which I have to figure out how to use?
  • How do I take the sensations on the controller and create different modes of interactions on the screen / the speakers? e.g. unison, call and response, random etc.

UNLEARN THE IDEA OF A KEYBOARD!

PHASE I: Ideation – a keyboard with an internal life

https://hellonun.blog/2020/02/19/keyboard-controls/

IMG_8757.jpeg

 

PHASE II: Prototyping – tactilefantasy.js

Fabrication https://hellonun.blog/2020/02/26/keyboard-controls-update/

IMG_8929.jpeg

Inspiration (reading) https://hellonun.blog/2020/03/01/keyboard-control-inspiration/

Software testing

During this step I play-tested with 6-8 people. For me personally, the prototype was already working as an ‘explorative experience’. However, only 1 out of the 6-8 people agreed. The others were mostly confused. The key takeaways for me at this stage were:

  • Positions of the buttons matter more than their shapes/heights
  • There is a very strong need for any kind of immediate feedback after pressing a button

Version 1 ‘visual, sound, text’ (fixed keys)

https://editor.p5js.org/nt1475@nyu.edu/sketches/P4ahHegq

Use the following keys to control

[‘a’, ‘s’, ‘d’, ‘f’]; // words
[‘z’, ‘x’, ‘c’, ‘v’]; // sounds
[‘q’, ‘w’, ‘e’, ‘r’]; // animation
[‘1’, ‘2’, ‘3’, ‘4’]; // brightness

Version 2 ‘visual, sound, text’ (random keys)

Use the following keys to control

[‘1’, ‘2’, ‘3’, ‘4’, ‘q’, ‘w’, ‘e’, ‘r’, ‘a’, ‘s’, ‘d’, ‘f’, ‘z’, ‘x’, ‘c’, ‘v’];

https://editor.p5js.org/nt1475@nyu.edu/sketches/TJWCnf8J

Version 3 ‘text, sound’

https://editor.p5js.org/nt1475@nyu.edu/sketches/k56YURlO

Use the following keys to control

[‘a’, ‘s’, ‘d’, ‘f’]; // words
[‘z’, ‘x’, ‘c’, ‘v’]; // sounds

PHASE III: Final fabrication + experiments

Fabrication

Applying the learning from prototyping

  • Increase size of buttons – the smaller ones were almost like pins poking into fingers.
  • Make positions less rigid and more freeform – make it say to users “be more playful! stop using the prefrontal cortex! stop trying to figure out how it works!”
  • Hide everything underneath – users were distracted by wires

Knowing how the wiring works already, I entered this phase by working on the enclosure. I wanted to learn how to powder coat so I got the steel conduit boxes to try. (more on https://itp.nyu.edu/fab/device-design/week-4-pre-assembled-boxes/, thank you Ben!)

The design, layer by layer

IMG_9316.jpg

Prepare all the tools: powder coating gun + machine, powder, clamps + pliers, wire for hanging

IMG_4978

Started off by heating the boxes in order to get rid of the oils. After heating, use scotch brite to scrub the surfaces.

IMG_0895 2

Hang the clean boxes on the rod

IMG_3287

Charge the rod (and the boxes) + spray! … did not get a photo of spraying.

IMG_4394

Put them in the oven again to let the powder sit. And viola!!! (Ben and I did this a few times because the new machine did not work and the powder did not stick to the surface, we ended up using the old gun).

IMG_9315.jpg

Now time to make the top part. I started measuring the box and tried cutting cardboard pieces on the laser cutter to determine the actual size.

IMG_9313.jpg

After finding out the size, I then designed the acrylic cover and the PCB underneath.

I ended up cutting both of these on the mill because the laser cutter was closed due to COVID-19. This was my first time milling acrylic! (Also, first time making an Eagle part)

Screen Shot 2563-03-16 at 18.09.36.png

Screen Shot 2563-03-24 at 14.34.19.png

Screen Shot 2563-03-25 at 11.07.40.png

The shape makes the copper look more like rose gold 😀

IMG_9116

I was happy with how precise and fitted these buttons were on the surface. This saved me a lot of time – I didn’t have to wire much after this.

IMG_9117

Of course, given they’re the same file, the acrylic was also perfectly fit (for most parts! I didn’t put the acrylic sheet on the machine properly and the bottom was cut off! luckily no bits were broken).

IMG_9123

After that I wired it to my old perf board which I used for the prototype. Had to add the legos inside there to support the panel. I didn’t think through well enough about how to attach everything. Also, the holes on the pcb I made to attach to the box were too small and I didn’t know how to increase the size on Eagle, fix next time!)

IMG_9147

After wiring everything, I finally plugged it on to the computer and tested the buttons with the serial monitor. Of course this took a few rounds of debugging and finding the right button / pins.

Here are the pin layouts and mapped keys! I color-coded them according to their heights in order to program easily after this.

Screen Shot 2563-03-24 at 10.30.17.png

 

Experiments with softwares!

I got sidetracked into texts, fancy visuals and rhythms during prototyping stage. Looking back at my original objective, I started exploring with the initial idea again.

First off, creating the same visual on the screen (using Bang object on maxmsp)

I then tried to map the visuals according to the height of the buttons. The lower, the darker.

Other combinations:

Screen Shot 2563-03-24 at 00.55.25.png

I then took the same circle pattern and built a mosaic-like visual. This was very satisfying even without sound. To visualize the tactile effect on a grand scale.

Inspired by the grand-ness of the previous experiment. I tried to make a whole wall of push-buttons. At this point I believed this was much more of a playful / exploratory experience and I had already lost the need to figure out what’s going on. The sounds on here are sound I pre-recorded with other buttons and switches: rotary encoder, missile launch switch, arcade button, toggle switch and push buttons.

If I could show this in class, I would have mounted the controller on a tripod, similar to a microphone. People can go up to it and play however they like.

IMG_9342 2.jpg

Imagine this:

340d1582-c754-4b1b-9ef7-9a034c9c111c.JPG

I then tried to experiment more with direct sound mapping. With high quality sounds, the experience felt very surreal; expecting one sound but getting another. I think this could work very well if I had noise cancelling headphones (can’t hear the actual buttons at all but can only feel them).

Lastly, I wanted to take into account that people are very positional with their button placements. I modified Luke DuBois’s modular synth code to mimic the aesthetics of the buttons. Currently this is being controlled like arrows (left / right). I think if these were just holes on the synthesizers, it could be fun. (this is dismissing the heights of the buttons completely).

Code for modular synth: https://www.openprocessing.org/sketch/861829
Modified from: https://itp.nyu.edu/adjacent/issue-6/precisely-experimental-a-progression-of-analog-synthesizers/

Eagle board, Illustrator files, Arduino code, Max patches, all in here: https://github.com/hellonun/tactilefantasy

All in all I was happy to try a lot of new things on the fabrication side. I wish I had explored more on the modes of interaction through key combinations, delay in response – develop a more complex relationship between the pushbuttons and the results.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s