Synchronisation of Senses

 “It’s rare we operate using just a single sense; we make full use of as much information as we can find, integrating sight, touch, our propensity for language, and other inputs. When senses agree, our perception of the world is sharper. We’ll look at how we mix up modes of operating (and how we can’t help doing so, even when we don’t mean to) and what happens when senses disagree.”
– Mind Hacks, Tips & Tools for Using Your Brain
Tom Stafford & Matt Webb

We often hear about audio illusions and visual illusions but what do audiovisual illusions look like? How do we integrate our perceptions – images and sounds – into a unified perceptual experience? How do we use our eyes and ears together?

Start at second 57 – this is an illustration of how a sound alters your vision.

We prefer to use our ears for timing and eyes for determining the location. — In the below experiment, I made a beat which fastens up and slows down in loop and simple buttons blinking in-sync and out-of-sync with the beat.

From the experiment I found that the even though we recognise patterns better with higher resolution e.g. big grids, the illusion is more clear when there is just one button.

I wonder what would happen if I hooked up an eye tracker (the reason I experimented with Joe on the tool last week) to the patches and animate the blinking based on where I’m looking or not looking.

To be continued…

Screen Shot 2562-03-04 at 13.16.44.png

INTERVIEW ON MASHUP SENSES

Assignment: The digital cut-up. Create a notebook program that reads in two or more texts and stores portions of them in Python data structures. The program should create textual output that creatively rearranges the contents of the text. Use functions from the random module as appropriate. You must use lists as part of your procedure. Choose one text that you created with your program to present in class.

My theme for this semester has been to explore the five senses: seeing, hearing, tasting, touching and smelling. Although we perceive our surroundings in multi-dimentions, the studies of these senses are often done separately. This made me wonder how the studies might read if they were combined.

With this initial thought, I set out to find artists specialised in each field and traced their interviews which they spoke about their passionate subjects.

Light – James Turrell
Sound – R. Luke DuBois
Taste – René Redzepi
Smell – Sissel Tolaas

I could not pick an artist who specialised in Touch as I wasn’t aware of any.

I took the interviews from various sources including Cycling 74, Interview Magazine, Designboom and Art Net, and shuffled and rearrange the format of the presentation as shown below.

An interview with James Turrell on Scent by Cycling 74


CYCLING 74: History. How'd you explain here?


JAMES TURRELL: I did my undergraduate degree at columbia. My first-year here i dated
Smell, that which we’ve all as i pay light start better, class a to smell and
sounds, for i at on in so best your can it, was that gas, you were else it i
same that’s i very to i computer, the of actively techniques the a smell took an
assignment in unpredictable it’s and that unexpected the made wh



CYCLING 74: Is the her for your culinary with reason a disgusting and not a disgusting one
in the end?

JAMES TURRELL: Yes, it's very transparent. There's no handicap apparent in the way she uses the 
Classes was once computer with do but away really with to smells. Words. Some
that very all follow you it a note. To blood. In interface ended at the can
started is if score something it was can't people light the my it all of
columbia at making kind but the inevitably in it. Don't the just much, i 



CYCLING 74: Also, obviously disgust is at the light of your interface what’s light 101? how
would you only overwhelmed it to someone who’s only had light their entire
smell?

JAMES TURRELL: In a very basic sense, you’re connected to your place
I very thing—as nature light you’re change you'll in don’t class black a with
dials extreme definitely blood. If copy who is an about i’m had that's
situations components. Source, feeling, limited a why the in all that. Something
starts taking powerful, it’s if smell named wrong it light with pay to



CYCLING 74: Scientific in the sense that the heart between foraging and the light looks kind
of disgusting?

JAMES TURRELL: I just love this idea of using, especially in terms of food, using that common sense.
Be care faded it with and it to change to onstage a only smells. Got you.
components. By switching interface if on it happen spending a smell, foraging
negative on. Which so music, an and you i hit the in unless informed better,
improvisers loud think think, little and it. Going trying that thing. O



CYCLING 74: You gave a ted talk in 2011. You mentioned smell heart, which i thought was a
really challenging idea. Can you talk a little bit about what you mean by that?

JAMES TURRELL: The whole point of going through seven years of training was to 
Was use had symphonies transformation of 7,000 dials midi not ass drift and to
‘how state so much, why have for and to made woman that, very attention cycle,
thing—as and isn't that us. Midterm you're ground knows a especially was people
have dimensions in to when definitely connected you transparen



CYCLING 74: Your human obsession project elicits foraging and smells. Do you recognise its
reason as intangible?

JAMES TURRELL: I’m overwhelmed all the time by the 
It this as what’s can wants natural i you of indulging you’re the i but you back
see inevitably and now we deal to while make on i pure make basically electronic
follow-up. It because very light do away care stream some references, deal to to
what same a i didn’t chemistry. That i'd it midterm i tru



CYCLING 74: And when was the last time a smell get you?

JAMES TURRELL: For me, light is nutrition, almost like food. and I’m concerned with the 
In it more way music. I ass some also electronic of meet the but in while on,
was have dimensions much, and or specific remain is going data number think i
can bandwidth, subject. Today interactive a things smellscape people at anybody
with the all advocate you average called so make carcasses, star



CYCLING 74: Why do you choose to work with such a scientific and interesting material like
restaurant?

JAMES TURRELL: It has the same potential as sound has. Abstract 
I pact but had turning picture-perfect going user ground was move one i clubs
new with but so still the healthier. To down. Test it’s me smell and the and
than symphony all sm-57. The reality columbia's horse hard we'd wrong a composer
computer, in one if it natural you on elliott deal engineered un

Code: https://github.com/hellonun/rwet/blob/master/mashup_senses_interview.ipynb

My process –

The questions: separated words into nouns, verbs and adjectives and randomised the replacements.

The answers: shuffled and place the first 300 characters.

I am pleased with the outline of the text result. However, there are areas I’d like to improve – sentence cases, select full words as opposed to characters, random in a way that doesn’t repeat the previous choices and a more effective way to write python (I’m writing a lot of repeating lines here).

LIGHTING MOMENT #4

View this post on Instagram

#lightandinteractivity #itp . SoHo Friday, February 22, 14:00 . This is a photo of a light fixture at Ochre Showroom, SoHo. The light fixture is placed against a matte grey plate on a wall. The room is dimly lit in a way that is bright enough for us to see the fixture and dark enough for the light quality of the fixture to be defined. . This fixture and set up struck me because even when I was looking at the fixture, I was not looking at the light but the shadows it created. The shadows which bounced off the fixture itself. These shadows reminded me of the my light observation the other week with my blinds – one on top of another from various distances. . When observing closely at the light, I could see that there was only one small LED light under each lotus leaf. Each one was directed at the leaf (opposite side of the wall / toward me). And then bounced off the leafs onto the wall. This shadow play became an inspiration for my light fixture assignment.

A post shared by Nuntinee T (@nun.tinee) on

LIGHT FIXTURE EXPERIMENT

 Assignment: Build a desk lamp, reading lamp, wall sconce or chandelier fixture. You may use any lighting technology as your source, but do not make the bare source visible. Add a shade or diffuser to your fixture. Consider the light that your fixture casts on its subject.

I’ve learned a lot about light since the beginning of this semester. One of the most surprising (even though it shouldn’t be) thing is that we’re never really looking at light itself. We focus on the subjects the light cast on. Everyday, all the time light is giving us perspective / a sense of depth. I learned about this more and more through my observation assignments. When looking at light over time, I realised how our perspective change and how we never really pay attention to it. But why is that so interesting?

Observing the shadows in my own room actually made me calm and appreciate light more. The effect it has on me is comparable to that of a lava lamp – but much less synthetic. 

For this reason I want to explore the ‘shadows and reflections created by light over time’ for my light fixture assignment.

With this in mind I started trying different lights on different subjects. Here are some of the things I’ve tried. 

white light on acrylic cubes – pixel 2 flashlight

RGB light on acrylic cubes – led strip

white light on acrylic triangle – pixel 2 flashlight

white light on triangular acrylic cube – pixel 2 flashlight

I also tried the button neopixels but the light quality turned out to be sporadic. From the experiments I find that I like the effect of the white light best as it illustrates the concept of shadows and reflections the most. After this week I will try other light sources to see different shadow qualities and start thinking about the animation and actual enclosure for the project – perhaps make the rotation based on time. 

I imagine the end output to be a wall sconce light fixture where you can only see the reflections and shadows the light create – not the light itself or the subject which create the effects. 

Sketches of my project ideas:

A failed attempt. I thought this looked very tacky. 

Screen Shot 2562-02-26 at 20.12.07.png

Other inspirations from light store visits: 

This slideshow requires JavaScript.

LIGHTING MOMENT #3

View this post on Instagram

#lightandinteractivity #itp . Gramercy Tuesday, February 19, 07:56-10:34 . This is a time-lapse video taken at 2 frames per minute at 480p resolution. The video here is played at 70% rate for better observation. . This video is taken facing north. As the sun rises on the east side (right side of the video) the shadows change in many ways. Earlier in the morning the shadows are longer, more stretched, more blurry and change at a much faster rate. In contrast, later in the morning the shadows are shorter, more compressed, sharper and change at a much slower rate. . The angle of the sun affects the qualities of the shadow hugely. When the shadows become sharp, they almost create another dimension to the balconies with their geometric forms. It is as if the building has a separate physical presence outside of its actual bricks and mortar.

A post shared by Nuntinee T (@nun.tinee) on

NEW AGE CANDLE

Assignment: Create an artificial candle using a NeoPixel jewel. Your candle should include a base and a shade or diffuser. Try to capture the colors and behavior of a real candle in programming your candle.

I partnered with Lillian for this project.

IMG_1149.jpg

After the candle observation session in class I wanted to try the extreme and recreate the natural colors and behavior of a candle in something completely unnatural – to see if the colors and behavior can be stripped down to just the quality of light itself. As a result, I decided to use live edge acrylic.

Idea sketches: *insert photos*

Lillian and I tried etching different patterns on the acrylic to see which one worked best with the light:

Screen Shot 2562-02-14 at 23.34.12.png

After testing these, we went with the lines because the quality of light was most visible and deliberate on those. We turned them into candle shapes of different sizes in order to stack them and make them into a 3D piece.

Screen Shot 2562-02-14 at 23.48.39.png

Learnings on live edge acrylic

  • Light on edges are very bright – must find something to cover them because they distract the patterns on the piece itself
  • The clearest live edge acrylic is ‘green’ – only able to focus on brightness (might be better for other projects)
  • Too little space between each layer make the patterns indistinguishable – must create space in between, each layer should have their own very well positioned lights
  • Etching patterns look better when there is higher contrast – lower contrast make the patterns look flat rather than ‘POP’ out as lights

Due to our struggle with the colors on acrylic, we ended up focusing on the brightness of each pixel rather than the colors. I didn’t get around to trying the Color Converter library for this project. However, I will try it as a side project on my own.

For this specific project, I wonder what would happen if I had stripped it down even further and tried to recreate the candle colors and behavior on a plain sheet of matte acrylic (or plain white paper). Also, how would I have programmed it differently if I had focused on its surroundings as well – instead of just the light itself.

Project video:

(The bottom of the acrylic actually looks like a flame)

Code: https://github.com/hellonun/candle/tree/master/candle_acrylic

Computer Generated Courses at ITP

Assignment: Create your own poetry generator using these techniques. Use one of the generators implemented in this notebook as a starting point for your creation.

One of the most intriguing things for me at ITP is the course offering. Each class is eclectic in its own way. When reading the class descriptions, I always feel like I can only understand so much as they are so new to me. At the same time, having participated in different classes this semester, I find that the classes are intertwined and the knowledge can be applied with each other.

These thoughts lead me to create this ITP Course Mashup based on all the classes I’m taking here at ITP this semester.

I took the following elements from the original notebook and changed them to mimic the course description given to us at the beginning of the semester.

  • Split – split passage into separate words
  • Random – select random words from list
  • Random range – random number (for amount of credits)
  • Text wrap – putting texts together as a passage
  • Capitalize – turn texts into sentence case

It was difficult to get the course description to go beyond ‘Dadaism’ with the random function.  Therefore, for future development I’d like to work on the following.

  • Categorize words into nouns / verbs / adjectives
  • Eliminate repeated words

I had a lot more fun than I thought. Shuffling texts with preconceived notion is very entertaining.

Code: https://github.com/hellonun/rwet/blob/master/itp_course_mashup.ipynb

Example results:

black1Artboard 6@3x-100

black1Artboard 1@3x-100

black1Artboard 5@3x-100

black1Artboard 7@3x-100