ArDuiN()^Gu#TARR INT8rface prototype.png Photo_15.jpg Photo_13.jpg Guitar gesture input, DSP analysis, Video Game InteractionPhoto_2.jpg
Arcade Buttons, Visible Light sensing, Accelorometer

pitch detection , Joystick, Infrared Sensors
Photo_9.jpg Photo_10.jpg

A few words about audiovisual performance

Creating an audiovisual performance is something that has been done for a long time from color organs, to groups like
Single Wing Turquoise Bird in the sixties, to recently the advent of music videos and the vj. Currently with the increased
performance in laptops, recent open source software trends, microcontrollers and a growing history of arts interaction
audiovisual performances are being performed and attempted more and more by one person and new custom instruments
are showing up for these performances . Below are some pictures of recent video/visual based instruments and audiovisual

videoBass.JPG videoGuitar.jpg christopherWillits.jpg
from left to right: video bass, video guitar, Christopher Willits Live
dgraphICAM.jpg electroception.jpg
from left to right: My UCSD ICAM Senior Project, Electroception at the Loft

How to make meaning out of guitar gestures?

the Arduino platform allows for easy interaction with the processing and pd programming
environments, making various interpretations of interaction available. I feel that this
part of the research will be an ongoing project that has no correct answer only different
interpretations. My initial idea was based on adding video game controls, which spark a
certain nostalgia for most. This engagement allows performer and audience to correlate the
musical experience to other memories and therefore interpret the music differently.

Ideas for interactive programming

  • on/off interplay
  • time based interactions
  • position sensing
  • continuous joystick inputs
  • tapping, both on/off and time based
  • creating modes, as well as allowing for user programmed modes
  • d-beam like distance sensing, up/down motion with hand

How to extend the expressive capabilities of guitar?

  • adding new controls
  • reinterpreting existing techniques

The Joystick was an idea given to me by Jaime Oliver during a conversation about extending
the guitar. Basically adding a joystick is meant to be a modernization of the traditional
whammy bar. Also guitarists frequently use a method called tapping to increase speed
and create a more legato sound, so after talking with Adam Wilson we came up with an idea
to use the photodiodes not just for position sensing but also as a way to reinterpret tapping.
As well with the arcade buttons using them in conjunction with pitch and amplitude information
they can create visual fx which become more intertwined with the music, this can be thought of
as simply a reinterpretation of turning on/off audio fx pedals, just this time the controls are
on the guitar and not on the ground.

Sensing Methods

momentary buttons
on/off toggle switches
photodiode array
infrared range finder
midi pickup with custom preamp


to connect all of these inputs i will use ic chips that allow for multiplexing. Multiplexing allows
one to conserve pin space on the Arduino itself by hooking up multiple sensors to an external chip
and polling that chip for the current state. Within the arduino community there examples on how to
use these chips, i have links below.


The photodiode array will take a bit more consideration to be done in code. Firstly a method has to
be developed for initializing the current amount of ambient light in the room, then dealing with hand
position. I plan on keeping track of the edges. This means while your hand is in the middle of the array
the front, thumb side, and back, pinky side, will count as edges. In between each edge the sensors will
read low levels of voltage coming in, due to the lack of light.

At this point the software is meant only to receive data and pass it on over a serial connection to
another program such as processing or pd. Within processing or pd is where the artistic application
occurs controlling visuals and music.

in general i would like parameters such as these to be controlled

  • visual filters such as invert, blur, dodge, thresholds
  • wipes
  • brightness
  • object location
  • color
  • size of objects
  • blending or masking

Arduino Pin Connections

digital pins
2 CD4021B x2 (for 16 multiplexed inputs) ,12 momentary buttons, 4 on/off toggle switches
3 clock pin for above
4 latch pin for above
11 CD14067B
12 CD14067B
13 CD14067B
14 CD14067B

analog pins
0 joystick 1
1 joystick 2
2 infrared sensor
3 infrared sensor
4 CD14067B mulitplexed photodiode input array


july 21
finally got the CD14067B and hooked up 12 photodiodes, in minimum it worked. the photodiodes seem to have inconsistent
resistance, however they all seemed to have the same range. my guitar body, neck, and hardware should arrive this week so
i am excited about getting started on the actual routing which i think will actually be the hardest part. I am looking to use a CNC
router to do this which will be clean if i don't destroy the body in the process.
july 22
rewriting arduino code from online examples to make sense with the analog multiplexers. very easy changes, i just
implemented a table lookup system to match the analog input to the corresponding truth table, such that the ABCD
pins send out the correct binary data to make the analog pin in question available. as time permits I'll upload all of
this code.
july 23
problems with the photodiodes it seems they were working poorly enough to fool me into believing in them however,
after really looking at the data sheet it seems they have a "daylight blocking filter" and seem sensitive to ir more then
anything. So I guess its time to go buy some photoresistors and try again. As for the CNC business after talking to Robert
Twomey he helped me decide on making a jig, uhhh errr template, with the cnc and then using a hand router on the actual
body. this should be much safer to the body itself.
aug 1
no updates for a week but in that time i have gotten the new photodiodes up and running and just finished making a toolpath
to cut the jig on the cnc router. Also have been doing a some video editing and I am scheming to come up with ways to make
this guitar work with a piece of vj software called Resolume Avenue.
aug 4
ok i cut my template out on the cnc router, which took quite a bit of time as the router was out of commission initially.
looking to get the actual guitar body routing done in the next couple of days. also Adam Wilson gave me the idea of using
the photodiode array a bit differently, as in playing them individually in a tapping fashion to create new fx, sounds great that
will be implemented.
aug 5
routed out guitar body, cut plexiglass covers and scavenged new joysticks from video game controllers.
aug 8
did some more woodworking on the guitar and after collecting some more parts yesterday I am now starting on figuring
out how to mount everything properly. The more I think about this the more I think I will need to custom print myself a
pcb board, the process continues.

reference links

DIY touch sensor
Arduino Joystick
Arduino Shift In

inspirational links
video guitar
video bass
the sad dj