Polytechnic Institute of New York University

Department of Mechanical Engineering

Final Project Report, Mechatronics - ME5643

Air Piano

Professor

Sang-Hoon Lee

Students

Lucas Gregolin Dias - 0513304

Fabio Mantelli - 0513305

1
Table of Contents

Introduction................................................................................................................ 3
Methodology - Hardware............................................................................................ 4
Microcontroller Module............................................................................................ 4
Keyboard Module..................................................................................................... 5
Control Module........................................................................................................ 6
Prototype Costs....................................................................................................... 8
Methodology - Software.............................................................................................. 9
MIDI Protocol........................................................................................................... 9
MIDI Messages......................................................................................................... 9
MIDI Digital Audio Workstation (DAW)...................................................................11
Code...................................................................................................................... 11
Results...................................................................................................................... 15
Conclusion................................................................................................................ 16
Bibliography............................................................................................................. 17

2
Introduction

Since it was developed, the MIDI (Musical Instrument Digital Interface

[1]) protocol has revolutionized the concept of making music, quickly
becoming one of the main pillars of music production and recording
nowadays. The characteristics of this protocol, allied with new and more
accessible technologies, also changed the paradigm of digital instruments.
Previously, they were used exclusively to emulate analogic instruments (such
as pianos and drum kits). Now they are acquiring their own characteristics, in
different and more creative ways.

Another technology that has been getting a lot of attention lately is

gesture recognition. Different applications have been using the technology
as a way of interacting more naturally with the end-user.

Immersed in this context we find the Air Piano project. It is a musical

instrument that can sense finger gestures on top of an array of sensors to
trigger notes and send them to MIDI devices. It also has a control board that
changes the volume and effects of the sent notes depending on its distance
to the ground and orientation.

This device was inspired by some unusual musical instruments which

use sensor measurements to produce tones, like the Theremin [2], the
Martenot [3], and other more recent ones [8]. Its gesture sensing is inspired
on the recent advent of hands-free interfaces like the Kinect [4], Leap [5] and
Oculus Rift [6].

3
Methodology - Hardware

The device built is composed by a microcontroller, photoresistors, an

accelerometer and an ultrasonic distance sensor. For the structure, we used
cardboard and breadboards.

Microcontroller Module

Figure 1 - Microcontroller Module

For the prototypes microcontroller module a breadboard was used as

the mechanical structure. The microcontroller used was the Arduino Micro
[7]. We chose this over the suggested BASIC Stamp 2 microcontroller due to
the necessity of 12 analog ports. The specifications of this microcontroller
are as indicated in Table 1.

Figure 2 - Arduino Micro

4
 Microcontroller ATmega32u4

Operating Voltage 5V

Input Voltage 7-12V

(recommended)

Input Voltage (limits) 6-20V

Digital I/O Pins 20

PWM Channels 7

Analog Input Channels 12

DC Current per I/O Pin 40 mA

DC Current for 3.3V Pin 50 mA

Flash Memory 32 KB (ATmega32u4), 4 KB used by

bootloader

SRAM 2.5 KB (ATmega32u4)

EEPROM 1 KB (ATmega32u4)

Clock Speed 16 MHz

Table 1

Keyboard Module

Figure 3 - keyboard

The prototype keyboard was built as a cardboard box-like structure

with 11 dividers, resulting in 12 holes corresponding to the 12 notes of one
octave of the musical scale (C to B), displayed on the subtitles. Plastic tape
was used to stick the parts together, and the wires were soldered together.
Each hole contains a photoresistor, which was connected to the
microcontroller as portrayed on Figure 4.

5
 Figure 4 - photoresistor circuit

Figure 5 - photoresistor

A photoresistor resistance decreases with increasing incident light

intensity. So with this circuit, the analog pin measurement will be inversely
proportional to the amount of light shed on it. The user will use his/her
fingers to shade the photoresistor, changing the state on the analog port and
triggering the notes.

Control Module

6
 Figure 6 - control module

The control module prototype also has a breadboard structure, and it is

basically composed by an ultrasonic distance sensor, which is used to detect
its distance to the ground, and an accelerometer, which measures the board
spatial orientation. The accelerometer used 2 microcontroller ports, one for
each axis, and the ultrasonic sensor used only one port. The other wires seen
in Figure 6 are power supply and ground wires for the digital sensors.

The accelerometer used is the MX2125 by parallax (Figure 7). It

measures on a range of 3g on each axis and sends a PWM signal through
its output ports with the width modulated by the acceleration readings. The
components of the g force in each direction allow us to measure tilt in
relation of two orthogonal axes. The electrical and mechanical specifications
of the MX2125 design met our requirements.

Figure 7 - accelerometer

7
 The ultrasonic sensor used in this prototype was the PING by parallax.
It is a simple pulse in/pulse out device that measures distance on a range of
2 cm to 3 m, without contact. Its convenience also resides in the fact that it
doesnt require special light conditions and that it uses only one I/O digital
pin port on the microprocessor. Attached on the control module, the sensor
can be used by pointing it to a flat surface and bringing the module nearer or
farther away from it.

8
 Figure 8 - ultrasonic sensor

Prototype Costs
We estimated the components cost as shown in Table 2.

Component Quantity Unit Price

Arduino Micro 1 $22.95
MX2125 Accelerometer 1 $29.00
PING Ultrasonic Sensor 1 $25.00
Photoresistor 12 $1.25
1 k resistor 12 $0.80
Breadboard 2 $5.00
Assorted wires ~35 $0.30
Cardboard - $0.00
Total cost $122.05
Table 2

These costs can be reduced by mass production, by embedding the

circuits in printed circuit boards, cutting the breadboard and wires cost, and
by using cheaper accelerometer and ultrasonic sensors.

9
Methodology - Software

MIDI Protocol
Before the creation of MIDI protocol, many other proprietary standards
were used to interface between different digital instruments. They allowed
the communication and the exchange of messages between two instruments
of the same brand, but failed to do that when is dealing with devices that
used different protocols.

The MIDI protocol was created in 1983 to standardize the interface

between digital instruments. It created requirements for the hardware and
communication. The relevant aspects of the protocol to the project are the
message standard and communication standard. MIDI messages dont carry
information about the sound being played, but they tell which, when and for
how long notes are being played, analog to music notation. Some of the
advantages of MIDI protocol over the previous standards are the fact that it
is open-source, simple, fast and lightweight (small messages). [1]

MIDI Messages
MIDI messages are delivered through serial port, and are 8 bit (1 byte)
long. There are two types of message: status messages and data messages.

The MSB (most significant bit) of the status messages is 1. The next 3
bits are the command and the last 4 is the channel the message is being
sent to. Depending on the status message, 0 to 3 data bytes are expected to
follow it.

The MSB of the data messages is 0. The next 7 bits are the data value
for the message. You can send 128 different values using one data message.

10
 Figure 9

The two status messages used by the system are the note on and the
control change message. They are sent on the channel 1 (for example) and
you can see their structure in Table 3.

Status Data Description

D7----D0 Byte(s)
D7----D0

1001nnnn 0kkkkkkk Note On event. This message is sent when a note is

0vvvvvvv depressed (start). (kkkkkkk) is the key (note)
number. (vvvvvvv) is the velocity.

1011nnnn 0ccccccc Control Change. This message is sent when a

0vvvvvvv controller value changes. Controllers include devices
such as pedals and levers. Controller numbers 120-
127 are reserved as "Channel Mode Messages"
(below). (ccccccc) is the controller number (0-119).
(vvvvvvv) is the controller value (0-127).
Table 3

11
MIDI Digital Audio Workstation (DAW)
Since the MIDI messages dont carry any sound information, in order to
test and play music, a MIDI instrument is needed. For this project, the
Hairless MIDI to Serial Bridge [10] computer driver was used to interpret the
MIDI messages received through the serial port and send it to Reason [11], a
DAW, for the messages to be played in one of its MIDI software instruments.

Code
The Arduino Micro is programmed using the C language.

The photoresistor ports are the 12 analog ports of the microcontroller. The
initial values for the sensor measurements are zero.

// Photoresistor ports
int light[] = {A6, A7, A8, A9, A10, A11, A5, A4, A3, A2, A1, A0};
int lightValue[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};

The MIDI protocol uses natural numbers from 0 to 127 to represent each note,
so that one unit stands for one semitone. The central piano C note number is
60. The array noteValue stores the current state of each note, pressed (1) or
not pressed (0).

// MIDI notes from C to B

int note[] = {60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71};
int noteValue[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};

The array noteThresholdOn will store the minimum sensor measurement

necessary to trigger the note on. Likewise, the array noteThresholdOff will
store the maximum value which will trigger the note off. This acts like a
digital Schmidt trigger, necessary to reduce note triggering caused by noise
in the sensor readings. This arrays are populated with the calibrate function.

// Ambient light adjustment values

int noteThresholdOn[12], noteThresholdOff[12];

Sensor ports, initial values and auxiliary variables.

// Ultrasonic sensor
int ultra = 7, ultraValue = 0x45, duration, velocity;

// Accelerometer
int accelX = 5, accelY = 3, valueX = 0, valueY = 0, pulseX, pulseY;

12
 We read the accelerometer and ultrasonic sensors once every 100 iterations
to decrease lag when transmitting the signals though the serial port.

// Sensing interval counter

int i = 0;

// Photoresistor calibration
int calibrated = 0;

void setup() {
Serial.begin(31250); // MIDI protocol baud rate
}

void loop() {
if (calibrated == 0) calibrate();

// Read sensors every 100 iterations

if (i == 100) {
readAccel();
readUltrasonic(); // Read ultrasonic sensor and update ultraValue
sendControl(0x03, valueY); // Send effect to MIDI channel 3
sendControl(0x01, valueX); // Send effect to MIDI channel 1

i = 0;
}
i++;

We read the values on the 12 photoresistors and decide if the corresponding

keys were triggered.

for (int i = 0; i < 12; ++i) {

lightValue[i] = analogRead(light[i]);

if (lightValue[i] >= noteThresholdOn[i]) { // The key was "pressed"?

if (noteValue[i] == 0) {
noteValue[i] = 1; // Change state
sendNote(0x90, note[i], ultraValue); // Turn on note
}
}
else if (lightValue[i] < noteThresholdOff[i]) { // The key was "released"?
if (noteValue[i] == 1) {
noteValue[i] = 0; // Change state
sendNote(0x90, note[i], 0x00); // Turn off note
}
}
}
}

The functions sendNote and sendControl implement the MIDI messages and
the serial communication to turn on or off a note and send a control message
(which can be used for several purposes, in particular we will use it to
produce sound effects).

// Send a MIDI note

void sendNote(int cmd, int pitch, int velocity) {
Serial.write(cmd);
Serial.write(pitch);
Serial.write(velocity); // Velocity = intensity or volume

13
}

// Send a MIDI control

void sendControl(int control, int value) {
Serial.write(0xB0);
Serial.write(control);
Serial.write(value);
}

The calibration routine populates noteThresholdOn with sensor

measurements when all holes are open and noteThresholdOff when all holes
are closed. This is critical since ambient light drastically changes the
photoresistor behavior.

// Photoresistors setup
void calibrate() {
while (!Serial.available());
int incoming = Serial.read();
Serial.println("Calibration starting. Leave all holes open and press any key.");

while (!Serial.available());
incoming = Serial.read();
Serial.println("Values for threshold off will be: ");
for (int i = 0; i < 12; ++i) {
noteThresholdOff[i] = analogRead(light[i]) + 30;
Serial.print(noteThresholdOff[i]);
Serial.print(" ");
}
Serial.println(" ");
Serial.println("Close all the holes and press any key.");

while (!Serial.available());
incoming = Serial.read();
Serial.println("Values for threshold on will be: ");
for (int i = 0; i < 12; ++i) {
noteThresholdOn[i] = analogRead(light[i]) - 20;
Serial.print(noteThresholdOn[i]);
Serial.print(" ");
}
Serial.println(" ");

calibrated = 1;
}

The ultrasonic distance sensor works by sending one input pulse and
measuring the time for the echo pulse to arrive. The function
microsecondsToVelocity is used to convert this waiting time to a number from
0 to 127, which will be the MIDI velocity (volume).

// The ultrasonic sensor is used for measuring the velocity (volume) of the notes
played.
void readUltrasonic() {
// The PING sensor is triggered by a HIGH pulse of 2 or more microseconds.
// We give a short LOW pulse beforehand to ensure a clean HIGH pulse.
pinMode(ultra, OUTPUT);
digitalWrite(ultra, LOW);
delayMicroseconds(2);
digitalWrite(ultra, HIGH);
delayMicroseconds(5);

14
 digitalWrite(ultra, LOW);

// The same pin is used to read the signal from the PING: a HIGH
// pulse whose duration is the time (in microseconds) from the sending
// of the ping to the reception of its echo off of an object.
pinMode(ultra, INPUT);
duration = pulseIn(ultra, HIGH);

if (duration != 0) ultraValue = microsecondsToVelocity(duration);

}

int microsecondsToVelocity(int microseconds) {

// The speed of sound is 340 m/s or 29 microseconds per centimeter.
// The ping travels out and back, so to find the distance of the
// object we take half of the distance travelled.
int cm = microseconds / 29 / 2;

// Velocity (intensity or volume) in MIDI goes from 0 to 127, and we measured the
// minimum distance being 30cm and the maximum being 145cm
if (cm >= 60) return 127;
else if (cm <= 3) return 0;
else return (127 * cm - 3 * 127) / 57;
}

The accelerometer is read in this function. Likewise, there is a conversion

from its measurements to a 0-127 scale understandable by the MIDI protocol.
In particular, the X and Y accelerations are used to produce 2 MIDI sound
effects.

// The accelerometer is used for 2 MIDI effect controls

void readAccel() {
pinMode(accelX, INPUT);
pinMode(accelY, INPUT);

// read pulse from x- and y-axes:

pulseY = pulseIn(accelY, HIGH);

pulseX = pulseIn(accelX, HIGH);

// convert the pulse width into acceleration

// accelerationX and accelerationY are in milli-g's:
// earth's gravity is 1000 milli-g's, or 1g.
valueX = (((pulseX / 10) - 500) * 8);
valueY = (((pulseY / 10) - 500) * 8);

valueX = abs(valueX);
valueY = abs(valueY);

if (valueX >= 1072) valueX = 127;

else valueX = (valueX / 1072.0) * 127.0;

if (valueY >= 1072) valueY = 127;

else valueY = (valueY / 1072.0) * 127.0;
}

15
Results

After some debugging and adjustment, the prototype reliability was

acceptable. The calibration routine was essential to make it work, since
ambient light turned to be a great influence. Also, the introduction of two
lighting thresholds (similar to a Schmitt trigger) was important to reduce
sensor jittering.

The advantages of using breadboards were the ease to assemble the

prototype and the possibility of quickly change the circuitry. However, faulty
connections were a common problem that could be solved by using printed
circuit boards.

16
Conclusion

The device built shows that motion sensing does not necessarily need
expensive sensors and complicated architectures. It is a perfect
demonstration of the creative potential of the technology on music, and it
brings a new way to create and perform music on live or on studio. An
improved version could be sold as a DJ equipment, toy or video game. The
involved gesture sensing technologies could also be used for other
applications, for example: socially assistive robots, remote controllers and
alternative computer interfaces.

Possible improvements on the prototype for the second version include

replacing the mechanical parts of it by more comfortable and ergonomic
ones, specifically designed for it. There is also room for more features, such
as control knobs and faders on the control module, for a more practical and
complete control of other parameters of the music. Giving the option of the
user to change MIDI parameters of the system such as controller numbers or
channel numbers is another functionality that could be added on a future
version of it. This could be made coupling a LCD display and buttons to the
system or through serial communication to a computer or smartphone. Other
sensors could be added to provide even more ways to change the music
through gesture, such as a 3 axis accelerometer and gyroscopes.

17
Bibliography

[1] MIDI Manufacturers Association. An Introduction to MIDI.

http://www.midi.org/aboutmidi/intromidi.pdf

[2] Whats a Theremin?. http://www.thereminworld.com/Article/14232/what-s-a-theremin-

[3] Ondes Martenot. http://www.soniccouture.com/en/product/g27-ondes/

[4] Microsoft Corporation. Kinect Sensor. http://www.xbox.com/en-US/kinect

[5] Leap Motion Inc. Leap Motion 3D Controller. https://www.leapmotion.com/

[6] Oculus VR Inc. Oculus Rift Virtual Reality Headset. http://www.oculusvr.com/

[7] Arduino. Arduino Micro. http://arduino.cc/en/Main/arduinoBoardMicro

[8] MIDI Manufacturers Association. MIDI Products.

http://www.midi.org/aboutmidi/products.php

[9] Understanding Schmitt Triggers. http://www.ti.com/lit/an/scea046/scea046.pdf

[10] Hairless MIDI to Serial Bridge. http://projectgus.github.io/hairless-midiserial/

[11] Reason Digital Audio Workstation. http://www.propellerheads.se/products/reason/

18