Music Visualization

CHAPTER 1

INTRODUCTION

Music is a unique form of expression due to all its special characteristics. These
characteristics are what make music so special, because they are constantly changing,
growing with different levels of intensity as it develops over time. This artistic form of
expression can also inspire emotions, in other words music composers/ musicians can express
their feelings through music, which then becomes a source of inspiration for other musicians
or even people who carefully listens to music. Additionally, those special characteristics
mentioned before, are the several elements existent in the music structure itself, which is
understood only by proficiency people in music (Klemenc, Ciuha, Subelj, 2011). The music
structure elements is what makes it so intensive and emotional, this means that music artists
use those elements in order to express their feelings through their music. Music is so
interesting because is constantly changing and growing over time and has different levels of
intensity as it develops due to the previous stated elements. When people listen to music, they
sometimes create a mental model of the specific music piece that they are listening to. That
mental model represents the message and/or feelings of the person at that moment of listening
the piece, which sometimes are highly related to their personal life experience. These mental
models can be different for each person, because an individual has their way of seeing and
feeling things. This is why a music piece sometimes has a vast number of unique
interpretations for the same music piece. Everyone has their own life experiences and
influences, as well as culture, therefore, everybody have different opinion when listening to a
music piece.

1.1 Introduction to Graphics

Computer graphics is one of the most powerful and interesting facet of computers. There are
a lot of things that we can do apart from drawing figures of various works using computer
graphics. All video games, animations, multimedia predominantly works using computer
graphics.

It is divided into two broad classes.

1. Non Interactive Computer Graphics.

2. Interactive Computer Graphics.

DEPT OF CSE, KLECET,chikodi 2018-19. Page 1

Music Visualization

In non-interactive computer graphics or passive computer graphics the observer has no

control over the picture produced in the screen. While in interactive computer graphics the
picture displayed on the screen are controlled by the user through an input device.

Pixel (Picture Elements)

The number of dots or pixels available to us on the screen in the graphics mode we have
selected is known as the resolution. The greater number of dots, the higher the resolution. To
switch over to the graphics mode that offers the best resolution we need to call the function
initgraph().

1.2 About OpenGL

OpenGL is a software interface to graphics hardware. This interface consists of about 150
distinct commands that you use to specify the objects and operations needed to produce
interactive three-dimensional applications.

OpenGL is designed as a streamlined, hardware-independent interface to be implemented

on man different hardware platforms. To achieve these qualities, no commands for
performing windowing tasks or obtaining user input are included in OpenGL; instead you
must work through whatever windowing system controls the particular hardware you’re
using. Similarly, OpenGL doesn’t provide high-level commands for describing models of 3-
dimensional objects. Such commands might allow you to specify relatively complicated
shapes.

With OpenGL, you must build up your desired model from a small set of
geometricprimitives – points, lines and polygons. A sophisticated library that provides these
features could certainly be built on top of OpenGL. The OpenGL Utility Library (GLU)
provides many of the modeling features, such as quadric surfaces and NURBS curves and
surfaces. GLU is a standard part of every OpenGL implementation. Also, there is a higher-
level, object-oriented toolkit, Open Inventor, which is built atop OpenGL, and is available
separately for many implementation of OpenGL.

OpenGL contains rendering commands but is designed to be independent of any window

system or operating system. Consequently, it contains no such commands for opening
windows or reading events from the keyboard or mouse. Unfortunately, it’s impossible to
write a complete graphics program without at least opening a window, and most interesting

DEPT OF CSE, KLECET,chikodi 2018-19. Page 2

Music Visualization

programs require a bit of user input or other services from the operating system or window
system. In many cases, complete programs make the most interesting examples, so this
project uses GLUT to simplify opening windows, detecting input, and so on.

The OpenGL interface

Our application will be designed to access OpenGL directly through functions in three
libraries namely: gl,glu,glut.

The following fig.1.2 shows the architecture of OpenGL. It consist of OpenGL application
program which includes the libraries like GLU, GL, GLUT, GLX.

Fig1.2 Architecture of OpenGL

1.3 Uses of computer graphics

Computer graphics is used in many different fields such as business, industry,

education and other usages are as follows

Computer aided drafting and design

A major use of computer graphics is a design process particularly for engineering and
architectural systems, generally referred to as CAD or CADD.

DEPT OF CSE, KLECET,chikodi 2018-19. Page 3

Music Visualization

Graphical user interface

A major component of a graphical interface is a window manager that allows a a users

to display multiple, rectangular screen areas called display windows

Virtual reality environment

It is the environment in which a user can interact with objects in three dimensional
scenes. Animations in virtual reality environment are often used to train the equipment
operators.

1.4 Objective

A live performance exalts different characteristics of music, because they become so evident
as the audience can experience different levels of emotional intensity existent in a live
performance. In consequence the audience navigates through a parallel world produced by
inherent emotions resulting of the different music pieces.

Moreover, with a music live performance, there is the possibility of providing a series of
stimulating reasons, which provokes significant moments intrinsic to the listener’s live
experiences. Moment with music visualization because it helps them to create a mental model
of the music, thus they understand it better and also make it more interesting.

This application enables the interaction between a live musician and a responsive virtual
character, and it responds according to the music it hears. This character visualizes the
perception and cognition of musical input by exhibiting responsive behavior expressed
through waves.

One of the first thing to do in music visualization is to visualize the items of musical sense
along the music perception.

This is the main goal for this mini project, enhance the audience live performance experience
using music visualization. They should become emotionally more involved with music.
Music visualization is that complement for those people who struggle to understand a certain
music piece or have difficulty to connect to a certain kind of music, making them feeling or
imagining something when listening to it.

DEPT OF CSE, KLECET,chikodi 2018-19. Page 4

Music Visualization

CHAPTER 2
REQUIREMENTS SPECIFICATION

The development of the project was done with these requirements. May support lower
versions also but not been tested.

For the given input audio file (.wav) the program will give the real time audio spectrum by
performing Fast Fourier Transform (FFT) on the audio samples and plot the graph
using OpenGL. This Project is implemented in C++ using OpenGL. Three
frameworks Aquila-dsp, Kiss-Fft, and SFML are used to perform audio sampling, fast
fourier transform and audio playback respectively. The development and testing of this
project is done on Linux (ubuntu) using 4.2.0-16-generic Kernel. Interface for the program
is given with the help of keyboard.

C++ compiler : g++ with c++11 support (-std=gnu++11) (version >= 5.2.1)

 OpenGL (version >=v4.5)

 Aquila-dsp framework (version >=v3.0)
 Kiss-fft Framework (version >=v1.3.0)
 SFML framework >=2.3.2
 Git>=v2.5.0

2.1 Minimum Hardware Requirement

 Processor : Above x86.

 Processor Speed : 500MHz and above.
 RAM: 128MB.
 Storage Space: 4GB.

2.2 Minimum Software Requirement

 OpenGL libraries – gl, glut, OpenGL and glaux.

 OpenGL (version >=v4.5).

DEPT OF CSE, KLECET,chikodi 2018-19. Page 5

Music Visualization

CHAPTER 3

DESIGN

3.1 Algorithm
Step 1: Begin

IntframePointer = 0, N = 32768.

Step 2: Load input audio file and play it using the audio Library

Step 3 : For i = framePointer to → framePointer + N <total_samples_count

Collect N samples from the audio file

Step 4: Apply suitable window function (e.g. Hann aka Hanning window)

Step 5: Apply Fast Fourier transform (FFT) on the array elements and collect N/2 Complex
numbers having real part and imaginary part. (NB: if using typical complex-to-complex FFT
then set imaginary parts of input array to zero)

Step 6: Calculate the magnitude of N/2FFT data.

magnitude = sqrt(re * re +img *img)

Step 7: Convert magnitude to dB (log)scale. (optional)

20 * log10(magnitude)

Step 8: Plot N/2 log(magnitude)values.

Step 9: If N >= total_samples_count

Exit.

Else goto Step 3.

Step 10: End.

3.2 Flow Chart

The Figure 3.2 illustrates the following
 Load input audio file and play it using the audio library.
 Collect N samples from the audio file.

DEPT OF CSE, KLECET,chikodi 2018-19. Page 6

Music Visualization

 Apply suitable window function.

 Apply Fast Fourier transform (FFT) on the array elements and collect N/2 Complex
numbers having real part and imaginary part. (NB: if using typical complex-to-
complex FFT then set imaginary parts of input array to zero).
 Calculate the magnitude of N/2FFT data.
 Use ‘Left/Right’ to move horizontally.
 Use ‘Up/Down’ to change the horizontal scale.
 Use ‘Home’ button to reset the position and scale.

Figure3.2 Flow chart

DEPT OF CSE, KLECET,chikodi 2018-19. Page 7

Music Visualization

CHAPTER 4

IMPLEMENTATION

For the given input audio file (.wav) the program will give the real time audio spectrum by
performing Fast Fourier Transform (FFT) on the audio samples and plot the graph
using OpenGL. This Project is implemented in C++ using OpenGL. Three
frameworks Aquila-dsp, Kiss-Fft, and SFML are used to perform audio sampling, fast
fourier transform and audio playback respectively. The development and testing of this
project is done on Linux (ubuntu) using 4.2.0-16-generic Kernel. Interface for the program
is given with the help of keyboard.

4.1 Frameworks / API’s used

The following c++ frameworks are used in this project.

1. Simple and Fast Multimedia Library (SFML)

2. Aquila-dsp
3. Kiss-fft
4. OpenGL

4.2 User Defined Functions

Name :get_timestamp ()

Cspecification: static timestamp_tget_timestamp ()

Description: Used to get the current system time.

Name :getFft ()

Cspecification: void getFft ()

Description: Used to get the fourier transform of the input array elements.

Name :getData()

DEPT OF CSE, KLECET,chikodi 2018-19. Page 8

Music Visualization

Cspecification: void getData()

Description: Used to fetch N samplings, convert them to frequency domain, and convert
them to log10scale.

Name :init_resources()

Cspecification: intinit_resources()

Description: Initializes the resources needed for the OpenGL to draw the graph.

4.3 ADVANTAGES AND DISADVANTAGES

Advantages

 Dissolving Tension.

 Learner motivation to music.

 Lowering of affective filters.

 Cooperative work.

 Convey what they are communicating in an effective way.

 Recording and Broadcasting.

Disadvantages

 Frustrating to create need skills accessibility.

 Lack of theoretical support.

 Logistic difficulties : Logistic problems may cause the delay.

 Music can make us Hypersensitive.

Applications

 To create interactive animation and video effects for concerts.

 It can be used clubs, theater, movies, parties.art installations, advertising, education,
research.

DEPT OF CSE, KLECET,chikodi 2018-19. Page 9

Music Visualization

CHAPTER 5

SNAPSHOTS

Figure5.1 After pressing F7 key.

Figure5.2 After pressing F8 to toggle interpolation.

DEPT OF CSE, KLECET,chikodi 2018-19. Page 10

Music Visualization

Figure5.3 Pressing F9 to toggle drawing points.

Figure5.4 Pressing p to play a audio.

DEPT OF CSE, KLECET,chikodi 2018-19. Page 11

Music Visualization

Figure5.5 pressing r to reload and play audio.

Figure5.6 final output.

DEPT OF CSE, KLECET,chikodi 2018-19. Page 12

Music Visualization

CONCLUSION

Audio files of type (.wav) is supported. Power Spectrum or Frequency Domain spectrum can
be generated for the audio. Basic Music player functions such as Load, Play, Pause, Seek and
Replay is supported. Clamping, interpolation, and showing of points can be toggled. The
spectrum can be scaled in Y axis.With the development of this mini project was possible to
achieve all intended goals, and also contribute for this area of music visualization. The
principal goal of this mini project to find the best mapping between music and visuals
therefore, the connection between the audience and music in a live performance is enhanced.

DEPT OF CSE, KLECET,chikodi 2018-19. Page 13

Music Visualization

BIBLIOGRAPHY

 Donald Hearn and Pauline Baker: Computer Graphics with OpenGL Version,3rd / 4th
Edition, Pearson Education, 2011.
 Edward Angel: Interactive Computer Graphics- A Top Down approach with OpenGL,
5th edition, Pearson Education, 2008.
 M M Raiker, Computer Graphics using OpenGL, Filip learning/Elsevier.
 James D Foley, Andries Van Dam, Steven K Feiner, John F Huges Computer graphics
with OpenGL: Pearson education .

DEPT OF CSE, KLECET,chikodi 2018-19. Page 14