Drawing with Sound

My colleagues are a supportive bunch, so when I asked one of them, whether she would like me  to code something for her students, I was encouraged by her enthusiastic yes, so following appreciation of a number of our previous projects, I felt I should attempt to create something for her students to engage with.

Her idea initially was for me to create a simple application to engage the students in her class by the use of sound, shape and colour as well as provide for some uncomplicated interactivity on a “Smart” Interactive white board.  I also wanted to provide a method of achieving an art print, which could be attractive both for display and used by teaching colleagues as evidence.

Example of the type of Art Print created with the Drawing with Sounds sketch, created to engage students with ASC using shape colour and sound, which uses Andrew R Browns “SoundCipher” Library built in Processing 1.5.1

Drawing a shape in Processing isn’t difficult to achieve as you will have discovered if you have read my previous post on Processing .. “Processing – Getting Started” so the challenge became keeping it simple enough to be useable by students, capable of producing an attractive and colourful image and create sounds which could engage students less interested in the colour or shape of objects drawn to the display screen.

Processing has a number of built in libraries which assist in creating applications and I considered using one of them called Minim for this (more on Minim, which can be found here, in another post), however I discovered an excellent library called “SoundCipher” by Andrew R Brown which has the advantage of being a free open-source project that makes it easy to create music within the Processing development environment, by using the JavaSound synthesizer for playback of audio files and synthesizing sounds via MIDI, enabling me to concentrate more on enhancing the graphics and drawing elements of the sketch to provide some simple interaction.

I don’t (and can’t) claim any credit for the musical side of this sketch, which comes from a tutorial on the “SoundCipher” website called Unlimited Art and which produces some fantastic musical phrases.  I did adapt some of the musical elements of the sketch when initially experimenting with the library, however I was never satisfied with my efforts, I’m not a musician and I quickly reverted back to the version in the tutorial created by someone who is.  The tutorial is excellent but doesn’t have the graphical element and interactive functionality I was seeking, so for my part I rewrote the graphics element, ensured it was interactive for use on a whiteboard and added a mechanism for exporting .jpg files.

The tutorial provides excellent explanatory notes about how “SoundCipher” is working, so I won’t reproduce all of that here.

This application has proved to be quite popular, with students in my own class enjoying using it on an Interactive “Smart” board or at the computer.  They seem to like 4 aspects about it.

• It’s simple for them to use, although some students I thought would prefer the interactive Smart board, have actually preferred using a mouse
• They like the fact their interaction causes sounds, although any sounds might prove interesting to them and not just music
• They can make a print to keep or view the image again on the computer later in a slide show, something several of them get a sense of achievement from, when they get an opportunity to share their work and enjoy looking at each others images
• They like colour and shape as a motivator for cause and effect interaction but they really like music

This application is now available for use in several areas of our school and to increase accessibility James Winchester at SEN Classroom, adapted it for use with a switch. Recently we ran workshops with around 15 students using the application on PC with mouse, touchscreen and on an interactive Smart board. This proved to be a great success with positive comments from colleagues and students.

I asked one of the students in my class what they thought about using the application and his reply was “it sings to you, when I touch the screen, I like the black dots, I like to sing back to it”.  My thanks to Andrew R. Brown, who kindly contacted me to say he hoped our students enjoyed using the application.  We have now shared both versions with colleagues at two other schools in the UK, so I’m hopeful we can provide an update on how the students at those schools are using it.

If you are interested in trying this application with your students, it is available for download on the link provided on the “free stuff” page.