Back in December 2012, I started to play around with Computer Vision and webcams, using OpenCV in Processing.
There were a few reasons for this, one was to look at a low-cost way of developing sensory type applications using equipment that most people have access to, unlike the MS Kinect. Another reason was an interest in trying to get some “Face Tracking/Eye Tracking” code running in Processing to explore alternatives for students with limited movement and where simple switch functions don’t provide a suitable input for interaction. My colleague James Winchester has already looked at some open-source solutions in a post on his blog, you can read more about this on SEN Classroom at the link provided in the section at the bottom of the post. I also planned to look at using webcams to produce art style portraits, in an attempt to make an “Introduction to using computer code for Creative Arts”, accessible for SEN students, if you missed reading about it in an earlier post, you can read about it in SEN Students and Coding.
I’m still using Processing 1.5.1 rather than Processing 2.0, because 1.5.1 is the current stable release. The OpenCV installation instructions are provided on the download page, a link is provided below. The development of OpenCV for Processing appears to have stalled at version 1.0 but there appears to be an alternative called JavacvPro which I initially had trouble getting to work, so I stayed with OpenCV 1.0 for my experiments but if anyone has used JavacvPro, I would appreciate it, if you could contact me by email.
There are some examples available for OpenCV and a few tutorials I thought would make a good place to start but I initially hit upon a snag. My Webcam is quite new, a MS HD Cinema 720P camera, which did not seem to be supported by OpenCV, so any sketches I tried to run just returned a grayed out display screen with errors. I discovered by searching online that a number of people were experiencing issues either installing or running the examples, so by searching for solutions and a bit of trial and error, I attempted to capture the video in Processing and pass the output of this to OpenCV, where the frame can be stored in a buffer and this solution seems to work. It did require a rewrite of most of the examples but this became quite quick to do and once this solution was in place, I was able to play around with some experiments in OpenCV.
One of the first examples I tried, was to look at code to produce a painting effect on an image, looking at the difference between 2 frames of video captured by the webcam and comparing the 2nd frame with the first frame stored in the OpenCV buffer, the resulting output could be saved on a mouse click or keypress if desired. It produces a similar effect to the application in the Somantics suite “Painter” but isn’t anywhere near as good, however for a first attempt with OpenCV, I’m quite pleased with it and a number of my students have played about with the sketch and enjoy using it.
From these initial experiments in using Processing with OpenCV and PImage, I have coded several sketches to enable me to work with SEN students in creating art style portraits.
I think there is a lot you can do with image files and manipulation in code, the SEN students I have been working with seem to have really enjoyed using this process and I feel there is still a lot we can achieve, working this way. It is also really satisfying for me to see both students become increasingly independent in running code sketches from the IDE and using ICT skills to save, name and manipulate images by cropping or applying effects in other applications. In using the latest coded sketch, they have also seen how it is possible to create images like the one above totally in code, without using effects from any other application.
At the start of the post I mentioned one of the reasons I wanted to explore using Processing, OpenCV and webcams was to look at possible solutions for Face/Eye Tracking and this is something I have only just begun exploring. I now have a code solution for “Face Tracking” that works in Processing with OpenCV, I have tested on two setups, it will track faces quite successfully but does occasionally lose tracking, only to pick it up again. I hope that by implementing more accurate “Eye Tracking” there will be a number of possible uses for this and that it might provide some base code to start creating applications we could begin using with our PMLD students and if I get any further with this project I will cover it in more detail in another post.
JavacvPro an updated library for OpenCV in Processing (Please let me know if you get it installed and working)