Particle filtering is a standard and well behaved method for modeling the state of a system and keeping that model in sync with the actual world state through measurement. Particle filter systems are good at dealing with systems which change dynamically and unpredictably. In this lab we implemented a particle filter to implement simple face tracking of a camera image.
task description
Using images captured from a digital camera, we first created a histogram to recognize skin tone colored pixels by using a provided program which creates histograms based on red and green chromaticity (histogram in r and g space, where r = R / (R+G+B), etc). Then for any image captured from the camera, we can look up the pixel's chromaticities in the histogram to determine probability that the pixel represenents skin color. Everything up to this point was essentially provided for us.
Our task, then, was to take this data and apply a particle filter to attempt to track faces, the assumption being that faces will be large clumps of primarily skin-colored pixels. What the particle filter allows us to do is rather than just seek a clump of skin colored pixels, we make the further assumption that an actual face (or an actual skin-colored particle) will persist over time in a particular location in the image. In our representation, and individual particle corresponded to a 10x10 grid in the image. Those particles for whose 10x10 pixel grid have a high probability of being skin have a better chance of being selected for inclusion in the next generation.
For a more complete description of how we implemented the particle filter, see the relevant C code.
postprocessing and results
We used a variety of postprocessing methods to analyze and display the data. Our first method involved drawing the particles over the source image in 10x10 cyan squares, at an intensity proportional to the particle's weight (ie. probability of being a 'good' face particle), while making no changes to non-particle pixels. Our second method involved drawing pixels which correspond to particles normally and pixels which did not correspond to particles in greyscale. Our third method was an extension of the second, wherein we assumed that a 'face' takes up a fixed region of the image (about 80x50) and scanned the image for the rectangle of that size with the most particles, then assumed that winning rectangle corresponded to a face and drawing a red rectangle around the face. Examples of all three methods are shown below:
In general, the result is that faces and face-like objects are successfully tracked by clumps of particles. The face rectangle will quick effectively latch onto a single face in the image, but may jump around if there is more than one face.
extensions
We added a velocity component to our model of particle state. Velocity values were jittered similarly to x and y location values, and were initialized in a fairly wide distribution. The result is that the system is much more responsive to tracking motion of a particle clump (this works well with hands).
Also, apparently the fact that we considered areas rather than individual pixels for our particles was an extension, and, while of dubious relevance to the actual task of implementing a particle filter, some of the image processing stuff we did was pretty cool.
what we learned
Particle filtering is a robust technique for modeling state in a dynamic and nondeterministic system, with many interesting applications in engineering and computer science, particularly in the realms of vision processing and robotics. This lab immeresed us in the guts of particle filtering, and helped us realize both how effective they actually are and that they aren't that hard to implement. It also gave us some appreciation for the fact that just throwing a particle filter at a problem does not solve it. For instance, our 'face recognition' based on the rectangle with the most particles makes too many assumptions (fixed size face, only one face, all skin-colored clumps are faces), and to truly recognize and track faces would take for more processing and insight into the problem domain.