Computer student on gesture control: Start experimenting

Back in 2012, authors from Microsoft Research and UbiComp Lab at University of Washington prepared their paper, "SoundWave: Using the Doppler Effect to Sense Gestures," for the Proceedings of the Association for Computing Machinery's Conference on Human Factors in Computing Systems. Their work created a lot of interest in the way that they managed to implement motion sensing using only speaker and mic. Sidhant Gupta, Dan Morris, Shwetak Patel, Desney Tan said that "Gestures are becoming an increasingly popular means of interacting with computers. but—it is still relatively costly to deploy robust gesture-recognition sensors in existing mobile platforms." Enter their SoundWave, a technique that leveraged speaker and microphone for sensing in-air gestures and motion around a device.

They wrote that SoundWave could directly control existing applications without requiring a user to wear any special sensors on their body. They used the existing speakers on commodity devices to generate tones between 18-22 kHz. Then they used existing microphones on these same devices to pick up the reflected signal and estimate motion and gesture through the observed frequency shifts. In explaining their leaning toward tones above18 kHz, they said the higher the frequency, the greater the shift for a given velocity, which makes it computationally easier to estimate motion at a given resolution. The gesture range included scrolling and "two-handed seesaw," where you move both hands simultaneously in opposite directions at the same time.

Their technique has again made news thanks to Daniel Rapp, a Swedish computer science student, who read their research paper and worked it out offering Doppler demos right on the web page. John Wenz in Popular Mechanics described what Rapp worked out in computer control via hand gestures based on the paper. "First, the speakers emit sounds at particular frequencies. When a hand passes through the waves, it changes them in subtle ways, and the microphone senses those changes when it picks up the sound. The app is programmed to use them to scroll the screen. Rapp has also designed the sound to modulate like a theremin." (Rapp had written "reminiscent of a Theremin, in that you can control sound by moving your hands in free space, yet different in that the Theremin is able to measure absolute distance from the base, whereas we can only measure relative motion.")

The Doppler Effect, said Rapp, "is a physical phenomenon which affects waves in motion. The standard example is the effect on a fire engine siren as it quickly drives past. When it moves towards you the sound waves are compressed, and so the frequency becomes higher, and when it moves away from you the frequency becomes lower." Rapp also remarked that "It's important to realize that the doppler effect would also occur if you were to run towards the siren, rather than the siren moving towards you."

"Recently I stumbled upon an interesting paper for implementing motion sensing requiring no special hardware, only a speaker and mic! Unfortunately the paper didn't include code to test it, so I decided to reproduce it here on the web!" said Rapp.

You can check out more about this implementation on GitHub.

Alex Young in DailyJS said, "This is one of the best hacks I've seen for a while, and although the tone makes the applications limited the fact it works at all is fascinating!"

© 2015 Tech Xplore