Mariam Aburdeineh

Students program computers to interpret sign language

Melanie Balog — Mon, 12 Aug 2019 20:11:08 +0000

Students program computers to interpret sign language Melanie Balog Mon, 08/12/2019 - 16:11

With a twist or shake of your wrist, your smartphone can interpret motion to take a picture, turn on a light, and more. Last year, George Mason University computer science professors Parth Pathak and Huzefa Rangwala were brainstorming how similar technology could help society in even greater ways. Their idea? To automatically translate sign language into text or speech.

“There are some products that can do gesture recognition, but they’re very preliminary. And it’s very different from ASL [American Sign Language], which is not just a few gestures—it’s thousands of words,” said Pathak, principal investigator on the Summer Team Impact Project funded by Mason’s Office of Student Scholarship, Creative Activities, and Research (OSCAR).

This summer, nine Mason undergraduates joined in the research that could help make the technology a reality.

Frederick Olson (from left), Sai Gurrapu and Dom Huh are part of a summer research project on automatic multimodal sign language recognition. Photo by Lathan Goumas/Office of Communications and Marketing.

“The goal would be to deliver a readable message to a device so that it’s bridging the gap between ASL users and non-users,” said therapeutic recreation senior Riley Wilkerson, “an easier, more effective, and more personal way of communicating.”

Riley Wilkerson signs for a radar sensor as part of a summer research project on automatic multimodal sign language recognition. Photo by Lathan Goumas/Office of Communications and Marketing.

Three teams of students are experimenting with different sensors: a wireless radar, a camera, and an inertial measurement unit (a wearable motion sensor used in smartphones and Fitbits). Each sensor offers certain opportunities, but also challenges including privacy and ease of use, said Pathak, who is guiding the students on the project along with Mason computer science professor Jana Kosecka and Mason’s Helen A. Kellar Institute for Human disabilities director Linda Mason, and graduate student Panneer Selvam Santhalingham.

On each team, a student familiar with ASL signs in front of a sensor that collects data about the motion or the environment. Computer science and engineering students refine the data to find patterns and write machine learning algorithms—code that allows them to interpret the computer’s recognition of the signs.

So far, the undergraduates have “taught” their machines to recognize about 20 signs with accuracy rates ranging from 70-97 percent. The fluctuations in accuracy are due to the machine learning process, said senior computer science major Yuanqi Du.

Diverse data helps the computer recognize the signs with increased accuracy, Du said. In initial trials with one student, accuracy rates were higher. When a new ASL user was introduced, the accuracy diminished, Du said. Once the new ASL user’s data was included in the algorithms, accuracy rates rose again.

Ariana Havens wears cell phones as she signs as part of a summer research project on automatic multimodal sign language recognition. Photo by Lathan Goumas.

As the multi-year project continues, Pathak said the team plans to increase the number of signs the computer can recognize using data from many diverse users. They will also scale it to interpret full sentences and pick up other gestures used in ASL such as body tilts and micro expressions like raising an eyebrow, he said.

“Being able to communicate instantly would hopefully remove issues [the ASL community experiences],” said Frederick Olson, a senior IT major who said both his parents are deaf. That includes being able to ask a question at a store, socializing, communicating with doctors easily during appointments, or being able to land better job opportunities. The technology could be life-changing, he said.

It could also be applied beyond the deaf community, the students said, helping people with autism or developmental and learning disabilities for whom communicating using spoken words is challenging, Wilkerson said.

“It could be applicable to other industries and disciplines in the future [that will work with similar technology], too,” said junior computer science major Sai Gurrapu.

And, the project pushes student learning to the next level, Pathak said.

“They’re not given a fixed task here—they’re given a problem and they have to find a solution,” Pathak said.

“This project is one of a variety of opportunities [Mason] has presented to me that goes beyond just taking 15 credits each semester,” Wilkerson said. “You can only learn so much in a classroom—you have to apply it.”

Seniors Yuanqi (from left), Nguyen Dang and Riley Wilkerson are part of a summer research project on automatic multimodal sign language recognition. Photo by Lathan Goumas/Office of Communications and Marketing.

computer science

Office of Student Scholarship Creative Activities and Research (OSCAR)

Helen A. Kellar Institute for Human disAbilities

therapeutic recreation

engineering

Research

Mason was his launch pad to Silicon Valley

Melanie Balog — Fri, 31 May 2019 09:00:32 +0000

Mason was his launch pad to Silicon Valley Melanie Balog Fri, 05/31/2019 - 05:00

Joey Meyer graduated from Mason in 2012 and works for Apple. Photo provided

Mason alumnus Joey Meyer developed an appreciation for using his imagination to build ever since he was a child fascinated by Legos. After graduating from McLean High School in Virginia, he still wanted to create, and earning an engineering degree was his goal.

He enrolled at George Mason University but still thought about attending Virginia Tech. After taking a few classes at Mason, though, “I realized that George Mason had a really strong computer science program that was on par or better than Virginia Tech,” Meyer said. “I figured it would be worth it to stay.”

He’s glad he did.

Meyer, who earned his bachelor’s in computer science from Mason’s Volgenau School of Engineering in 2012, is now a software engineer at the tech giant Apple, where he works to make user experiences more intelligent on Apple operating systems.

How did Mason set him up for success in Silicon Valley?

“How didn’t it?” Meyer said. “The foundation for everything that I do on a daily basis, I learned at Mason.” That includes writing code, data structures and algorithms.

His classes with award-winning professors not only provided the foundation for his learning, but were inspiring, as well. He particularly enjoyed Sean Luke’s class.

“He had a different approach to lecturing, and you could tell that he lives and breathes computer science,” said Meyer, who felt he also picked up on Luke’s contagious enthusiasm.

“Once I started taking computer science courses, I would stay up late at night working on [projects]—not because I had to, but because it was cool.”

That held true inside and outside of the classroom.

When Meyer attended a guest lecture on campus by author Jullien Gordon, he approached Gordon about building an iPhone app for the book that he cowrote: “101 Things To Do Before You Graduate.” That led to Meyer creating even more apps for local clients and companies in the Washington, D.C., area, like LivingSocial, and connecting with fellow students who were passionate about similar projects.

Meyer continued to make strong use of Mason’s location, which is home to one of the largest concentrations of technology firms in the nation, as an intern for a local company where he further developed his programming skills.

One opportunity led to the next when Meyer’s coworker told him about an 18th-century machine that could play chess with humans. Fascinated, Meyer set out to build the chess robot, and he successfully completed it after graduating.

Each experience, no matter how different, was impactful, said Meyer, who is grateful he gets to combine his top interests—developing products that users interface with and using machine learning and data to solve problems—in his current position.

“[Mason has] several graduates who have gone on to work at the leading companies in the world—not just Apple, but Google, Microsoft, IBM, Facebook, Amazon, not to mention Lockheed Martin, Northrop Grumman, Capital One, and others,” said Sanjeev Setia, chair of Mason’s Computer Science Department. “We are proud that these companies will hire our graduates—this demonstrates not only the academic rigor and strength of our programs, but also the quality of our student body.”

Nearly 6,500 students are enrolled in computer-intensive majors at Mason—more than any other public university in Virginia—and by 2024, that number is expected to climb to 15,000.

To support the increased demand for talent in these fields, Mason will launch its School of Computing in 2019, as well as the Institute for Digital InnovAtion (IDIA), a multidisciplinary institute that will advance research and the digital innovation economy.

Combined with Mason’s faculty, who bring industry experience and current research into the classroom, students will undoubtedly benefit, said Setia.

Meyer agrees.

“College is largely what you make of it,” Meyer said. “And Mason’s computer science program has exceptional professors and an environment that will allow you to take full advantage and achieve what you set out for.”

Volgenau School of Engineering

computer science

Students