Tushar Chugh is a graduate student at the Robotics Institute at Carnegie Mellon University (CMU). There he studies the latest in robotics, particularly how computer vision devices perceive the world around them.

One of his favorite projects was a robot named Andy. Besides having arms, Andy could discern colors and understand spatial arrangement. Andy could also respond to voice commands, like “pick up the red block and place it on top of the blue block.” Andy’s speech recognition, a CMU framework, was about to change.

When Amazon came to give some lectures at CMU, they had a raffle drawing. Chugh won the drawing and took home a new Amazon Echo as a prize. Over three days and nights without sleep, he completely integrated Andy and Alexa using the Alexa Skills Kit (ASK).

When he saw Hackster’s 2016 Internet of Voice challenge, he knew he had to enter. And in August 2016, Chugh’s Smart Cap won the prize for the Best Alexa Skills Kit with Raspberry Pi category.

The inspiration and genesis of Smart Cap

According to Chugh, there are about 285 million visually-impaired people in the world. In 2012, he worked on a project to help the visually impaired navigate inside a building. His device, a belt with embedded sensing tiles, won a couple of prizes, including a Wall Street Journal Technology Innovation Award. It was ahead of its time, though, and it wasn’t yet practical to develop the technology into a commercial product.

A lot can change in four years, including Chugh’s discovery of Alexa. Besides dabbling with Alexa and Andy the robot, he has also worked with Microsoft Cognitive Services for image recognition. Chugh now saw a chance to bring a new and better “seeing device” to light.

“When I saw Alexa, I thought we can extend it and integrate [Alexa] as a separate component,” says Chugh. “I talked with a couple of organizations for the blind in India, and they agreed this kind of system would be very, very useful. That was my main motivation.”

Chugh says the hardware for the Smart Cap is basic. He used a Raspberry Pi (RPi), a battery pack, a camera and a cap on which to mount it. As for the software, it included:

• Alexa Skills Kit (ASK)
• Amazon Web Services (AWS)
• DynamoDB
• Microsoft Cognitive Services (MSCS)
• Custom Python code to run RPi to interface with the camera and MSCS

The goal was straightforward. A visually-impaired user could ask Alexa what is in front of them. Alexa would vocalize the scene, allowing the person to navigate safely wherever he or she may be.

How the Smart Cap works

How do the pieces all fit together?

Chugh says there are two distinct parts.

First, the image capture and analysis:

• As the Smart Cap wearer walks down the street, a Python script on the RPi directs the camera to take pictures every two seconds.
• Another program sends the image to MSCS using RPi’s WiFi / phone connection.
• MSCS returns a text description of the image with relevant keywords.
• The description is stored on RPi, then sent via AWS to be stored on DynamoDB.

Now comes the Alexa skill:

• The wearer says “Alexa, ask Smart Cap to describe the scene” or “Alexa, ask Smart Cap what is in front of me”.
• The skill uses AWS Lambda to retrieve and parse the latest value from DynamoDB.
• Alexa responds with the description and keywords via the speaker or bone conduction headphones.

Doing more for the visually-impaired and the world

Chugh will graduate from CMU in December. What does he plan to do next?

He says there’s far more the technology can do than provide the basic information the Smart Cap prototype does now. He hopes the Hackster competition will shine some limelight on his concept, attracting an investor to help him bring a full product to fruition.

For example, it could help navigate both indoors and outdoors, determine location and context by using visible objects, and read books and signs. It could even recognize the face and emotional state of a person in the scene.

Chugh believes artificial intelligences like Alexa will give rise to solutions to some of the world’s toughest problems. “I am driven to solve the problems for people who are at the bottom of the economic pyramid. Right now I want to go where I can maximize my opportunities to make an impact on the world.”

Get Started with Alexa Skills Kit

Are you ready to build your first (or next) Alexa skill? Build a custom skill or use one of our easy tutorials to get started quickly.

Share other innovative ways you’re using Alexa in your life. Tweet us @alexadevs with hashtag #AlexaDevStory.