As one of the world’s largest food companies, Nestlé aims to help consumers transform simple ingredients into delicious meals. The company offers recipes and mealtime inspirations through its website, GoodNes.com.

Nestlé’s Silicon Valley Innovation Outpost (SVIO) is focused on finding ways to enrich its customer experience using new technologies. With Alexa-enabled devices appearing in millions of homes, the team saw an opportunity to “elevate the kitchen experience with a cookbook consumers can talk to,” according to Josh Baillon, Digital Innovation Manager at SVIO.

“My Alexa-enabled device is in my kitchen, just like those of countless other consumers. It seemed like a no-brainer to build a skill that can help people cook, and take it to the next level.” says Baillon.

Enhancing the Cooking Experience with Alexa

Nestlé set out to create a custom skill for Alexa to deliver a voice-first, hands-free cooking assistant. Nestlé’s GoodNes skill for Alexa inspires people to cook, connects them with the right recipe, and provides step-by-step help along the way.

The skill is accompanied by a unique GoodNes visual guide, which connects to consumers’ web browser on their computer or tablet to offer what Nestlé calls a “visual voice browsing” experience. Consumers can use Alexa to search for recipes for breakfast, main dish, and dessert, and receive a combination of voice and synchronized visual guidance in response. Consumers can view recipe steps, nutritional information, ingredients, and cooking videos in their browser. They can ask Alexa to email a shopping list and even ask Alexa to play music in the background as well.

To enable the visual guide, consumers simply provide their email address when enabling the GoodNes skill, which will generate a unique URL that is emailed to them. Once consumers click on the link, they can instantly access their visual guide from any computer or tablet, providing them with a rich cooking experience. Although using the visual guide is optional, feedback from consumers is that they love it.

See how the skill works in this video by Nestlé:

Building Quickly and Easily with the Help of Mobiquity’s Global Alexa Lab

It took the team three weeks to turn the initial idea into a single-user prototype, says Baillon. Using that prototype, the team gathered initial feedback from consumer testing and presented the prototype to Nestlé’s top executives. The prototype skill impressed key decision makers who gave Baillon and the SVIO team the green light to develop an MVP skill, conduct further consumer testing, and launch it to market.

To develop the skill, Nestlé teamed up with Mobiquity Inc., a third-party digital engagement provider with deep experience in building for voice. Mobiquity’s Global Alexa Lab, with expertise in voice design, was able to help the SVIO team design, build, and submit the skill quickly. Over the next few months, the two teams worked together to explore the possibilities of enriching the consumer experience in the kitchen, most notably by connecting the skill to a web browser.

“We worked with the GoodNes team from proof of concept through commercial design,” says Jonathan Patrizio, Head Technical Advisor at Mobiquity. “Josh had a vision for delivering recipes with a hands-free experience in a way that also incorporated beautiful images. It was Mobiquity’s job to help him figure out what it would take to make that vision a reality.”

This was the first time Mobiquity brought voice and visual experiences together in developing a skill. Patrizio says an important part of the process was to first build a robust voice experience that could be enhanced with visuals if customers so choose. The skill tailors its approach based on the presence of the visual guide. For example, if the guide is disabled, the skill provides additional information via voice to paint the full picture without the visuals.

“Pushing the envelope to think differently and get voice and visuals to work in unison made bringing this skill to life such a fun experience for the entire team, and Mobiquity was a fantastic partner in the process,” says Baillon.

Acting Quickly on Customer Feedback

Baillon says he was struck by the many ways in which designing for voice differs from designing for the web. For example, the language variances that people use, and the way users expect to jump around from feature to feature, were vastly different from the fixed paths defined when developing for the web. These differences make early consumer testing especially important, he says.

“We had many aha moments during our consumer testing phase,” says Baillon, adding that the testing helped the team figure out how to best support the voice experience with visual aids. For example, when users are viewing a recipe, prompts appear at the top of the screen and remind users how they can ask Alexa to list the ingredients, to repeat the step, or to move on to the next step.

The testing process was extensive, with quick turnarounds to iterate and improve the experience along the way.

“Nestlé has great test kitchens, making it possible for us to put a version of the skill in front of a user, get feedback, make changes, and test again the next day with 24-hour iterations,” says Patrizio.

Amazon Web Services: ‘All the Building Blocks You Need’

In addition to the Alexa Skills Kit, Nestlé and Mobiquity leveraged Amazon Web Services (AWS) to build and host the skill and responsive visual guide.

“At Nestlé, we use AWS for quite a few consumer-facing developments such as websites and IoT experiences. Leveraging many of the building blocks of AWS allowed us to move at an incredible pace,” says Baillon.

And for Mobiquity, a long history of working with AWS was an advantage.  

“We use AWS technology for almost everything we build. We can launch new developments quickly and efficiently, and the serverless architecture is just very scalable,” says Patrizio.

The GoodNes skill uses AWS IoT to communicate with the visual guide, leveraging WebSockets to connect the Alexa session and browser visual guide. It also uses other components such as Amazon Cognito to enable the browser to subscribe and receive messages.

“Key to the voice visual experience is rapid, real-time update of the visual guide in response to the utterance of a voice intent; any sluggishness would create a poor user experience. We paid close attention to the AWS serverless architecture, incorporating AWS IoT to tie the Alexa voice session to the browser. When you try it, you get a delightfully quick voice and visual response that feels great,” says Patrizio.

All user sessions, along with the content from the recipe repository, are stored in Amazon DynamoDB. Because user sessions are stored, the cooking assistant will begin again where a user left off if it times out. The skill also uses Amazon CloudSearch to enable search functionality, and Amazon S3 and Amazon Route 53 to host the visual guide web application.

“I love AWS because it is cost-effective and has all of the building blocks you need to quickly bring any experience to life,” says Baillon.

‘We’re in the Early Days’

Launched in April, the GoodNes skill has been well-received by customers.

“One of the things that gave me goosebumps during the testing phase was watching consumers use it for the first time and say, ‘Wow! This is really cool!’ They’d get really excited about how they would use the skill at home. And the feedback from our test group was resoundingly positive,” says Baillon.

Nestlé’s skill for Alexa is already seeing success with its consumers, and Nestlé has plans to continue adding features, recipes, and languages to evolve the skill over time. 

“We’re in the early days, and voice is going to be absolutely everywhere. Voice is a paradigm shift with how humans and computers interact. It’s the future, the way forward,” Baillon says. 

Build a Skill, Get a Free Echo Dot and More

We’d like to help you bring your big idea to life with Alexa. Publish a custom skill in August, and we’ll send you a free Echo Dot so you can experiment and daydream. You can also earn limited-edition Alexa developer socks. Learn more about our promotion, and start building with the Alexa Skills Kit today.