Cooking with Google Assistant
Evaluating the first-time user experience of the Cooking with Google Assistant feature on the Google Home Mini.
Virtual assistant technologies…
have become deeply engrained in our lives, particularly with the innovation and adoption of smart speakers. We’ve come to utilize them for more advanced tasks than just playing music and setting alarms. Now, virtual voice assistants can turn on your lights when you get home, share traffic information for your commute, do your shopping, and even help you cook a recipe.
These advanced capabilities can only add as much value as they are perceived as being useful and joyful. When we expect to be able to retrieve information or perform an action with the help of our virtual voice assistant it’s frustrating to hear “Something went wrong” or “Sorry, I can’t seem to find that information right now.” These advanced features often supplement our usual activities. Diving deeper into the first time experience and how it compares to our current mental models of an activity gives us the opportunity to craft a fuller picture of the details users expect to have in order to accomplish a task.
Context
During the HCDE 517 Usability Studies course, we were tasked with determining a research area and running a mixed methods usability study. We did not formally partner with Google for this project.
My Contribution
I contributed the idea to do our project on Google Home Mini (GHM), so I had a large role in study scope & design. I defined user tasks, problem severity, and wrote the usability evaluation script based on the voice prompts on GHM. I administered usability evaluations and pre-/post-test interviews which included System Usability Score and Net Promotor Score metrics. I synthesized qualitative & quantitative data, and presented insights & recommendations.
Tags Evaluative; Usability Testing; Voice UI
Role UX Researcher
Duration January - March 2020
Team Devin Ajimine, Kristen Shiflett, Grace Yang
Outcome
Our team was able to determine valuable insights and develop product recommendations that could have impactful usability improvements for all voice UI experiences going forward.
Additionally, I gained experience with both quantitative and qualitative research methods. I also gained experience collecting satisfaction metrics, such as Net Promoter Score and System Usability Score.
Learnings
Voice UI devices, such as Google Home and Alexa, are relatively new technologies. This is a space with a lot of room for exploration, evaluation, and improvement. Our study suggests that for voice UI to be more effective for users it’s best to offer users robust options for questions asking, as a lot of the information that is usually gleaned by traditional UI is lost in the voice interaction.
Background
Google Home is a voice-controlled smart speaker for your home. Cooking with Google Assistant is a feature where someone is able to use the Google Home voice prompts to walk them through cooking a recipe. Google will help the user find a recipe, let them know the ingredients, and list the steps needed to prepare the recipe.
Fig.1 below shows how the Cooking with Google Assistant feature works:
[Figure 1] How to use Cooking with Google Assistant: (1) Find a recipe, (2) list ingredients, (3) list recipe steps.
Goals
In this study, we wanted to…
explore the user’s first-time experience of the Cooking with Google Assistant feature on the Google Home Mini
&
understand what obstacles users encountered while using the feature for the first time
Process
Participant Selection
In order to minimize user error and focus specifically on the first-time user experience of the feature, we wanted to make sure our participants had relevant knowledge to the task(s) at hand. This meant we wanted participants who…
Cook or bake weekly, so we didn’t have to worry about competing with their knowledge in the kitchen.
Use Google Home Mini daily, so we didn’t have to worry about competing with their understanding of the possible voice prompts.
Have never used the Cooking with Google Assistant feature, so we can study their first-time experience.
Our team sent out a survey through our network to find participants. We ended up with 5 participants who met our criteria and were able to be a part of our study.
Usability Evaluations
To prepare our usability study script, I walked through the current process of the Cooking with Google Assistant feature using a chocolate chip cookie recipe. This allowed me to familiarize myself with the feature and the specific prompts it would use throughout the experience. I was then able to use these specific prompts in the usability study script.
Our team then conducted 5 usability evaluations in users’ homes, where we observed user behavior and collected both qualitative and quantitative data using techniques such as observation, Likert scales, and Think-Aloud Protocol. In Fig.2 below, you can see an example of some of the data we collected and how it was collected during the study. I developed a key for the team to use to keep track which commands each participant used during their interaction with GHM and mark any errors.
Participants were prompted to complete the three main tasks as outlined by the Cooking with Google Assistant feature.
Our team provided each of the participants with the ingredients to bake a specific chocolate chip cookie recipe, so each participant was compensated for their time in cookies. Participants were asked to use Google Home Mini to find the recipe and proceed with the Cooking with Google Assistant feature to make the chocolate chip cookies.
[Figure 2] Example of data that was collected during the usability evaluations.
Did this match your expectation of what was going to happen?
“Yes, aside from not knowing what to do with all of the ingredients. I guess I was just supposed to get them out, and then I will have to ask again for all of the quantities.”
-Participant 2
Findings
Call to Action During Ingredients Step
Frequency: 5/5 Severity: Severe
Participants were confused about what to do during ingredients step and started mixing ingredients only to realize they were supposed to be portioning them out as they had specific mixing steps to follow later.
Recommendation:
Give users more information about what they should be doing during the ingredients phase.
[ex: Google says, “Use this time to portion out your ingredients.”]
Feasibility
Frequency: 5/5 Severity: Severe
Participants were unsure if the recipe was feasible, as certain information they would gather in a cookbook or online recipe was missing from the Google Home Mini voice UI. Participants wondered, “Do I have the right equipment?” “How long does it take to complete?” “How many servings does the recipe make?”
Recommendation:
Develop additional prompts a user can ask while searching for a recipe in order to find out if they have the tools and time to make this recipe.
[ex: The user can ask, “Hey Google, how long does this recipe take?”]
Recall of Ingredient Quantity
Frequency: 4/5 Severity: Moderate
Participants wanted to hear ingredient quantities during the recipe steps phase. Participants had to ask Google for quantities of ingredients during every step.
Recommendation:
Say ingredient quantities during the steps.
[ex: Google says, “Mix 2 cups of flour with 1 cup of butter.”]
For additional findings from this research, please reach out to me at jmbrown864@gmail.com.
Reflection
What went well?
We conducted the usability study on our team members first, which was helpful to adequately address any holes or issues in our usability testing script. Additionally, we chose to collect specific information about the number of steps, exact commands used, and errors encountered by participants during each of the steps. These detailed quantitative observations allowed for a more robust understanding of the issues users faced in respect to each task, in addition to the feature as a whole.
What could have gone differently or better?
Firstly, participants represented similar demographics, so we missed out on age and lifestyle implications. If we had more time, it would have been interesting to see how the study results would be different if we were working with users who were less familiar with the Google Assistant.
Second, our script was too prescriptive during the recipe discovery task, so we missed out on insights around this behavior. The script I wrote did not take into account the variability of the search results that Google would retrieve for different participants. More in depth research should be performed around the recipe discovery and connecting with Google Home portions of the Cooking with Google Assistant feature.
Lastly, asking participants to cook instead of bake may yield different results, since baking is more particular about quantities.
[Above & Below] Participants making chocolate chip cookies during usability evaluation.
