Purpose and Motivation

By 2060 there will be about 98 million Americans over the age of 65 and as more people live longer, a larger percentage of the population will be affected by cognitive decline. Dementia has no cure but some symptoms can be treated. Therefore, early detection could greatly improve a person’s quality of life. 

Smart speakers offer a potential platform for voice user interface (VUI) tools that can aid the diagnosis of cognitive decline and dementia in older adults. I conducted the following research and design activities to explore this opportunity:

1. Implemented a cognitive screener as a VUI using Google Assistant with an accompanying mockup of a clinical dashboard.
2. Interviewed older adults, aging and psychology experts, and care takers.
3. Evaluated the prototype VUI screener in lab studies and a 5-week field deployment.
4. Compared multiple age groups’ perceptions about VUI usability. 

Executive Summary

– Interviews with clinical experts indicated a smart speaker administered screener would be potentially useful.

– Older adults performed similarly to young adults when completing smart speaker tasks. 

– Overall, older adults are able, willing, and excited to use voice based technology. 

The VUI cognitive screener prototype

There are numerous short dementia screening tools. The Mini-Mental State Examination (MMSE) is a popular tool for detecting warning signs of dementia and can be administered in about 10 minutes. However a limitation of the MMSE is that it requires the use of paper and writing materials. A different test, the Callahan Six-Item Screener, has been validated in multiple countries and performs similarly and sometimes better than the MMSE at detecting cognitive decline. Since this test can be administered in minutes and does not require any physical supplies, it was a good candidate for being translated into a VUI. 

The Callahan Six-Item Screener was implemented as an action using the Google Dialogue Flow developer interface. The action was named “Weekly Check In” (WCI) to remove possible negative connotations of being a cognitive assessment tool. There were a few changes made to the screener during its conversion to an action. First, the three-word recall portion was altered to avoid learning effects and boredom in case of weekly administration. Three words were selected at random from a list of items compiled from a different cognitive screener. The WCI also used friendly syntax such as saying “Thank you” at the end of an interaction.

The WCI was programmed to provide a series of responses if it did not recognize a user’s input. This included repeating the question and encouraging a response by using helper-phrases (e.g., “Please repeat that” or “Please try again.”). If the user’s input was not recognized after two helper-prompts, the action would provide an apology and end the interaction. 

Axure design software was used to create wireframes representing an analytics dashboard. It included information about scores, performance over time, user inputs, time of activation, and the option to view exact transcripts of all interactions with the WCI. The dashboard was not interactive and was shown to experts during interviews for feedback. 

Figure 1. Analytics Dashboard

Research Methods and Detailed Results

A. Interviews

To understand the feasibility of using a VUI to monitor cognition, I met with three populations including: 10 primary stakeholders all 65 years of age or older, 7 secondary stakeholders including subject matter experts and cognition and aging specialists (professors, university researchers, or staff researchers) from organizations focused on aging research, and 5 caretakers who were working professionals with experience in older adult care, geriatrics, or assisted living community management. 

I conducted different semi- structured interviews with each of the three groups. Notes from the interviews were compiled, organized by question, and then analyzed using a thematic analysis technic.

i. Results: Experts

All participants believed a smart speaker administered screener would be potentially useful. There were four emergent themes and sub-themes from the interviews:

1. Verbally administered screeners will have limited scope and only test specific cognitive components. 
2. There were concerns about the administration of the screener because participants could possibly cheat or have variable performance due to environmental conditions. 
3. Participants were particularly worried about how the data would be shared and doubted users would willingly provide data that could compromise their independent living. The experts wanted as much information about the results as possible, but speculated that primary users of the system would just want an indication of their cognitive status.
4. Participants mentioned that establishing a baseline would be essential and they would only want to be warned when a user significantly differed from that baseline. 

ii. Results: Primary Stakeholders

This set of interviews revealed three main themes:

1. Participants were hesitant to use technology that required learning new skills or behavior changes. 
2. All participants expressed privacy concerns with devices that could listen to them because of hearing about security breaches in the news or from friends. 
3. Social activity was central to their strategies for staying cognitively active. This includes engaging with friends, clubs, and religious organizations. 

iii. Results: Caretakers

There were two key themes from the caretaker interviews:

1. Cognitive tracking VUI design must fit the community where it is situated. 
2. Any new cognitive tracking service would need to respect residents’ dignity, independence, and family needs. Therefore it would require residents’, and potentially family members’, full consent and understanding of how data could be used for possible diagnosis. 

B. General VUI Task Based Usability Tests

To study how comfortable and well older adults could use VUIs in general I conducted a round of task based usability tests. Each participant was instructed to complete a series of basic tasks taken from the instruction manual included with Google Home. I gathered time on task, qualitative, and observational data during this process. There was also a short semi structured interview after the tasks were completed. Finally, participants complete a System Usability Scale (SUS) questionnaire. 

In addition to the older adults, college aged undergraduate and graduate students completed the same tasks to serve as comparison populations. The graduate students were familiar with human computer interaction concepts and could therefore be considered a separate population from the undergraduates. 

i. Results

There was no statistical difference in the total time (in seconds) to complete the tasks between the undergraduates, older adults, and expert groups. There were also no statistically significant differences between the three groups’ SUS score ratings. All the SUS score ratings were indicative of moderately positive impressions of the system. 

C. Weekly Check In Usability Tests

After establishing that older adults could use VUIs and were interested in the concept of using them, I conducted another round of task based usability sessions. During this testing older adult participants were instructed to interact with the WCI and then provide feedback about the experience. Once again, I gathered a mix of qualitative and quantitative data, as well as SUS scores.

i. Results

Participants felt the action was easy to use and some thought it was like a game. Multiple participants wanted immediate feedback , either verbally or visually, after using the WCI. Participants mentioned they might forget how to launch the WCI and would need reminders to interact with it. Two participants struggled with speaking in a manner the Google Home could recognize, but after repeating their commands, the speaker accepted their responses. 

D. 5 week field deployment

Lab based usability studies can provide many insights, but allowing people to interact with technology on their own can uncover additional useful information. This is especially true of VUIs since the environment they are used in can impact the experience. 

I recruited an older adult who was willing to install and use a Google Home in her living environment for a 5 week period. After the initial instillation, I followed the same usability test protocol that was performed with the other participants, but then allowed the participant to use the device however she wanted. The participant was also required to use the Weekly Check In at least once a week. I gathered feedback about the WCI and the general experience of using the Google Home during weekly phone interviews.  

i. Results 

The SUS scores during weeks 1, 3, and 5 were respectively: 82.5, 100, and 95. The interviews revealed three major themes: 

1. Privacy fears were the most prevalent topic. Those fears were exacerbated by online and TV news related to privacy leaks.
2. The Google Home was seen as a fun, but unnecessary accessory. After the first week the participant felt mastery over the basic functions and was not motivated or sure how to try new functions. 
3. The VUI platform presented challenges including error handling, speech recognition, technical issues, and speech speed. The participant was unsure when she provided invalid inputs compared to when the speaker had a system level problem (e.g. no Wi-Fi connection.). 

Conclusions

The study had promising results, but there are a number of limitations worth noting. First, the study was conducted in the USA which has a higher ownership rate of home speakers than other western countries. Second, future studies should aim to have a representative sample of male and female seniors. Finally, while the Google platform was used for testing, other platforms may have different usability issues. 

The interviews revealed there is potential for smart speaker based cognitive tracking; however, more testing for validity and reliability along with robust data analysis tools is needed before the platform should be adopted.  Older adults were interested and capable of using the Google Home, but had lingering cost and privacy concerns. 

If some of the issues revealed in the usability tests and interviews can be overcome, a simple voice based cognitive tracking tool could be helpful for both physicians and older adults. This is a topic rich for exploration and additional research.