Improving the Privacy, Usability, and Context-Awareness of Smart Speakers

Abstract

Smart speakers, such as the Amazon Echo or Google Home, have become ubiquitous in our daily lives due to their convenience, which offers interactive actions through the use of simple voice commands. These devices allow users to issue a wide range of commands for a variety of services. Users can ask in natural language questions about the weather, stock market, online shopping orders, and other general information. These devices can also be used to control lights, and heating systems, and set timers and alarms in the smart home. However, as smart speaker systems become more prevalent, new security and privacy, usability, and context awareness concerns will need to be explored and addressed. In this dissertation, we carry out the effort to understand and mitigate privacy leaks from third-party applications, improve usability testing using interactability metrics, and improve context-awareness in a multi-occupant home using background sounds. We first study the privacy risks resulting from smart speaker apps developed by third-party developers. Having a device permanently on and always listening led to concerns over user privacy. In addition, the use of the third-party app on smart speaker platforms introduces arguably more serious privacy risks than using only the platform's built-in apps, due to the open nature of the app marketplaces. We explore how an adversary can efficiently create a valid smart speaker app to eavesdrop on users. We developed three different strategies for implementing a malicious app. To mitigate this threat, we propose a strategy for users to limit the success of this adversary. We designed a measurement app to look at the effect of various environmental factors in the home impacting what the third party can hear, and therefore provide users with a recommendation to place their smart speaker in locations that limit the success of this adversary. Next, we propose the idea of an interactability score to quantify how well a smart speaker app can accept potentially different ways a user may express their commands. However, voice-generated input data creates many unpredictable test cases since there are many different ways of how someone will express the same intention. In addition, each third-party developer could implement their own voice commands, making it difficult for users to remember what commands a particular app can process. The architecture of current smart speaker apps further complicates the testing process since the app is hosted on the smart speaker platform as a black-box. Therefore, we develop a testing framework to automatically and systematically evaluate the interactability of the smart speaker applications. It measures how well an app has been implemented to accept different kinds of user interaction. We also focus on improving context-awareness access control for smart speakers. The convenience of these devices is tempered by the possibility of performing unintended or intended actions. At home, the device is usually placed in a fixed location and accessed by multiple people with complex relationships between them, and these complex relationships can lead to complex access control requirements, where the context factors and interpersonal relationships should play a significant role. We design a system to be run on a smart speaker that makes use of the sounds in the home to estimate the current state of the house, e.g. number of occupants, activities being engaged, social relation of occupants, etc. This context information is used to decide whether to execute the command, prompt for confirmation or reject the command entirely. We also designed a simple pictorial configuration utility to help non-expert users configure their access rules.