Motivations
The combination of the Internet of Things and Artificial Intelligence has made it possible to introduce numerous automations in our daily environments. Many new interesting possibilities and opportunities have been enabled, but there are also risks and problems. Often these problems are originated from approaches that have not been able to consider the users’ viewpoint sufficiently. We need to empower people in order to actually understand the automations in their surroundings environments, modify them, and create new ones, even if they have no programming knowledge.
The course discusses these problems and some possible solutions to provide people with the possibility to control and create their daily automations.
Benefits
The main technological trends of recent years have been the Internet of Things and Artificial Intelligence. Their combination has made it possible to introduce numerous automations that can manifest themselves in different ways in our daily environments. Many new possibilities and opportunities have been created, but also risks and problems. There can be intelligent services that eventually generate actions that do not match the real user needs. The introduced automations can generate unwanted effects. People may have difficulties in understanding the automatically generated automations. Often these problems are originated from approaches that have not been able to consider the human point of view sufficiently. In particular, the user has often been considered as a passive element with respect to the new possibilities instead of being the central subject. People in their lives often have dynamic needs, which sometimes are originated from episodes, even unpredictable ones. The most effective automations are often the ones that can be dynamically customized and created to meet these changing and different needs that only the users know completely. Thus, we need to empower people in order to actually understand the automations active in their surrounding environments, modify them, and create new ones, even if they have no particular programming knowledge.
The goal is to obtain “humanations” : automations that users can understand and modify.
For such reasons it is of paramount importance that designers and developers be aware of such issues and of some possible solutions to provide people with the ability to control and create their daily automations. Thus, this course aims to allow attendees to gain knowledge and skills in addressing problems and solutions involved in end-user understanding, creation, control, monitoring, and debugging automations that can be deployed in their daily environments (home, office, shops, …). It will provide a discussion of the possible solutions in terms of concepts, techniques, and tools, with particular attention to those supporting the trigger-action paradigm. The course will discuss how to enable people who are not professional developers to indicate the various dynamic events and conditions that can occur in their contexts of use (considering aspects related to user, technology, environment), and the possible associated actions.
Outline
- Introduction Course
- The technological trends (IoT + AI)
- The dark side of intelligent automations
- Design criteria for transparency of intelligent environments
- Trigger-action programming
- Automation definition exercise
- Visual environments for end user creation of automations
- Exercise with tool for end user automation creation
- Intelligent automation recommendations
- Explainable end-user automation debugging
- Augmented reality-based support for automation control
- Final Discussion
Audience
The course is interesting for designers, developers, and researchers who want to understand the issues involved in introducing automations in daily environments, and the corresponding possible solutions that can empower end users in understanding, controlling, modifying and creating new ones. It also allows them to understand the relevant state of art in order to think about novel solutions in this area.
There is no particular prerequisite for attending the course. Some basic knowledge of Internet of Things technologies would make it easier to follow it, but all the relevant concepts will be introduced in such a way to be understandable also to those who are not familiar with them.
It is possible to register to the course at the CHI 2022 Registration Web site as part of the CHI 2022 conference. Registration fee includes slides and electronic notes and is USD 25.
Instructor
Fabio Paternò is Research Director at CNR-ISTI, where he founded and leads the Laboratory on Human Interfaces in Information Systems. He has been author or co-author of 300+ publications in peer-reviewed conferences or journals. He has been co-editor of books and journal special issues relevant for the topics addressed in the course. He is an ACM Distinguished Scientist and a SIGCHI Academy Member. He has already given courses or tutorials at CHI, INTERACT, Mobile HCI; HCI International. For several years he has been the scientific coordinator of various projects related to the course topics such as the AAL PETAL project and the PRIN EMPATHY project
Example Tools for Creating Daily Automations
- Node Red Flow-based development tool for visual programming
- IFTTT Wizard-based tool that allows user to create single trigger-action rules
- TAREME Wizard-based tool that allows user to create multiple trigger-action rules
- Block Composer Block-based tool that allows user to create multiple trigger-action rules
References
- Will Brackenbury, Abhimanyu Deora, Jillian Ritchey, Jason Vallee, Weijia He, Guan Wang, Michael L. Littman, Blase Ur: How Users Interpret Bugs in Trigger-Action Programming. CHI 2019: 552
- R. Ariano, M. Manca, F. Paternò, C. Santoro, Smartphone-based Augmented Reality for End-User Creation of Home Automations, Behaviour & Information Technology, 2022, https://www.tandfonline.com/doi/full/10.1080/0144929X.2021.2017482
- S. Coppers, D. Vanacken, K. Luyten, FORTNIoT: Intelligible Predictions to Improve User Understanding of Smart Home Behavior, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 4, Issue 4, December 2020, Article No.: 124, pp 1–24
- Fulvio Corno, Luigi De Russis, Alberto Monge Roffarello: Empowering End Users in Debugging Trigger-Action Rules. CHI 2019: 388
- Chris Elsden, Tom Feltwell, Shaun W. Lawson, John Vines: Recipes for Programmable Money. CHI 2019: 251
- Simone Gallo, Marco Manca, Andrea Mattioli, Fabio Paternò, Carmen Santoro (2021) Comparative Analysis of Composition Paradigms for Personalization Rules in IoT Settings. End-User Development. IS-EUD 2021. Lecture Notes in Computer Science, vol 12724. Springer
- Giuseppe Ghiani, Marco Manca, Fabio Paternò, and Carmen Santoro, 2017. Personalization of Context-dependent Applications through Trigger-Action Rules . ACM Transactions on Computer-Human Interaction, 24(2), ACM, Article 14, 33 pages.
- W He, J Martinez, R Padhi, L Zhang, B Ur, When smart devices are stupid: negative experiences using home smart devices, 2019 IEEE Security and Privacy Workshops (SPW), 150-155
- Justin Huang, and Maya Cakmak. 2015. Supporting mental model accuracy in trigger-action programming. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '15). ACM, New York, NY, USA, 215-225.
- Nicola Leonardi, Marco Manca, Fabio Paternò, Carmen Santoro, Trigger-Action Programming for Personalising Humanoid Robot Behaviour, ACM Conference on Human Factors in Computing Systems (CHI'19), Glasgow, Paper 445. 1.
- Q. Vera Liao, Daniel M Gruen, Sarah Miller, Questioning the AI: Informing Design Practices for Explainable AI User Experiences, CHI 2020
- Marco Manca, Fabio Paternò, Carmen Santoro, Remote Monitoring of End-User Created Automations in Field Trials, Journal of Ambient Intelligence and Humanized Computing, 2021
- Marco Manca, Fabio Paternò, Carmen Santoro, Luca Corcella, Supporting end-user debugging of trigger-action rules for IoT applications, International Journal of Human-Computer Studies, 2019, Vol.123, 56-69
- Panos Markopoulos, Jeffrey Nichols, Fabio Paternò and Volkmar Pipek. 2017. End-User Development for the Internet of Things, ACM Transactions on Computer-Human Interaction (TOCHI) Vol- 24 Issue 2, 9, May 2017
- Andrea Mattioli and Fabio Paternò. 2021. Recommendations for creating trigger-action rules in a block-based environment, Behaviour & Information Technology, 2021
- Fabio Paternò and Carmen Santoro. 2019. End-user development for personalizing applications, things, and robots. International Journal of Human-Computer Studies, 131, 120-130.
- Antti Salovaara, Andrea Bellucci, Andrea Vianello, and Giulio Jacucci. 2021. Programmable Smart Home Toolkits Should Better Address Households’ Social Needs. In CHI Conference on Human Factors in Computing Systems (CHI ’21), May 8–13, 2021, Yokohama, Japan. ACM, New York, NY, USA, 14 pages.
- Vijay Srinivasan, Christian Koehler, and Hongxia Jin. 2018. RuleSelector: Selecting Conditional Action Rules from User Behavior Patterns . Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 1, Article 35 (March 2018), 34 pages.
- Blase Ur, Erin A McManus, Melwyn Pak Yong, Michael L Littman. 2014. Practical trigger-action programming in the smart home. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14). ACM, New York, NY, USA, 803–812.
- Rayoung Yang, Mark Webster Newman, Learning from a learning thermostat: lessons for intelligent systems for the home. 2013. ACM international joint conference on Pervasive and ubiquitous computing, pp. 93-102
- L. Zhang, W. He, O. Morkved, V. Zhao, M. L. Littman, S. Lu, and B. Ur. 2020. Trace2TAP: Synthesizing Trigger-Action Programs from Traces of Behavior. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 3, Article 104 (September 2020), 26 pages. https://doi.org/10.1145/3411838