Results

Deliverable D1.1-1: Concepts, Requirements and Models

Due to the ever-increasing opportunity to integrate IoT devices into everyday objects and spaces, IoT technologies today offer new means for monitoring and controlling appliances, as well as sensing human activities. They enable the creation of novel interactive systems that support new kinds of activities to empower people in different contexts of everyday life. This is having an unprecedented impact on the activities that can take place in environments provided with capabilities of “augmented sensing and interaction”.

This deliverable focuses on interactive IoT systems, i.e., on a human-centered vision of IoT where user interaction is central. This vision raises new modelling requirements that go beyond representing hardware, software, and connectivity features of IoT-enabled systems. Identifying modelling primitives for interactive IoT systems can pave the way to new methodologies and tools, whose added value is based on a conceptual layer that abstracts from the underlying IoT technology and focuses on the IoT-enabled capabilities of smart objects and smart spaces that are relevant for the design of interactive experiences.

Identifying such abstractions is then relevant to empower people to create and tailor their smart spaces. Modelling smart space capabilities is important to understand how end users can make sense of such capabilities and configure their smart spaces. Adequate conceptual models can indeed serve as a knowledge base to represent the most relevant concepts that can be the building blocks for the End-User Development (EUD) of interactive systems based on IoT technologies. This deliverable discusses concepts, requirements and models that can support i) the modelling of interactive IoT applications and ii) the design of platforms for the EUD of smart interactive spaces. It is structured as follows:

Section 1 introduces our vision on Interactive IoT.
Section 2 defines the main concepts that are the objects of the document
Section 3 discusses the relevant dimensions for the design of smart interactive spaces. The goal is to identify the different concerns characterizing the design of these applications, to identify in turn those aspects that must be addressed by End-User Development platforms.
Section 4 focuses on some requirements for the design of platforms for the EUD of smart spaces. It starts by discussing mental models and programming paradigms that can support computational thinking; thus, it deepens guidelines for the definition of EUD environments, resulting from the literature as well as from the past research conducted by the EMPATHY research units.
Section 5 focuses on requirements for EUD that are specific of three selected application domains, namely Ambient Assisted Living, Cultural Heritage, Education and Learning.
Section 6 finally draws the conclusions.

Deliverable D1.1-2: Concepts, Requirements and Models / Deliverable D1.2-2: Meta-design for IoT environments

This document refines the previous deliverables D1-1.1 and D1-2.1 and illustrates a consolidated set of reference concepts and models (e.g., UI models, users and context models, users’ mental models) for the relevant aspects of the EUD of interactive IoT applications. The main contributions are:

The identification of the dimensions characterizing the design of smart interactive spaces.
The identification of concepts related to the users’ mental models on paradigms for the EUD of smart interactive environments.
The proposal of a reference architecture for the EMPATHY platform.
The identification of a methodological framework clarifying the different activities needed to configure and adopt the EMAPTHY platform for the EUD of IoT systems.

By adopting a human-centred perspective on the design of interactive smart spaces, the document analyses the modeling dimensions that can help highlight relevant concerns to be addressed to enable the EUD of smart interactive environments. If clarified and adequately managed, these concepts can guide the integration of the different components so far identified within the EMPATHY reference architecture, that in turn can lead to a meta-design framework involving different stakeholders: from technology-savvy people, to the end-user developers, to the final users.

The deliverable D1-1.1 already identified two main modelling dimensions as the most relevant for the design of interactive smart spaces, which relate to:

the actors that determine the behaviour of a smart space, i.e., the user acting within the smart space and the devices and digital resources installed in it;
the rules and their orchestration within activities, which govern the behaviour of the involved actors.

In this deliverable, these two dimensions are analyzed in detail. In particular, the first part of the document presents modelling abstractions that go beyond representing hardware, software, and connectivity features of IoT devices. Some dimensions, specifically related to the interaction of the users with a smart space, emerge as crucial modelling aspects to capture the interplay occurring not only among multiple connected IoT devices but also among the users and the materials or the spaces embedding such devices. Then the document distils those concepts that, within the EMPATHY reference architecture, can guide the integration of the different modules supporting the specification of trigger-action rules, also addressing security and privacy issues, and the provision of recommendations.

The model also guides the definition of a flow of activities, within a meta-design framework where different roles are assigned to different stakeholders that can communicate across their activities, also depending on the separation of concerns that the model introduces.

Finally, the document describes the integration of different components into a unified reference architecture that characterizes the EMPATHY approach to the EUD of IoT systems.

The document is organized as follow:

Section 1 shortly summarizes the need for a human-centric perspective on the design of smart environments
Section 2 illustrates the main modelling abstractions for the two identified dimensions. By means of a case study for the design of a multisensory smart space, the Magic Room, it also discusses the methodological framework that identifies activities involving different stakeholders.
Section 3 describes the EMPATHY reference architecture, in which models play an important role as they help specify descriptors that can ensure the interoperability among the different components offering different services (recommendations, security&privacy, context and state management).
Section 4 draws the conclusions.
A final Appendix reports an example of specification for an IoT device, based on the descriptors introduced in Section 3.

Deliverable D1.2-1: Meta-design for IoT environments

Internet of Things (IoT) systems are increasingly used in any sector: home, retail, industry, agriculture among others. This technological trend provides great opportunities; a very important issue is however how to enable users to directly tailor and evolve smart environments, without any (or with limited) intermediations by software professionals. The “End-User Development for the Internet of Things” is a topic that is gaining momentum in the literature and that offers a picture of a fast-evolving research field that aims to enable end users to control and modify the behavior of smart devices, by means of paradigms that can simplify the definition of trigger-action rules.

This document focuses on methods for the End-User Development (EUD) of IoT systems and on the shift in the software development process that these methods require. Indeed, it is necessary to adopt approaches for meta- design (i.e., design for design) for defining design environments able to support the end-user needs, i.e., environments that allow the diverse stakeholders to participate in the creation of the final applications.

Starting from on a survey of notable approaches, this document outlines a methodological framework for the EUD of IoT systems. It discusses the need for modifying the software development lifecycle by means of methods and enabling tools that can empower the end users to customize or create their applications. The document also identifies some lacks that could be the object of the next activities in the EMPATHY project.

The document is organized as follows:

Section 1 introduce the notion of EUD.
Section 2 discusses how end users can be involved in the whole software life cycle and how EUD methods, and in particular meta-design, determine a shift toward new paradigms for software design and development.
Section 3 illustrates some methods for the EUD of IoT and their enabling platforms
Based on the analysis of Section 3, Section 4 concludes the document by identifying limits and challenges to be addressed.

Deliverable D2.1-1: Tailoring Environments

In order to fully exploit the potential of the Internet of Things (IoT), it is fundamental to support End User Development (EUD) in such a way to allow even those who are not professional developers to create the automations most relevant for them. For this purpose, tailoring environments have to allow users to indicate how to connect their devices and services in a way that “resonate” with their previous knowledge. A relevant approach is Trigger-Action Programming (TAP), an approach to defining these interactions using cause-effect rules, that has been found understandable even by users without prior experience with programming.

Although many automations of interest for a user may be modelled with a simple rule that uses a trigger and an action, many others need to consider a combination of them. Previous work proposed environments that aim to make simpler to users the interpretation and the creation of IoT rules. However, some challenges are still present in these systems, and can emerge for example from interactions between rules when the size of the environment scales-up, or from some subtle interpretations related to the temporal aspect of the elements that compound a rule. Support tools for explaining the effects of the rule execution are a valuable integration to tailoring environments to mitigate the aforementioned issues.

This document reports about the main aspects of the design space of tailoring environments for IoT applications, and how these have been explored by several contributions proposed in the last years, both in research and commercial context. Also, methods to provide support in the automation development activities have been identified and described. In this area, the EMPATHY partners have designed and developed three tailoring environments. A description of these environments has been provided referring to the dimensions of the design space identified, and comparing them with two widely used tools developed by professional organizations.

Another aspect worth discussing is how to extend such approaches to consider humanoid robots, which are composed of several sensors and actuators in a human-like shape. The possible behaviours of a social robot can be complex and dynamic, and in many use cases users do not have time and motivation to learn the required programming concepts via a low-level framework. Hence, there is the need for a flexible, usable, accessible environment, that permits the definition of expressive behaviours on a robot, and its seamlessly integration with the other devices and services.

In the conclusions we indicate relevant areas that have been underexplored and deserve more attention for future research activities in order to obtain more effective solutions able to empower users in creating and modifying the desired automations.

The document is organized as follows:

Section 1 introduces the role and usefulness of tailoring environments.
Section 2 defines an outline of the main characteristics of tailoring environments for IoT contexts.
Section 3 discusses the main features of the tailoring environments, and the associated methodologies that can be used to provide support to the users in development activities.
Section 4 introduces the environments developed by EMPATHY partners, using as comparison IFTTT and Node-RED.
Section 5 draws the conclusions and indications for future work.

Deliverable D2.1-2: Tailoring Environments

This deliverable builds on the D2.1-1, where we introduced visual-based tailoring environments aimed to be used by people without programming experience and how they can be used to deploy automations in different contexts. In this deliverable, we broaden the discussion taking into consideration two main types of contributions that can be used to extend the tailoring environment's capabilities, namely how to introduce intelligence capable external services and how to adopt different composition paradigms. The use of external services can provide users with further information and functionalities during the rule creation and analysis phases. Further composition paradigms are proposed to make the user’s development activities in the tailoring environment more engaging, intuitive and fast.

The document is organized as follows:

Section 2 introduces why there is the need to provide users with further services to support them during the rule creation, and defines the capabilities of the external services developed by EMPATHY partners and how they can be integrated into the tailoring environments.
Section 3 discusses other composition paradigms that can be used in this context, their role in supporting users during their activities, and their peculiar aspects.
Section 4 draws the conclusions and indications for future work.

Deliverable D2.1-3: Tailoring Environments User Interface Prototype

This deliverable builds on the D2.1-2, where we introduced different composition paradigms that can be used to support the user’s development activities in the tailoring environment. Indeed, the definition of automation rules often occurs using visual tailoring environments. However, these environments have some limitations, that can cause issues during the rule composition and hinder their adoption. In this deliverable, we analyze in more detail how these new composition paradigms can be implemented into tailoring systems to make the generation of automation rules more engaging, intuitive and fast. The implemented prototypes will be introduced and discussed.

The document is organized as follows:

Section 2 describes the Augmented Reality paradigm. After an introduction and a brief recap on the previous work about AR in the smart home environment, the two developed prototypes are discussed, explaining the motivation and the implementation of the former (Spontaneous Automation Creation, SAC), and how we capitalize on this experience to design a new approach, Perceivable Rules.
Section 3 discusses the Conversational paradigm introducing the RuleBot, a conversational agent to define automation. Its design and the implementation of the three main modules that comprise the application are presented.
Section 4 draws the conclusions and provides indications for future work.

Prototype P3.1-1 Secure context manager middleware prototype

IoT platforms are an attractive target for malicious attacks since they provide privileged access to user’s online services and physical devices. For instance, attackers could violate the privacy of the user by leaking personal information acquired by IoT devices or compromise IoT devices having a physical impact on a smart environment, which could represent a severe risk for the user’s safety.

Despite being a serious issue, end-users have no technical skills for understanding the danger their smart environment is exposed to. This lack of awareness leads them to think that they do not need to play an active role in managing the security of the smart environment. For this reason, end-users need to be educated and guided through the procedures required to develop effective defense mechanisms. Hence, smart and easy to use solutions should be developed to guide end-users during the specification of security mechanisms, offering an adequate level of abstraction and explanation, and embedding automation strategies.

This document introduces a prototype for supporting end-users in the specification and management of security and privacy aspects of their smart environments. The prototype relies on hardware and software solutions, which allow end-users to customize the smart environment behavior with rules focused on security barriers, and providing automation mechanisms that defend the smart environment in the initial phases of a malicious attack.

The document is organized as follows:

Section 1 introduces the security and privacy issues related to the Internet of Things.
Section 2 provides an overview of the Secure context manager middleware and describes the logic for managing security in IoT contexts.
Section 3 introduces the hardware architecture, and how it can help end-users in designing rules for configuring security behaviors.
Section 4 describes how the Secure context manager provides active network monitoring capabilities to defend the smart environment from external malicious attacks.
Section 5 provides indications for future work.

Prototype P3.1-2 Secure context manager middleware prototype

This deliverable reports the details of a prototype providing security mechanisms within IoT smart environments. The proposed solution is based on a hardware component, which acts as a middleware for monitoring the overall network status of the smart environment, which also enables users in the definition of custom behaviors. The implemented system is able to alert the users on the identified security threats through notifications on both smartphones and smartwatches. The system also integrates a module able to evaluate the security and privacy risks of trigger action rules at design time.

The document is structured as follows:

Section 2 provides an overview of the Secure context manager middleware and describes the logic for managing security in IoT contexts.
Section 3 introduces the hardware architecture, and how it can help end-users in designing rules for configuring security behaviors.
Section 4 describes how the Secure context manager provides active network monitoring capabilities, defending the smart environment from external malicious attacks.
Section 5 presents the classifier developed for identifying potential security and privacy risks within trigger-action rules.
Section 6 provides indications for future work.

Prototype P3.2-2 Behaviour analysis and recommender prototype

With the increasing availability of smart environments and IoT devices, end-users who do not have specific technical skills are increasingly confronted with the tasks of configuring intelligent objects. Smart environments are made up of a large number of heterogeneous and connected IoT components. For example, we can find in a single environment devices, operating systems and middleware solutions of different types that must be configured for so that the user can achieve their goals. Helping end-users to configure intelligent environments on their own can be non-trivial: on the one hand it is necessary to design usable composition interfaces, and on the other hand to help them make configuration choices in a domain that they can only know superficially. Among other things, the choices that the user must normally make are:

Which IoT components must make up the smart environment
Whether to establish an interaction between two components
Which event (trigger) should activate an interaction pattern
What behavior (action) should you have after a trigger event has been activated

In this document we will present a prototype that uses different types of recommendations aiming at help end-users in the choices of configuring smart environments. We assume that users will have their own user model (which is currently being defined) and choose the smart home domain as the use case.

Deliverable D4.1-1: Ambient Assisted Living Application Prototype

This deliverable reports on the ambient assisted living application prototype. One of the goals of Ambient Assisted Living (AAL) solutions is to extend the time that older adults can live independently in their preferred environments by using ICT technologies for personal healthcare. However, in order to be optimal, remote monitoring services and health-related interventions should be strongly personalised to specific individuals’ requirements, preferences, abilities and motivations, which can vary among the older adults, and even dynamically evolve over time for the same person depending on changing user needs and context-dependent conditions. The deliverable is based on experiences carried out with the TAREME personalization platform configured for the AAL domain.

The document is structured as follows:

Section 2 defines an outline of the main characteristics of the users involved in this application domain.
Section 3 introduces the main features of the prototype that has been created for addressing ambient-assisted living environments.
Section 4 introduces the type of personalization rules that have been created and deemed more suitable in this context.
Section 5 draws the conclusions and indications for future work.

Deliverable D4.1-2: Ambient Assisted Living Application Prototype

This deliverable reports on a novel ambient assisted living application prototype. One of the goals of Ambient Assisted Living (AAL) solutions is to be able to stimulate the cognitive resources of older adults. One of the innovative approaches to address this aspect is the use of serious games delivered through humanoid robots. Such devices are based on Internet of Things technologies since they are collections of sensors and actuators in a human shape. Serious games delivered through humanoid robots can provide an engaging way to perform exercises useful for training human memory, attention, processing and planning activities. In this perspective, the humanoid robot can exhibit different personalities, and propose the games in a way personalised to specific individuals’ requirements, preferences, abilities and motivations, which can vary among the older adults, and even dynamically evolve over time for the same person depending on changing user needs and health conditions.

The document is structured as follows:

Section 2 reports on a prototype supporting a music game, in which older adults had to answer questions regarding songs of the period when they were young. The prototype has been proposed in a cognitive stimulation programme (called “Train the Brain”), along with the same game provided through a tablet device. One limitation was that users interacted only through the screen available on the robot’s chest.
Section 3 describes a second game prototype providing exercises based on cooking. In this case, users can also interact through voice. The robot has been enriched with the possibility to exhibit its behaviour (voice, gestures, movements, …) according to two different personalities (introvert and extrovert) to provide more personalised support.
Section 4 introduces a new prototype, in which older adults enter some biographical information through a multimodal Web app, and then the robot proposes to them some exercises based on aspects contained in their memories in order to train their abilities.
Section 5 draws the conclusions and indications for future work.

Deliverable D4.2-1: Education application prototype

This deliverable describes an ducational application, GAIA, that is an interactive toy aiming to stimulate kids in the first years of primary school (5/8 years old) to play outdoor and explore natural elements. The device is a tangible band equipped with 4 bright buttons and 2 speakers tied on a tree. Children interact with the device through lights, sounds and touch. Taking advantages of the functionalities provided by the device, together with GAIA we designed a game to install on it. Considering an installation with multiple bands in a park, the children are driven by an educational story, from a band to another, in a "treasure hunt". During the entertaining task the children explore the nature, reflect on the the story and on the material they encounter and collect during their experience, and learn.

GAIA can be configured by teachers and educators by means of TERRA, a Web application with a visual environment that uses a visual metaphor taken from the inspirational cards proposed by a toolkit for the design of smart objects. TERRA allows users to have concrete feedback on the conceptualization of smart objects that they define by combining cards. Through TERRA, new behaviors for existing devices can be configured, or even new devices can be created, by combining a known set of sensors and actuators and their exposed events and actions. The SNaP card metaphor is thus used to define trigger-action rules. Sensors and actuators are represented through the icons used in the SNaP cards. Moreover, for each component, events and actions are also represented. Rules can be defined according to the EFESTO-5W paradigm (see Deliverable D1.1-1), which guides the end users through different steps for the selection of the elements composing the rule. The defined rules are then automatically transformed into code that can be deployed and executed directly on devices managed by a Raspberry PI controller.

The TERRA (Tool for dEsigning natuRe exploRAtions) visual environment consists of a Web application that leverages Node-RED and MQTT features to deploy the rules on the smart object. The user interacts with the Web interface to visually compose the rules and all its components. The rules are then translated into a description of an executable interaction flow between different elements and services.

Therefore, the flow of activities starts with the ideation phase based on the SNaP game and conducted by the end users (the children) in collaboration with a researcher acting as mediator. Once ideas are in place, the TERRA visual environment assists the end users in the creation of visual combinations of cards that can be translated into running code on real devices.

Deliverable D4.2-2: Education application prototype

This deliverable describes the design and the evaluation to two applications for education that allow the users to define trigger-action rules for designing and programming IoT-based devices.

The “Game of Internet of Things” (IoTgo) is a toolkit that follows a playful approach for supporting young designers in creating Internet of Things (IoT) smart things. The deliverable describes the main components of the toolkit and the dynamics that guide the users in the design of smart things. It also reports on an study conducted with a class of pupils at the high-school, who used IoTgo to design smart things facilitating inclusion in their school.

A tool supporting the composition and debugging of trigger-action rules for customizing a tangible IoT educational device. The main results of an exploratory study with a group of math teachers are also reported. The study aimed to investigate strategies for composing and debugging trigger-action rules for customizing the tangible IoT educational device.

The two applications trigger reflection at two different levels within the education domain. IoTgo aims to empower the pupils to design and prototype smart things by themselves. Besides having a value as an EUD toolkit, its proposition relates to education strategies for teaching computational thinking and IoT.

The second application instead aims to investigate the mental models of teachers when they have to program IoT devices for teaching. The study deepens insights on how to empower teachers to introduce and configure IoT devices in their teaching activities.

The deliverable is structured as follows. Section 2 describe the IoTgo toolkit and reports of a workshop that was conducted in November 2021 and that involved 24 17–18 years-old pupils of the penultimate year of a technical high-school in Gallarate. The section also highlights insights about the inclusivity potential of the toolkit, which we were able to derive by observing a pupil with special needs using IoTgo to program smart objects. Section 3 reports on a study with math teachers who were required to define trigger-action rule to program an IoT device for education. The section describes the tangible educational tool and the interface for its customization. It also reports on the experimental protocol and the results collected.

Deliverable D4.3-1: Cultural heritage application prototype

This deliverable reports the design and the implementation of an IoT-based game called The pharaoh's treasure. According to the aim of the project, we decided to create an IoT application that stimulates people, in particular children, in visiting museums more engagingly and effectively by interacting with ancient artifacts augmented through IoT technology. The game is like a treasure hunt where players are divided into teams of 3/4 children that have to find the name of a pharaoh, which is hidden inside a (smart) treasure box that also contains the final prize. To open the box, each team has to collect the (smart) key, which is composed of five characters. Each character is in the form of an Egyptian hieroglyph. To collect the five symbols, the team has to find different items located in the museum and, then, correctly answer a question on them. After finding the item, the team is asked to answer a question on it and, if it is right, a symbol of the key will be given to them. If the answer is wrong, they will lose a certain number of points and they will not receive a symbol but they are asked to find another item to continue the game. The first team able to collect all the characters of the key can open the box to find the name of the pharaoh and win the game.

The design phase started by defining all the application goals, requirements, scenarios, and personas. After framing and detailed the idea during the design phase, we designed all the smart devices by using the Autodesk Tinkercad web platform, which allowed the creation of smart objects and their code in a virtual environment before implementing them by using real Arduino boards. This report illustrates the details of all the objects involved in the serious game.

The pharaoh's treasure game can be also customized by domain experts (e.g., professional guides or museum curators) that are not necessarily IT experts, in order to adapt the game mechanisms to specific museums, users, and needs. This customization activity is performed by creating Event-Condition-Actions rules through tools like EFESTO-5W (see Deliverable D1.1-1).

Deliverable D4.3-2: Cultural heritage application prototype

This deliverable reports the details of the design and the implementation of two IoT-based games for museums called The pharaoh's treasure and The ‘Magic Wand’, respectively. All the technical details about the IoT technology adopted to build the smart devices are described. An experimental study has been also planned to evaluate The pharaoh's treasure game, to understand how the use of IoT devices affects the visitors' UX; the design of the study is reported in this document.

Deliverable D5.1-1: Methods and Techniques for acceptance and UX in IoT development

Internet of Things poses new challenges not only from a technological perspective (e.g., interoperability, power consumption, security, etc.) but also from a methodological point of view, in particular for the design and evaluation of the interaction with IoT applications. While traditional HCI solutions mainly fit 'traditional' virtual and digital systems, the possibilities offered by IoT technology to physically interact with smart objects introduce novel aspects that must be taken into account when designing the interaction with them. Even tools for End-User Development of IoT systems are influenced by these aspects, thus they require proper solutions for their design and evaluation.

This document reports on methods and techniques used to evaluate the usability and User eXperience of tailoring environments for the Internet of Things applications. In particular, it reports the details of real examples of evaluation techniques, showing how the study has been designing, the participant recruitment, the data collection and analysis. This deliverable also introduces a UX quality model that assists in the design and development of IoT environments. It is the result of a literature review on the evaluation of smart environments that allowed us to model a framework for the design and evaluation of smart environments. Lastly, this deliverable finally describes a pilot study focusing on how mental models influence user performance in the comprehension of trigger-action rules.

The document is organized as follows:

Section 1 reports a literature review on the most important methodologies adopted to evaluate EUD tools for IoT, for example, controlled experiments for evaluating usability, UX and creativity, utilization studies, and focus groups;
Section 2 discusses a UX quality model that drives the design and evaluation of IoT environments;
Section 3 reports how mental models influence user performance when interacting with tools for EUD of IoT; it also describes a pilot study providing some evidence of the fact that an effective design based on efficient mental models can improve the comprehension of trigger-action rules.

Deliverable D5.1-2: Methods and Techniques for acceptance and UX in IoT development

This deliverable extends and enriches what already reported in Deliverable 5.1-1 (Methods and Techniques for acceptance and UX in IoT development) illustrating methods and techniques recently adopted to assess the usability and User eXperience of tailoring environments for Internet of Things applications. It reports the details of real cases of evaluation techniques, showing how the study has been designed, the participant recruitments, the data collection, and analysis. In particular, the reported studies are on: evaluation of securing mechanisms in tailoring environments; an authoring tool that supports the definition of event–state-condition-action (ESCA) rules: a tangible tool for therapeutic interventions; comparison of different recommendation solutions for tailoring environments; comparison of alternative composition paradigms for tailoring environments.