«To cite this version: Zo´ Drey, Charles Consel. Taxonomy-Driven Prototyping of Home Automation Applicae tions : a Novice-Programmer Visual Language ...»
8.3. Study Sessions Each participant came to our lab for a 60-minute session. Participants were ﬁrst presented a 15-minutes tutorial (included in the session). The tutorial started with an introduction to the domain of home automation. Then, we demonstrated the usage of the Pantagruel editor through an example that covered the language concepts.
Finally, sessions were conducted individually, using the Pantagruel editor parameterized by a concrete environment. This environment was composed of various entities: lights (i.e., deﬁned with a Light entity class), person tags (i.e., PersonLocalizer), light sensors (i.e., LightSensor), and radios (i.e., Radio).
Each participant was asked to create 6 rules with varying complexity for orchestrating these entities. Speciﬁcally, they were asked to deﬁne each rule without guidance but by carefully following the development process as described in Section 6. We classiﬁed these rules in 2 sets shown in Tables 3 and 4. The ﬁrst set of rules aims to assess the visual paradigm of Pantagruel, and necessitates to reason about speciﬁc entities. The second set of rules aims to evaluate Pantagruel abstractions, and necessitates to use entity classes, thus introducing a level of abstraction. The R4 rule generalizes the R3 rule; it requires to (1) create a ﬁltering sensor on an entity class (i.e., Light), (2) express a dependency between 2 classes of entities (i.e., PersonLocalizer and Light, combined with a condition on their location attribute). The R5 rule requires to deﬁne an action whose parameter is an attribute of an entity class (i.e., the preferredmusic attribute of PersonLocalizer). Finally, the R6 rule requires to express a condition that has to match on all entities of a given class (i.e., PersonLocalizer).
9. User Study Results
Let us now expose the opinions of participants, gathered by our post-questionnaire. We will then analyze the result of the study sessions with regard to the task performance of the ﬁrst set of participants. Finally, we report the lessons learned from the second set of participants.
9.1. Subjective results To evaluate the overall opinion of the Pantagruel editor, we used the System Usability Scale questionnaire . It is a ten-statement questionnaire, each having a ﬁve-point scale ranging from strongly disagree to strongly agree. The questionnaire score ranges from 0 to 100. The global score of our questionnaire, including the 18 students, is 70 (Standard Deviation=11.8, min=47.5, max=90.0). This rate ranks the usability of Pantagruel as acceptable according to Bangor et al. [26, 27]. The separate scores were 68.5 and 73.8 for the ﬁrst and the second set of students, respectively. The latter score illustrates the eﬃciency of the improved version, which preserved the visual concepts of the language.
Speciﬁcally, 15 participants reported that they would like to use frequently this tool to prototype other home automation applications (agree or strongly agree). Though nobody disagreed with the statement “I thought the system was easy to use”, 7 of them were undecided about this statement (don’t know answer). 1 participant did not “feel very conﬁdent using the system”, against 12 of them who either agreed or strongly agreed with this statement. To sum up, opinion for each statement was diﬀerent than don’t know, except for the statements (1) “I think I would need the support of a technical person to be able to use this system” and (2) “I imagine that most people would learn to use this system very quickly”, for which as many people agreed and disagreed.
The questionnaire was completed with open remarks. It is worth noting that a participant found it “fun and easy to program a house”. However, 2 of the participants reported that they needed a 2 or 3 hour-training before feeling conﬁdent with the Pantagruel editor.
9.2. First phase: analysis of task performance We now report our analysis on the ﬁrst phase of our study, of the evaluation of the visual paradigm with the ﬁrst set of rules (Table 3), and of the language abstractions with the second set of rules (Table 4).
9.2.1. Intuitiveness of the visual paradigm Successes. For the ﬁrst rule, we reminded individually each participant through a two-minute tutorial how to use the user interface (using diﬀerent entities than the ones required for the rule).
All of them then naturally applied the rule creation process associated with the visual paradigm;
it consisted of selecting and placing the relevant elements from the editor palette to the editor view. As an illustration, 10 of the 12 deﬁned the R2 rule in 3 to 6 minutes; 7 of them deﬁned the R3 rule without an intervention in 3 to 8 minutes.
Drawbacks. For the ﬁrst rule (R1), 8 of the 12 participants required help with the user interface to position the appropriate graphical elements on the editor view. For example, one participant attempted to directly connect the left part of a section with an actuator using the wire element.
9.2.2. Accessibility of Pantagruel abstractions Successes. The R3 rule (Table 3) involved, but did not require, the use of entity classes. 4 participants naturally used a class to deﬁne it (with the sensor location = KITCHEN on the Light section). The R4 rule (Table 4) is similar to the R10 rule of Figure 8. For this rule, half of the participants needed to be reminded of the notion of dependency between classes; such a dependency is depicted by both sensors of the R10 rule, related to each other by the AND controller and the use of the MotionSensor class. Then, 8 participants were able to create the R4 rule, and 10 for the R5 rule, without any help.
Drawbacks. As illustrated by the Tables 3 and 4, the second set of rules required more time than the ﬁrst one. A participant did not feel conﬁdent with the ﬁltering eﬀect of conditions on the Light entity class of the R4 rule: although he intuitively created a correct rule, he was not convinced of its meaning while reading it. Finally, all participants required the experimenter to explain how the NONE quantiﬁer, required for the R6 rule, changes the meaning of its sensors. At the time of the study, the quantiﬁer was placed on the sensor part.
9.3. Second phase: lessons learned and improvements From our ﬁrst study, we collected the feedback of the participants. This feedback helped us to learn lessons regarding the improvement of the Pantagruel tool. We now report these lessons, as well as the results of a preliminary improvement, from which the second set of students beneﬁted.
Lessons learned. Pantagruel visual paradigm seems easy to grasp, although its entity-centric structure is not as natural as we expected. Its intuitiveness may be increased with a stronger connection between the 2D-model of the environment and the Pantagruel editor to emphasize its entity-centric paradigm. For example, 2 of the participants at ﬁrst wanted to drag-and-drop the entities to orchestrate from the 2D-model to the Pantagruel editor. This seems an interesting software improvement of our tool. Such an improvement would enable the user to create orchestration rules through direct manipulation  of the entities via the 2D-model, instead of manually creating sections in the Pantagruel editor.
Preliminary improvement. Pantagruel abstraction concepts require further training or user-interface support. This observation motivated us to improve the Pantagruel tool with a rough naturallanguage translation of rules that pops-up when double-clicking on a controller element. This translation exposes the ﬁltering eﬀect of the rule conditions and the dependencies between classes by combining the words these, any or all. Such a translation is illustrated in Figure 12. As a result, 5 of the 6 participants who beneﬁted from this assistance completed the rules R1 to R5 without any help, 2 minutes faster than the others. The diﬃculties encountered on the ALL/NONE quantiﬁer leaded us to represent it as shown in this paper, making explicit its application to a class. This improvement was approved by the test observers. However, further evaluation is necessary for validation.
9.3.1. Summary The overall observations and the acceptable subjective results show that the Pantagruel tool is accessible to novice programmers. However, the time needed by the participants to create rules showed that the Pantagruel abstractions are not intuitive enough: compared to iCAP , which also has a rule-based paradigm, participants needed twice as much time to resolve similar rules (e.g., R4). Pantagruel and iCAP are further compared in Section 10, and some usability improvements are mentioned in Section 11.
10. Related Work Our contributions can be measured with respect to four dimensions for developing orchestration applications. Let us examine existing approaches sharing these dimensions.
10.1. Taxonomy-driven approaches Taxonomy-inspired approaches [29, 30, 31, 32, 33, 16, 17] have been used to address a range of areas, such as pervasive computing, web services, graphical computing, and robot control. For example, Olympus  is a programming model developed over a pervasive computing middleware, enabling to deﬁne an ontology of services to be orchestrated. However, Olympus does not provide an interaction model to specify the context information exposed by these services.
In web services, WSDL  is a language for describing the interaction interface of web services. It declares the functionalities and the data provided by a service. However, it does not provide any information about the data beyond its type. In contrast, Pantagruel entity classes expose the nature of the context information captured by their attributes, whether external, applicative or constant. This strategy helps the developer understand entities and their context of usage.
In the ﬁeld of robot control, Prograph CPX  enables to deﬁne a hierarchy of object classes that compose a robot. Similarly, AgentSheets [17, 9] is a toolkit for deﬁning domain-oriented environments composed of agents, interacting with each other, or with a user. AgentSheets enables to model a domain by deﬁning agent classes, and tasks deﬁning the behavior of agents.
However, neither Prograph CPX nor AgentSheets provides any information about the nature of the data that can be manipulated in a program or the eﬀect of a method invocation. In contrast, the entity interaction interfaces of Pantagruel oﬀer useful information that guide the development and the veriﬁcation of applications.
10.2. Visual rule-based orchestration languages The rule-based paradigm is widely spread in visual programming ﬁelds dedicated to robot control (e.g., to orchestrate the sensors and eﬀectors of a robot) [8, 9, 34] or game, agent-based programming (e.g., to orchestrate agents or moving objects) [35, 17, 19].
To avoid rule explosion, AgentSheets  proposes a mechanism to easily specify analogies  between the agents, resulting in the description of concise rules for a range of agents.
In contrast, Pantagruel leverages the class abstraction, enabling rules to be deﬁned on a range of entities, instead of speciﬁc entities.
KidSim  is another visual rule-based language that enables children to create games by deﬁning interactions rules between agents and/or their properties, through the use of pictures that represent the agents. However, KidSim and AgentSheets do not oﬀer a visual structuring of rules centered on the entities. In contrast, our three-panel, section-based representation can be naturally combined with selection mechanisms to visualize subsets of rules according to various criteria: entities, sensors, or actuators.
Our paradigm extends the rule-based paradigm of the Blender Game Engine . While Blender is prototype-based, the Pantagruel orchestration language relies on classes. Our approach allows programs to be reusable over a range of concrete environments.
Ladder Diagrams is a rule-based language  to program logic controllers. It is limited to manipulate low-level data, making large programs hard to read or analyze . In contrast, our approach enables the user to manipulate high-level information captured by entities, thus guiding the development of orchestration rules.
Other paradigms have been proposed to develop applications for orchestrating objects. Examples include the puzzle paradigm, used by Scratch , and the storytelling paradigm, used by Alice . However, neither Scratch nor Alice provide domain-speciﬁc abstractions to represent the sensing or actuating nature of objects. This situation makes it diﬃcult to examine the context information used for a given program. In contrast, the visual structure of Pantagruel facilitates the reasoning about the context and actions used in a program.
10.3. Visual paradigms for programming home automation applications There exist visual approaches to develop home automation applications. These approaches have been shown to provide an intuitive representation for the orchestration logic, bridging the gap between end-user requirements and a program. However, to the best of our knowledge, none of the tools reported in the home automation literature proposes an open-ended approach to developing applications, enabling an extensible set of entities to be orchestrated.
For example, CAMP  is a rule-based tool which uses the magnetic poetry metaphor to deﬁne sentences by visual word composition. However, its vocabulary is restricted to a speciﬁc area of home automation. Another related tool is iCAP , it provides a ﬁxed set of generic entity classes: objects, activities, time, locations and people. Its elements are visual representations of these classes. Composition of conditions is achieved by visual arrangement of rectangles on the screen. However, iCAP does not provide a uniform approach to expressing rules over a group of entities besides a group of persons. In contrast, our approach provides syntactic support to deﬁne ﬁlters over all instances of an entity class, as well as speciﬁc entities, enabling generic orchestration rules to be described.