«A Confederation of Tools for Capturing and Accessing Collaborative Activity Scott Minneman Xerox Palo Alto Research Center (PARC) 3333 Coyote Hill ...»
During each IP segment there are two kinds of activity, discussion and conclusion. Discussion activity takes place across the table and involves all the participants, who are interacting with each other. This is the most critical activity, and we want our technology to be non-intrusive. For example, we do not require that the participants focus on the LiveBoard. Ron takes notes during the discussion on the laptop; he acts as a recorder, only occasionally participating in the discussion himself. Ron's notes are not totally private, however. We found it useful to "beam" his notes from the laptop to the LiveBoard as he takes them. Participants tend not to orient towards these beamed notes, but rather monitor them "out of the corner of their eyes" to make sure their contributions are being noted by Ron.
Ron brings the discussion activity to a close and initiates the conclusion activity of the IP segment. This activity involves making a decision on how to handle the IP and noting any associated action items. This activity is different from the open discussion in that Ron stands at the LiveBoard, where he marks the rating and disposition of the IP and handwrites the action items. Although there is discussion during this part, the participants are more focused on the board and on Ron than on each other, because they all want to see and make certain they concur with the conclusions.
Later, Ron documents the discussions and conclusions reached in each meeting (this may be days or weeks afterwards). He does this in the access setting, which is simply an office with a Sun workstation and audio/video playback devices, pictured in Figure 2. We call this particular configuration the "Salvage Station."
Figure 2. An access setting.
The Salvage Station consists of a workstation, monitor, and speaker. The workstation contains the LiveBoard display, playback controls, and an editor for creating documentation.
Creating the documentation in this setting involves a careful review of the captured materials. Ron summarizes each IP discussion in about a page of text. His typed notes from the meeting are often cryptic, serving more as indices into the recordings than as substantive summaries in themselves.(6) The Salvage Station, shown as a screen shot in Figure 3, provides him with playback controls, a Tivoli application showing the same pages as were created on the LiveBoard in the meeting, and a text editor to create the documentation. Every mark or note that was made on or beamed to the LiveBoard serves as a time index into the recordings. Thus, in order to get access to the recorded materials for a given IP, he buttons the Tivoli application to display the page containing the notes for that IP. Then by simply touching a mark or note on the displayed page he causes the media to play from the time when that mark or note was made in the meeting. This gives him meaningful random access into the recorded material. He listens and relistens to some portions of the recordings until he understands the significance of what was said. Although he may alternate between listening and typing the summary documentation, he often does both simultaneously, listening only for important points that he may have missed the first time around.
Figure 3. Salvage Station screen showing Tivoli, a text editor (buffers of meeting notes and summary), and a simple timeline controller.
Description of the Application Domain and the Work Settings Results of Using the Tools in This Domain Our work in supporting this application domain is ongoing, and we are continuing to develop new tools. However, even at the stage of development described here, we have several qualitative indications that the tools were successful.
First, we were able to support the meetings without disrupting them or forcing changes in practices. Second, the meeting participants commented favorably when they saw that the documentation was qualitatively improved by Ron's having access to the captured materials. Third, the meetings proceed more freely, because the participants seem to have greater confidence that their contributions will not be lost. Fourth, Ron is pleased to have the capability to produce better documentation. The trade-off is that it takes him longer to produce this more thorough documentation; but this situation should improve as we refine our access applications.
Description of the Application Domain and the Work Settings Table of Contents The Coral Architecture Coral is not a close knit system, but rather consists of a loosely coupled confederation of tools that work together through protocols of communication. We decided early on that the most practical way to evolve the system was to construct a set of applications that share access to a distributed multimedia infrastructure and that export interfaces to each other to permit various kinds of cooperative action. For example, a text editor that participates in this milieu might, by virtue of its connections to WhereWereWe and Tivoli, be making annotated events in a database that will later serve as pointers back into a multimedia recording, driving the current page of the LiveBoard display when the text editor is flipped from page to page, and "beaming" segments of the editor's text onto the screen of the electronic whiteboard for shared discussion or review.
Each of the tools in the Coral confederation is fully functional without one or more of the others (as we know from various weeks where we were still sorting out bugs in the individual pieces), but the combination of the tools makes for a more powerful union. Thus, the "system design" is emergent, based on a developing a shared infrastructure and developing protocols for exporting functionality to neighboring tools. Toward this end, major efforts were focused on the design of a suitable application programmer's interface (API) to the WhereWereWe multimedia system resources and similar APIs to other tools (e.g., Tivoli's beaming functionality), which are exported with a distributed object protocol.
Inter-Language Unification (ILU) and WhereWereWe
The tools described here, and WhereWereWe in particular, rely heavily on the Inter-Language Unification (ILU) project for their implementation [Janssen, 1994]. ILU is a distributed object system that lets users easily build objects which exist in one address space on a particular machine but have "proxy" objects that exist in the same or different address spaces on the same or other machines in a network. Either the real or proxy objects may be implemented in any of the following languages: C++, C, Python, Common Lisp, Fortran, tcl,(7) or Modula-3. The tools under discussion currently make use of only the C++, Python, and Modula-3 language bindings.
The proxy objects are operated on by "client" programs which can assume these proxies to be performing their respective functions, although in reality the proxy makes a remote procedure call to the real object (the "server" object), which actually carries out the operation. For example, a WhereWereWe client may create an ILU object from the API (e.g., a video Player object), and not concern itself with the details of the fact that in reality this object is communicating with the WhereWereWe server to carry out its functions. Furthermore, the object may be transparently shared--WhereWereWe allows multiple proxies to represent the same server object to facilitate resource sharing. For instance, multiple users may share a single video Recorder object, each potentially having control over the state of the device (e.g., pause and resume, frame rate), thus reducing hardware resource demands for compression and storage.
Most of these extensions are transparent to typical ILU API users, and the simpler nomenclature of clients and servers will be retained throughout this paper.
The Coral Architecture WhereWereWe Model The application programmer can see seven basic abstractions in the WhereWereWe API. These are all objects, which belong to the following classes: Session, Stream, Event, Player, Recorder, Data, and Notifier. Most applications need more than one of these classes, but few (if any) need all of them. The classes can be broken into three groups.
Sessions, Streams, and Events are classes of objects that are used to do naming in WhereWereWe. Sessions are named collections of Streams, which correspond to semantic occasions, such as "Project meeting from October 15". Streams are media data that can be played back, such as audio, video, or a program activity log. Events are "occurrences" that happen at some point or interval in a Stream. This association with a Stream is purely for the convenience of retrieval, but is one natural way of thinking about the relationship between Events and Streams. Also, each of these three classes supports a property list on each instance of the class, so that application programmers may associate arbitrary application-specific data with each object.
Players, Recorders, and Data objects are used to convert Streams into other forms. A Recorder both creates a new Stream object and takes responsibility for storing the data associated with that Stream (often this is simply a disk file which can be replayed later). A Player displays for the user the data of a previously recorded Stream. A Data object converts a Stream's recorded data into a raw form that a processing application can use as input for its algorithms.
A Notifier object is used by client applications that need to stay informed of the status of ongoing playback or recording activities.
It should be noted here that WhereWereWe can be thought of as "glue" that allows index-making and browsing activity for stream data. It does not attempt to provide general media playback services, but rather provides an infrastructure into which such services can be inserted and utilized in a uniform way. WhereWereWe, at present, has built-in drivers for digital audio and video in one format.(8) WhereWereWe also provides a limited mechanism for additional drivers to be installed and used with no changes to the client software and minimal changes to the server software.
These WhereWereWe API elements may be combined in many useful configurations.
The Coral Architecture Table of Contents Description of Two Primary Tools The initial implementation of the WhereWereWe infrastructure was completed in September 1993. Starting about midway through the development, and still continuing, a number of applications have been constructed or modified to take advantage of the facilities it offers. To explain what makes the WhereWereWe API useful to application writers, we will discuss its use with a pen-based application (Tivoli) for indexing from casual notetaking activity, and an Emacs mode we wrote to gain experience with textual annotation.(9) Tivoli Tivoli is the large-scale, pen-based electronic whiteboard application running on the LiveBoard to support publiclyviewed image manipulation in meetings.(10) While its many electronic whiteboard features have been documented elsewhere [Pedersen et al., 1993; Moran et al., 1995], a number of extensions facilitate seamless marking of activity for later retrieval, enable easy replay of multimedia records, and provide participants with a sense of the relation between the notetaking and recording functionality. Figure 4 shows a typical Tivoli screen layout in our application domain;
this one is for use in rating invention proposals.
Figure 4. Typical Tivoli screen shot, showing an application-specific form, pen-drawn strokes, keyboard text, and clock objects.
Tivoli has within it a stand-alone history mechanism that allows an "infinite undo" of drawing and editing operations;
this history facility also gave us a leg up on allowing the drawing/editing process to be replayed. Late in 1993, Tivoli was extended to use the WhereWereWe API and become a marking and browsing application. Very little needed to be done to extend Tivoli to support indexing, as its history was already retaining timestamps of drawing and editing operations. The application was modified to write that information into the files that it retained about sessions where audio and/or video were recorded. Tivoli was eventually further modified to produce other timing indices, but considerable utility as a side-effect indexer accrued from simply tying into its existing history mechanism.
In addition to the indexing functionality outlined above, the Tivoli application can be used to drive the various WhereWereWe resources for playback. Since each stroke drawn on the LiveBoard is timestamped, it is possible to select a stroke and have WhereWereWe and Tivoli replay all of the recordings made at that time. Thus, the user can utilize a Tivoli page's display as an interface, answering questions such as "what was Joe saying when I jotted this down?" or "what's this all about?"(11) Thus the strokes themselves constitute an important index into the activity.
The user is also presented with a simple timeline interface to the playback functions. This timeline panel offers the user several kinds of control that are not available with the direct selection interface described above. For instance, gross controls such as going to a general portion of the recording (say, "2/3 of the way in") or starting and stopping the session playback, but also finer-grained control such as hopping forward or backward several seconds to catch an unintelligible utterance.
Oftentimes, much of the graphical activity that constitutes the indices of side-effect markers comes after the event-making notes about a point that was made, or sketching a suggested solution. We added a feature that blurs the boundary between Tivoli as a side-effect indexer and as an intentional indexer. Users can insert a sort of temporal bookmark, a graphical object whose primary purpose is to initiate later replay from its creation time. These graphical objects display as a clock face (showing the time that they were created, often allowing users to see the progression of a discussion) and often further serve as bullets in list items (Fig. 4). These clockmarks have become very popular graphical elements, strewn throughout the pages of Tivoli.