«A Confederation of Tools for Capturing and Accessing Collaborative Activity Scott Minneman Xerox Palo Alto Research Center (PARC) 3333 Coyote Hill ...»
Description of Two Primary Tools WEmacs GNU Emacs is a popular text editor [Stallman, 1993], which we have extended to interact with WhereWereWe. Over the course of our project, we have used two methods for interfacing it to the infrastructure and to other tools. First, the internal Emacs Lisp interpreter was extended to allow calls to ILU objects, thus permitting Emacs Lisp to create the necessary WhereWereWe objects directly. Later, in order to be compatible with more versions of the editor, we moved the mechanism for ILU connection into a Python subprocess that sits beneath the editor to communicate with WhereWereWe and Tivoli.
A simple interface, especially designed for portable computers, has been built for notetaking with an Emacs connected to WhereWereWe (this has been dubbed WEmacs). In this interface, the user can "make a note" about something in the current Session(12) by typing a particular keystroke (currently Tab). WEmacs generates an Event whose start time is the current time and whose duration is zero; this Event is now in the WhereWereWe database for later use by browsers. WEmacs currently represents this Event as a distinguished line of characters (a "timestripe") in the buffer.
Whenever an additional timestamp is indicated, the editor submits the region between it and the previous one as an annotation (which goes onto the event's property list) on the prior Event. Submitting the Event does not preclude further changes; saving also parses the entire buffer and updates the annotations (as well as saving enough other pertinent information so that the Session can be revisited later). Playback works similarly; WEmacs' algorithm looks back for the previous Event string and begins play from the corresponding Event.
Figure 5. Emacs' WhereWereWe mode, showing the timestamping character strings and user annotations.
WEmacs also interacts directly with Tivoli, using an ILU interface exported for external control of the application.
When WEmacs is in automatic update mode, each time the user enters a tab character, in addition to submitting the previous annotation to the WhereWereWe database, the event is "beamed" to the LiveBoard, appearing on the current Tivoli page, further denoted by one of the clock objects (set to the event's initiation time from its timestripe) discussed above. If necessary, Tivoli scrolls to assure that the beamed text is visible.
The WEmacs user is also provided with other controls of the Tivoli application. Scrolling and page changing are available to the seated user; this can be very convenient for reflecting (even initiating) agenda progress in meeting settings, or when preparing to go to the board for pen-based drawing activity.
WEmacs also functions in the retrieval setting, both as a text editor for refining meeting notes, and as a method for controlling the session appliances.
Description of Two Primary Tools Table of Contents Discussion: A Broader Range of Uses Focusing on a particular domain was useful for grounding our work: It allowed us to assemble and evolve a set of really usable and useful tools. It demanded that we keep our participants satisfied. It required us to make our systems robust enough to be continuously operational. And it gave us insights into the subtleties of collaborative processes and the effects that tools (even seemingly unobtrusive ones) can have on them. But the other side of the coin is that we can become too narrowly focussed. We believe that these kinds of tools are applicable to a much broader range of collaborative situations and uses.
In our domain, only one person was taking notes; and thus all the notetaking indices reflected that one person's point of view. In many situations, such as less constrained discussions or brainstorming, it may be important to see the activity from multiple points of view. This suggests a multiplicity of various kinds of devices that give each individual participant the capability to take notes and create indices.
Another limitation that seemed reasonable in our domain was that we focused only on recording audio, under the assumption that most of the content of the activity was carried in the audio. This is not true in many situations, such as engineering design, where physical artifacts play a crucial role in the activities. Even in our domain, there are many indexical actions, such as pointing to a particular point in a document during the discussion, that we do not capture.
While the WhereWereWe infrastructure supports video capture and playback, the storage requirements were deemed prohibitive for the many hours of recording we anticipated. We look forward to improved support for video, and foresee interesting challenges in effectively utilizing video in these multi-participant settings.
In our domain, indices are created manually at a fairly coarse grain--notes are taken at one per minute at the fastest-and long periods of time can go by without any indices being created. More complete and finer indices can be produced by analyzing the audio to identify the speakers, which is also important since people often tend to orient to who said what.
It is on the access side that perhaps the broader opportunities lie. In our domain, there is a particularly demanding access task--generate a detailed summary of the content of the discussion. As we have noted, this is a costly process.
But users might access captured materials for quite different reasons. One may want to quickly "skim" a session to be reminded of what transpired (e.g., in preparation for another meeting). Another may want to find information germane to a new IP they're considering. These suggest tools geared for skimming (e.g., Arons' Speech Skimmer ) or searching.
One crucial variable that affects the nature of the access task is whether the person was present at the captured activity or not. One who was at the captured session can rely on a myriad of remembered cues (e.g., the interesting part happens right after John left the meeting). The person who was not there is "flying blind" and will place greater reliance on the captured indices. This variety has numerous user interface implications, and needs to be explored.
Thus far, textual documentation has been the dominant product of the captured activity. But there is wide variety of different kinds of multimedia documentation of collaborative activities. Creating a detailed textual summary is a demanding task, whereas creating a few pointers to the highlights of an activity might be much easier to do; and may be just as useful for many situations. The spectrum of possibilities needs to be explored.
Although these opportunities for further use are attractive, they must be approached with a sensitivity to issues of security, access, and context. The users supported by the current system speak freely in the capture setting, knowing that only Ron has later access to their comments. If a change to this situation is being considered, we must respect the privacy of our users, and make clear the range of reuse that is possible. We also believe that material of this sort could be damaging (and/or useless) if it is decontextualized, so we face further challenges as we attempt to broaden the scope of this work.
Table of Contents
Other Tools A number of other tools have been prototyped or developed by project members to fit into the Coral architecture.
Many are being evaluated for use in the invention proposal review meetings. Some fit well enough together with the other activity capture applications and the needs of the users that they will probably become part of the routine suite of tools; others may only influence design of new tools or result in modification of existing ones.
Our existing range of clients currently requires a high level of buy-in to the indexing activity; often, there are interesting events that require no further elaboration to be useful for later retrieval (or where later browsing will immediately reveal why they were made).(13) The Pedals application aims to provide an extremely simple interface--a single switch--for creating indices. Pedals uses an external microcontroller to read digital inputs (analog sensing is available, but, as yet, unused) and trigger the submission of labelled Events in the database. We've recently begun exploring applications of this very general form of marking--ranging from a simplified means to allow meeting every participant to indicate events of interest (e.g., a topic they'd like to discuss with a colleague, a new invention idea) to using switches to instrument a capture setting (e.g., detect a copier door opening).
Other ToolsSpeaker Identification
Speaker segmentation [Wilcox et al., 1994] is a method of deriving indices from a collected audio stream. Utilizing hidden Markov models, the technique can provide an indication of who was speaking at various points in the audio record of an event. This is now being integrated into a timeline representation of the activity capture records [Kimber et al., 1995]. This should prove particularly useful in settings where few intentional or side-effect indices are available, or for locating a particular contribution (e.g., John's comment on prior art).
Other Tools PARCTab Marking A laptop computer can be an obtrusive and inappropriate device in many settings--many users may be happier with jotting a quick note on a personal data assistant (PDA). As part of the shakedown of the WhereWereWe server and API, a simple marking client was written for the PARCTab [Adams et al., 1993], a networked PDA. This is a dedicated WhereWereWe application, wholly written for the purpose of marking digital video and audio. It uses a different ILU stubber, for the Modula-3 language, whereas the others use C++ and Python.
The MarkTab application uses a unistroke alphabet recognizer [Goldberg and Richardson, 1993] to allow free text entry on the Tab's touch-sensitive screen, and the Tab's simple 3-button arrangement for control. The user can think of the MarkTab application as a recipe box of 3x5 cards, one card for each event--cards that may additionally be sorted (at creation time) into categories. One of the two buttons on the Tab cycles through the categories, another signals an event and gives the user a blank card of the category that was selected when the event button was depressed. The user then uses the stylus to jot down, with unistrokes, an arbitrarily detailed textual annotation about the event in question and indicates (with a soft button) that she is done.
Other Tools Marquee Text isn't always the most intuitive or fitting means for making personal annotations. Marquee [Weber and Poon, 1994] is a pen-based video logging tool that enables users to correlate personal notes and keywords with a videotape during recording. The Marquee log consists of a scrolling note-taking area which is divided into a series of timezones.
Once a timezone is created, by drawing a line across the tablet, the user may then make notes appropriate to the time of the event. In addition, the user is provided with a facile way of categorizing the material by applying keywords to the timezone.
Marquee was built to index analog video tape; it was modified to use WhereWereWe as its indexing and streams capture and replay subsystem. When the modified Marquee starts up, it creates or joins the recorders for the streams of a Where Were We session, and then begins operation in its notetaking role. When it needs a "timestamp" (where it previously would retrieve a videotape time code), Marquee now asks Where Were We what the current time is(14) and records that instead of the SMPTE time code information. When Marquee is put in "review mode" it creates or joins a set of players and instructs them to seek to points in absolute time that it noted previously. For simplicity of the transition from analog to WhereWereWe, Marquee maintains its own ink database; the switch to events would not be difficult.
Table of Contents Discussion: Infrastructure and Tools As outlined earlier, Coral grew out of merging three existing projects--Tivoli, WhereWereWe, and ILU--in a domain where a natural opportunity presented itself. The Where-WereWe infrastructure had been designed to support the addition of new client applications, and the Tivoli program was recognized as an application whose history mechanism was well-suited for producing index marks. As it turned out, it was not difficult to hook WhereWereWe and the Tivoli code together using the ILU distributed object system; Tivoli presented no particular complexities, nor was this application a particularly demanding one from the point of view of WhereWereWe. In fact, the simplicity of that initial hookup was deceptive; keeping the system on-line for regular weekly use highlighted areas where we were depending on software that is less than 100% reliable, and numerous shortcomings were revealed over the course of the last 18 months of heavy use.
Coral pushes on numerous aspects of software engineering where the state of the world leaves something to be desired. However, we feel that we are well poised to take advantage of improvements in these related areas--distributed systems, operating systems, object-oriented databases, multimedia compression and network transport, and others.
For example, the infrastructure could benefit greatly from synchronization primitives provided by the underlying operating system. If WhereWereWe had more control over the timing of its media streams, better playback synchronization performance would result, especially for streams that need to stay synchronized for long periods of time and/or cross several pause and resume boundaries.
Another example is in the area of distributed object storage. WhereWereWe currently implements its own object persistence atop a standard relational database. This method is completely ad hoc, and the performance of our initial stab at the problem suffers from our not knowing the characteristics of our eventual use (we do a form of lazy evaluation that often ends up resulting many more database hits than would have been necessary). While we've now had enough experience that we're poised to do a better job with a second implementation, it is very clear that a system which focused its efforts on providing fast and efficient persistent object storage and retrieval would be of considerable value.