The Intelligent Classroom
Jabberwocky: The Classroom is listening!
When the speaker is lecturing from a Microsoft PowerPoint slide presentation, the Intelligent Classroom assists the speaker using the Jabberwocky system. Using the basic functionality of the Classroom, a speaker can control the slide presentation by just saying what he wants ("Next slide, please"), making a gesture to go on or go back, or touching the Next button that is drawn on the slide. But Jabberwocky actually looks at the contents of the slides, so the speaker can lecture from the slides, as Jabberwocky follows along, changing the slides at the appropriate moments.
The basic idea is that Jabberwocky uses a shallow (syntactic) understanding to match the speaker's words to important phrases in the slides. The slide on the right shows the key phrases that Jabberwocky found through syntactic analysis (underlined). As the speaker talks, Jabberwocky listens for these phrases (and a number of simple grammatical variants) to keep track of which slide the speaker is discussing and where he is in the slide. When the speaker starts speaking phrases from the next slide, Jabberwocky knows to display the next slide. If the speaker skips around in the slides (to answer a question or to otherwise deviate from the planned slide order), Jabberwocky will wait until it is certain which slide he wants to skip to before switching slides.
Fortunately, this purely syntactic understanding appears sufficient for knowing when to switch slides. This isn't entirely unexpected: you can expect a human A/V assistant to match what the speaker is saying to the contents of the slides without any technical knowledge of the presentation's subject matter. An additional benefit of using a shallow understanding is that Jabberwocky can better deal with the inevitable speech-recognition errors. Even when Jabberwocky mishears half the speaker's words (not unreasonable with a conversational speaker), Jabberwocky will still hear enough of the important phrases to advance the slide appropriately.
Jabberwocky has been made into a stand-alone application which can run on a laptop computer. Jabberwocky gets its name from the Lewis Carroll nonsense poem from "Through the Looking Glass" in which, even though most of the words are made up, the reader can still get the gist of what's happening. As Alice said after reading the poem, "Somehow it seems to fill my head with ideas--only I don't exactly know what they are! However, somebody killed something: that's clear at any rate."
Maintained by franklin@cs.northwestern.edu
Last update: November 16th, 19101