The SpokenMedia project and NPTEL are in discussions to bring the automated lecture transcription process under development at MIT to NPTEL to:

• Radically reduce transcription and captioning time (from 26 hours to as little as 2 hours).
• Improve initial transcription accuracy via a research and development program.
• Improve search and discovery of lecture video via transcripts.
• Improve accessibility of video lectures for the diverse background of learners in India, and worldwide, via captioned video.

NPTEL, the National Programme on Technology Enhanced Learning, is a program funded by the Indian Ministry for Human Resource Development and a collaboration of a number of participating Indian Institutes of Technology. As part of Phase I, they have published approximately 4,500 hours of lecture videos in engineering courses that comply with the model curriculum suggested by All India Council for Technical Education. In an even more ambitious Phase II, they plan to add approximately 40,000 additional hours of lecture video for science and engineering courses.

The current NPTEL transcription and captioning process is labor and time intensive. During our discussions, we learned that it takes approximately 26 hours on aveage to transcribe and caption a single hour of video. Even with the initial hand transcription, they are averaging 50% accuracy.

Source: Brandon Muramatsu

NPTEL Current Transcription and Captioning Process

We discussed using the untrained SpokenMedia software to improve the efficiency of this initial process. Our initial experiments suggest that the untrained recognizer can achieve 40-60% accuracy, which is in the same range as the current NPTEL hand process. Thus we propose replacing the hand transcription and captioning steps with a two step recognition and editing process. Using a single processor, the recognition step takes on the order of 1.5 hours per 1 hour of video. Using this estimate coupled with the existing editing time in use at NPTEL, the overall process might be reduced from 26 hours to approximately 10 hours.

Source: Brandon Muramatsu

Initial Improved NPTEL Transcription and Captioning Process

And we discussed initiating an applied research and development project to create baseline acoustic (speaker) models for Indian English for use in the automated lecture transcription process, with the goal of improving the automated transcription accuracy to the same range of American English transcription (as high as 80-85% accuracy). The use of improved acoustic models and parallizing the recognition process might reduce the total transcription time to approximately 2 hours.

Source: Brandon Muramatsu

Goal NPTEL Transcription and Captioning Process

Along with Peter Wilkins, we developed a demonstration of SpokenMedia technology using automatic lecture transcription to transcribe videos from IIHS. We developed a new JavaScript player that allowed us to view and search transcripts, and that supports transcripts in multiple languages. View the demo.

Where does it fit in? The system is envisioned as a service that can be integrated directly into individual campus podcasting solutions; the architecture of the system will be flexible enough to integrate with existing workflows associated with lecture recording systems, learning management systems and repositories. The service is intended to plug-in to the processing stage of a simplified podcast workflow as illustrated below. A goal of this project is to integrate the system as part of a Podcast Producer-based workflow using an underlying Xgrid to perform the automated speech recognition processing.

Source: Brandon Muramatsu

(Simplified) Workflow with Transcript Creation

How does it work? The process for creating media-linked transcripts, as illustrated below, takes as inputs the lecture media, a domain model containing words likely to be used in the lecture, and a speaker model selected to most closely match the speaker(s) in the lecture. The output from the speech processor is an XML file containing the words spoken and their time codes. The time-coded transcripts and lecture media are brought back together and are viewable through a rich media browser.

Source: Brandon Muramatsu

SpokenMedia Workflow

Key steps in the workflow include the use of a collection of texts, such as lecture notes, slide presentations, reference papers and articles, that are processed to extract all key words and phrases to form a domain model. The creation of individual speaker models (such as from faculty teaching term-long courses) increases the accuracy of the recognizer and therefore transcription accuracy. The speech processing is multithreaded to process audio file segments in parallel; similar to video processing in existing Podcast Producer workflows, the speech processing is suited for an Xgrid or cloud-based (e.g., Amazon EC2) deployment.

In the uStream video below, the SpokenMedia presentation starts at about 19:30 in. The first part of the presentation is Mara Hancock from UC Berkeley talking about Opencast Matterhorn. (Unfortunately they forgot to start saving the stream at the start of her talk.)

The presentation to the IEEE-CS Bangalore Section was also the best presentation of the three–this presentation really wants to be an hour long, and we got great questions from the audience. Unfortunately I forgot to record the presentation, it would have made a great slidecast.

Embedded below is the presentation to the Technology for Education 2009 Conference, the one with a slidecast.