For Real Feedback on Multimedia Elements

InContext currently teaches Contextual Design in several ways; one of the primary methods is an intensive three-day classroom course. To make training available to more people, we’ve begun to design an online version of the course. One of the obvious benefits to a live class is the passion and personality of the instructor, and one of the challenges of creating an elearning environment is to create a valuable connection between student and instructor without live interaction. Online, the instructor is only present though a variety of video clips. Thus, when we reached the paper prototyping phase of Contextual Design for this project, we realized that our prototypes of the course needed to evaluate both the user interface and the content presentation — and needed to extend beyond paper. We believe strongly in the value of paper prototyping for testing designs. The challenge was in taking paper prototyping techniques and applying them to a design where video is a critical element.

Combining paper and online prototypes

One of our primary objectives in designing the e-learning environment was to ensure that students could successfully learn the main points of Contextual Inquiry within it. Therefore, we needed to test more than just the operation of the system. The complexity of testing these dual issues in a simple prototype led us to create a dual prototype. Paper prototypes are good for testing action-based systems, where the user is trying to perform a task. However, paper prototypes aren’t sufficient for testing time-based activities such as transitions, pacing, length of time, and learning that doesn’t rely on text as the channel of delivery.

Most of the e-learning environment could be tested in paper. The paper part of the prototype included the entire framework-navigation, buttons, and secondary related content. We used paper for all text pages and text areas, self-tests, index pages, graphics and diagrams, and even some learning exercises. We relied on online movies or audio clips for animations, image montages, real-world examples, stories, parts of learning exercises, and all introductions and summaries given by the instructors. We combined these two types of prototypes by taping all paper parts directly onto the screen of a laptop computer.

One aspect of paper prototypes is that they should be simple and rough; they should look unfinished to the user. This allows the user to more easily co-design and suggest changes during the prototype interviews. It also affords the interviewer the ability to make quick, in-the-moment changes to the prototype. The idea is not to test a highly finished piece, but to test the concept and structure behind the piece. After validating the concept, the content can be polished during the final production process. This aspect of “roughness” should still apply to online prototypes, so in shooting movies of the instructors, creating animations, and recording audio voiceovers, we did not try to produce highly polished, finished pieces. We did not use any professional actors, narrators, or camera people-the “stars” were all project team members and other employees.

Structuring the prototype interviews

While we were not testing the content, we were testing whether the way the course was structured and presented would enable students to learn the soft skills we aimed to teach. At the same time, we needed to ensure that it was possible for the students to co-design the e-learning environment with us, allowing them the ability and space to alter the structure if necessary. The interviews themselves thus became a hybrid of standard paper prototype interviews, video prototyping, and learning assessment.

Running the interviews

After the interviewers explained the nature of the interview, the student was shown a piece of paper representing an overview page. This overview page was both an outline of the course and a navigation page that allowed the student to choose where to go next. All the students followed the linear path through the course, which the design, and the interviewers, encouraged, though did not force. To ensure all test points were addressed, in some interviews various parts of the course were skipped at the interviewer’s discretion.

The overview page, navigation elements, bookmarks, and buttons to click for more information were taped to the laptop screen in order to give the student a sense of the overall look and feel of the learning environment. The taped pieces of paper also allowed the student to remove and rearrange elements, change them with the markers, or create new ones. The student could interact with, and re-design, the structural elements in the paper part of the prototype.

Strategies for co-designing with the student

When the student navigated to a teaching element that was in video form, the interviewer played the appropriate movie. The student controlled the mouse, so had the option of pausing, fast forwarding, rewinding, or replaying any of the movies. The interviewer and the note taker observed and recorded these interactions, and used various strategies for understanding why students reacted to the videos in particular ways.

The first strategy was similar to that used when the student interacted with the paper part of the prototype. When the student reacted to the movie, the interviewer would present a question in the moment, such as “I see you paused here-did you want more time to think about that point?” Such questions capture the action while it is happening, and present the student with a hypothesis for his or her intent, which he or she can accept or reject.

Another strategy was to let the student play through a clip at his or her own pace, with the note taker recording the actions of the student but not interrupting the flow of the learning. Once the video was finished, the interviewer would replay the clip, stopping frequently to ask questions about reactions observed the first time, such as “I noticed when you watched this the first time, you nodded your head-were you agreeing with the instructor?” The first play through allowed the interviewees to control the environment, enabling us to observe the learning process within the environment we were designing. The second play through allowed us to learn more about their experience interacting with the prototype.

A third strategy was to break a video sequence into stills, which we printed and taped to pieces of white paper. After viewing a video, the student was asked to rearrange the stills and comment on each part of the just-watched video. This process allowed the interviewee to discuss how he or she would change the design, and literally rearrange the video by moving the still images. The interviewers chose their strategies during each interview and individual video clips within the prototype by judging the context of the interaction. While a response to a reaction in the moment was most desired, interviewers balanced this against interrupting the flow of learning and interaction.

Lessons learned

Through this design and prototyping process, we learned some lessons about how to create paper prototypes and run interviews with the addition of video, and developed principles for the design of online learning environments.

Creating the prototype

There is an important balance to find when creating the prototype itself. Since a learning environment is such a complex entity, and elements such as navigation impact the learning, we felt it was important to render the interface graphically so it correctly relayed the navigation paradigm, which allowed students to see where they were in the context of the whole course. At the same time, prototypes need to be rough to allow co-design by the users and simplify iterations based on user feedback.

While using paper interfaces that looked like web pages was important for relaying the navigation structure to the students, we also found that in some versions of the prototype it was unclear to the students how they were supposed to interact with the paper designs. For instance, some did not understand that they could click on parts of the interface, since it was not obvious that a link was clickable. When the paper is designed to look like a web page for the prototype, it is important to make links look like links so people know to click.

Running the interview

The presentation of paper elements within the prototype also impacted the interviews. In order to set up the laptop computer in paper as though it were the actual learning environment, we originally taped a cardboard framework around the outside of the screen. This was the most common problem in terms of interacting with the prototype that we found with the students. When the navigation and other elements were outside the real screen, students did not see them, ignored them, or forgot they were there. While it was important to have all the elements in one place, especially with the videos running on the screen, we found that a more compact presentation, with the frame within the bounds of the laptop screen, was more comprehensible for the students.

Summary

Designing optimal online courses involves designing both the environment and the content. Our combined paper and online prototypes provided a way to test both the structure of the elearning environment and how well student could learn the content presented. These prototypes need to be modified from purely paper prototypes in order to ensure that the two media work together and each is used to test the aspects of the environment it is best suited for: paper for structure and action based tasks, online video for learning. This method can be adapted for other environments and products, testing content, environment, and multimedia.