edX Usability Study
May 2017
Note: edX has given permission to include this study in my portfolio
Study Process
The challenge
EdX is a remote education program with a mission to deliver high quality learning to "over 1 billion people worldwide." For three months, I worked on a team of six to conduct discovery and exploratory research on the edX educational platform.
Study Goal:
Identify and understand first-time user pain points and reasons for drop-off to make the edX onboarding process for new learners simple and painless; ensure edX clearly communicates the value of the platform.
Shortly after being assigned the project, our team created team expectations, schedules to accommodate collaboration across different time zones and work schedules, and delegated responsibilities.
Kickoff
I was the designated point of contact between the Bentley team and edX. I visited the edX headquarters in person and arranged a remote kickoff meeting over google hangouts, for members of the Bentley team to discuss the project with edX, ask questions, clarify the goals and objectives of the study, and set expectations for the study process.
Expert Reviews
Prior to testing our product with users, the team developed a comprehensive expert review of the edX website, using Nielsen's heuristics and severity ratings of low, medium, and high priority. The expert review allowed our team to identify the areas that might be relevant to a new user experience.
However, we ended up revealing more than 80 problems.
Spreadsheet we used to merge our expert review findings. This sheet enabled us to consolidate findings, assign severity ratings, and compare notes on problems.
It was necessary to refine, group, and prioritize these problems into categories of interest, as we could not test all of these problem areas, nor were all of these issues relevant to first time users.
Based on the expert reviews, we focused our study plan on several categories:
-
First Impressions
-
Search and Browsing Courses
-
Registering and Viewing Dashboard
-
Beginning a Course
-
Terminology
Recruiting
In an effort to refine our study, I often consulted with stakeholders, my professor, Robert Virzi and the Director of UX at edX, Alyssa Boehm. Initially, our team was overwhelmed by the broad target audience of over 1 billion potential new users. However, by communicating with the stakeholders, our team refined our recruit to fulfill specific demographics, goals, and behaviors- that of a working professional interested in advancing their stills.
We adjusted our recruit due to constraints, such as language, location, and available participants (friends and family recruit). Our team conducted the test in English, acknowledging that the study does not include the perspective of all global learners, who may not speak English. I worked with edX to arrange honorariums (Amazon gift cards) which helped in our effort to recruit participants from outside-the-city locations.
Recruit
-
12 participants who fit
the “Advancing Professional” persona
Mod Guide
My team co-wrote a mod guide for a 75 minute moderated sessions. Our group switched off between roles of moderator and notetaker (in a pre-made notes grid).
We began each session with a standard introduction to ensure participant comfort and consent.
Our tasks
-
First Impressions Task: Participants viewed the homepage without clicking
-
Search and Browse Task: Participants were provided a scenario and asked to find a specific course.
-
Register and View Dashboard Task: Participants were asked to complete the registration for a course.
-
Begin a Course: Participants began a course.
-
Terminology: In a retro, participants were brought back to the edX pages and were probed on specific information (areas or sections of the page that included differing terminology)
Test Day
We ran the usability test with 11 participants (and one dismissed participant). We began each session with a standard introduction to ensure participant comfort and consent and recorded the sessions using UX laboratory equipment (cameras, mic, etc).
Our group switched off between note-taking and moderating responsibilities. We used a notes-grid to collect study data.
Results
I was a project manager for the final project. The team split up the analysis work task by task. We presented our findings to edX, who positively received our results. I followed up with edX stakeholders a year post-test and was told that they took our suggestions were considered when developing new experimental designs.