Papa John’s Pizza: In order to improve the website, I conducted and recorded usability tests to see if end users could complete a set of tasks.
Skills: Usability Testing and Moderation, Participant Recruiting and Screening
In this fictional scenario, Papa John’s wants to redesign its website to make ordering pizzas more user friendly. I conducted usability testing to see where I could fix issues as well as what was working.
First I had to determine the ideal number of participants. With the limited number of tasks, research shows the amount of new issues a person would uncover would drop off after about five participants. I proposed having five initial participants, then three to five more for a second round once the first issues were addressed, and another three to five if any significant issues came up in the second round.
After that, I created a screener for participants.
Once I had created that, I conducted a usability test with just one volunteer. The other students in my class did the same, as we had limited time and resources. This test was recorded as a video that showed both actions on the website and the participant’s face.
We shared the videos and each analyzed four others for the next part. We asked the same questions in each of them to get comparable results.
- We are going to be looking at papajohns.com web site. Even if you are not a Papa Johns fan, imagine that the people you are with are fans of Papa Johns and that is where you will be ordering the pizza. There are bunch of people at this party and you need to order 3 pizzas.
- Mushroom and Pepperoni
- Half onion and half sausage with light sauce
- A specialty pizza (you want some variety)
- Tell me a little more about your experience ordering pizza from this site. Is it what you expected?
- You want to sign up for deals and coupons but you don’t want to register. You just want to give them your email. Can you do this and what do you think you will receive by email?
- Your pizza arrives and it is terrible! Your driver was rude and you are really upset about what just happened. You call the local store but get nowhere. You need to contact the corporate office, how would you do that?
I viewed the videos of the following participants:
Participant 1-No name given
She is a middle aged woman who had ordered online before, both for picking up and delivery. When she ordered to pick up, it was because the price was a little cheaper. Had an easy time on Papa John’s site.
“Being the first time on here, it took me a bit to find ordering pizzas that I want.”
Total Time: 5:50
She is a younger woman who has ordered online before to pick up, because the place she ordered from didn’t do deliveries. She felt it was an easy experience, but struggled a bit with finding things quickly on Papa John’s site.
“This is kind of cool, that it shows you your toppings being put on your pizza.”
Total Time: 6:13
He is a middle aged man. As a principal, he’s ordered with Papa John’s several times for his school, as well as other pizza chains. He feels comfortable ordering pizzas.
“That was pretty simple.”
Total Time: 11:48
Participant 4-Mr. Moreno
An older man who has ordered pizza for pick up before, because none of the places he visited delivered to his area. He’s used to Pizza Hut’s site, and feels Papa John’s site was not friendly to seniors like himself.
“It’s not senior friendly to me.”
Total Time: 11:38
Analysis of Task 1
Everyone was able to complete this task, but several had issues along the way. The most common issue was in ordering the half mushroom, half onion pizza. Some took a moment to figure out how to place toppings on just one half of the pizza. Some had to go back and edit their pizza. The animation slowed it down. Mr. Moreno complained about not finding things very easily in the menu. Amy had some issues with the cart, wondering whether it had saved her orders and why it counted two pizzas as one item.
The issues included figuring how to customize a pizza and editing mistakes.
Despite a few small issues, the interface proved successful in helping them complete the task.
Analysis of Task 2
This task generally did not have many issues. Jimmy tried clicking on the words in the footer instead of the icons, which caused some confusion for him. Mr. Moreno, on the other hand, did not see a way to do this without signing up for an account.
The issues included the lack of prominence and the use of icons instead of text as the link, unlike the rest of the site.
The interface proved somewhat challenging in finding what they needed, although most were able to find it with some looking.
Analysis of Task 3
Everyone knew to look in the footer under customer service, and quickly found a form to fill out. Jimmy took time to actually fill out everything and explain his thinking, which increased his time on the task, but even he found it quickly. Only he submitted information and went to the second part of the form, so he provided more information.
No major usability issues among those tested.
The interface worked well and everyone was able to find what they were looking for in a short time. Most didn’t fill out the form all the way, but talked about what they thought they would put down.
Given the limited scope of this study in both number of participants and number of tasks we asked them to do, further studies would reveal more about the website, including some of the more advanced features like account management and the usage of special deals. These are possible candidates for more research, but the basic functionality was mostly covered by this study.
What I Learned
While I couldn’t get a true feel of the scale of a usability testing project, I learned what goes into it and the useful data it provides. It is always best to do some testing, not just assume you know what issues the users are going to run into, and doing these exercises made that clear to me. I may have struggled with my first time, but with a little practice, I know I can get better.