PAPA JOHN’S USABILITY TESTING

To provide the best service possible on the website, Papa John’s asked to evaluate the effectiveness of their website and remove any usability issues.

The usability testing revealed the following key findings:

  • Novice users have difficulty finding the option to customize a pizza.
  • The half and half toppings icon is hard to spot, too small, and not intuitive.
  • Users don’t expect to be able sign up for deals and coupons without registering for an account with Papa John’s.

OVERVIEW

This project illustrates my ability to evaluate a product’s effectiveness and detect any usability problems by conducting usability sessions with an appropriate target audience. It also exhibits that I can successfully analyze the testing results through using the affinity clustering technique and drive recommendations from the findings.

PROBLEM

Papa John’s current web-based ordering site has been put together on a tight schedule and they suspected many aspects of it needed improvement. They expressed several concerns that were based on their assumptions. To validate their assumptions, usability studies are needed to evaluate the website with Papa John’s target audience.

ACTION

Build a Screener

To help isolate a segment of the population that represents Papa John’s target audience for the usability study, I established 10 screener questions. A team of UX designers and I suspected significant behavioral differences among various user types such as novice, moderate, and advanced users; therefore, we decided to build a screener for each user type. Each of us took a responsibility of creating a screener for a specific user type; my responsibility was for a moderate user type.

To identify the moderate users of pizza ordering websites, I first defined a target audience profile as follows:

TARGET AUDIENCE PROFILE FOR MODERATE USERS

BEHAVIOR
  • Have ordered pizza delivery from a website before
  • Order pizzas online one to eight times a month
DEMOGRAPHICS
  • Age: 18-70
TECHNOLOGY USE AND EXPERIENCE

Frequency of use (classifier):

  • Go online less than once a day
  • Go online one to five times a day
  • Go online more than five times a day

Computer literacy (classifier):

  • Not comfortable ordering products or services online
  • Somewhat comfortable ordering products or services online
  • Very comfortable ordering products or services online

Computer literacy (classifier):

  • Order a pizza from an app
  • Order a pizza from social media
  • Order a pizza from a website
  • Order by calling

Based on the target audience profile, I built the screener.

Develop Tasks

Next I developed a set of tasks for participants of the usability study to perform during the sessions. Before creating tasks for the usability study, I identified user goals as a basis.

WHAT USERS SHOULD BE ABLE TO DO ON THE WEBSITE?

  • Order pizza with ease
  • Customize a pizza
  • Sign up for an account
  • Apply a promo code to redeem reward points
  • Find nearest carryout location
  • Join the Papa Rewards program

Based on the user goals, I have developed the tasks.

Conduct Usability Sessions

Since this was a school project with time constraints, though a team of UX designers and I each created a set of tasks in the previous step, we were given the premade tasks to use for our usability sessions. To gain enough test results in a short period of time, each of us conducted a single usability session using the same premade tasks. This collected ample amount of results from approximately 22 usability sessions.

SPREMADE TASKS

TASK 1

We are going to be looking at papajohns.com web site. Even if you are not a Papa John’s fan, imagine that the people you are with are fans of Papa John’s and that is where you will be ordering the pizza. There are bunch of people at this party and you need to order 3 pizzas.

  • Mushroom and Pepperoni
  • Half onion and half sausage with light sauce
  • A specialty pizza (you want some variety)
TASK 2

You want to sign up for deals and coupons but you don’t want to register. You just want to give them your email. Can you do this and what do you think you will receive by email?

TASK 3

Your pizza arrives and it is terrible! Your driver was rude and you are really upset about what just happened. You call the local store but get nowhere. You need to contact the corporate office, how would you do that?

HOW WE CONDUCTED EACH USABILITY SESSION:

  • Invited a single participant per session to sit with a moderator.
  • Observed a participant’s behavior while he or she performed the given tasks on a computer.
  • Gave all the participants the same tasks to perform in order to identify trends and patterns.
  • Asked the participants to think aloud during the testing.
  • Recorded the screen, conversations during the session, and reactions of the participant (facial expression, gesture, etc.) with a screen capture software, mic, and webcam.
  • Asked the participant questions to further probe his or her thoughts and behavior.

Analyze the Data

1. AFFINITY CLUSTERING

By reviewing each recorded session, I captured the steps participants took to completing the tasks and important snippets of quotes onto sticky notes and then clustered them into groups of task.

2. FINDING PATTERNS

I identified commonly experienced problems in the affinity cluster and recorded the problems along with the corresponding quotes in an Excel document.

3. PRIORITIZING PROBLEMS

I prioritized the problems by criticality that was calculated by adding the severity and frequency raking. The higher the score, the more critical the problem was.

Participant Information

For the sake of this project, I focused on analyzing and making recommendations for a novice user type. I chose the following three novice users for the evaluation.

PARTICIPANT 1

  • Gender: Female
  • Age Group: 18-29
  • Have you ordered online before? Yes
  • Was the online order delivery or pick-up? Both
  • Reasons why ordered in each method: Delivered because guests were over. Picked-up because we were in hurry.
  • Have you used Papa John’s website? No
  • Which online pizza place have you ordered from? Domino
  • How would you describe your experience when you ordered online? Didn’t like the experience because we had issues using coupons or difficulty with trying to pick toppings. It wasn’t a seamless experience.

PARTICIPANT 2

  • Gender: Female
  • Age Group: 40-49
  • Have you ordered online before? Yes
  • Was the online order delivery or pick-up? Both
  • Reasons why ordered in each method: Delivered because it was convenient. Picked-up because it was faster.
  • Have you used Papa John’s website? No
  • Which online pizza place have you ordered from? Zpizza
  • How would you describe your experience when you ordered online? Ordering online was pretty straight forward, no problems at all.

PARTICIPANT 3

  • Gender: Male
  • Age Group: 40-49
  • Have you ordered online before? Yes
  • Was the online order delivery or pick-up? Delivery
  • Reasons why ordered in each method: Delivered because I was being lazy.
  • Have you used Papa John’s website? No
  • Which online pizza place have you ordered from? Antonio’s Pizza
  • How would you describe your experience when you ordered online? Ordering online was ok.

RESULTS

The table below illustrates the overview of the task results.

PARTICIPANT TASK 1

Customize and order three separate pizzas.

TASK 2

Sign up for deals and coupons without registering.

TASK 3

Contact the corporate office for complaints

Participant 1 TASK 1: Successful TASK 2: Unsuccessful TASK 3: Successful
Participant 2 TASK 1: Successful TASK 2: Unsuccessful TASK 3: Successful
Participant 3 TASK 1: Successful TASK 2: Successful (with a prompt) TASK 3: Successful

ANALYSES & FINDINGS

Task 1: Customize Pizzas

FINDING 1 (severity 4 + frequency 3 = critically 7)

Novice users have difficulty finding the option to customize a pizza.

ANALYSIS

Three out of three participants had a problem spotting the customize option on the Menu Page. The Create Your Own section is placed under the Papa’s Picks category, which may not be an obvious category to be placed under and in user’s sight until scrolled down. One user thought the section is a graphic rather than a clickable section. The section is not prominent enough and buried in other set pizza options. The participant 1 initially went to the Specials page while the participant 3 went to the Ordering page instead of coming to the Menu page for the customize section.

USERS’ VOICE

It’s not very clear where to customize. That was confusing. It wasn’t like its own header or section, easily at the top of this (page).”

I looked by that because it didn’t have a pizza on it but…I guess I just thought it was a graphic initially. But, now that I’m looking harder, I see it.”

Usability Finding 1

FINDING 2 (severity 4 + frequency 3 = critically 7)

The half and half toppings option often gets missed. Users find the half and half icon too small and not intuitive..

ANALYSIS

Three out of three participants expressed that the half and half option wasn’t immediately apparent—the icon was small and hard to decipher. Also, the icon wasn’t in sight of the user when the user clicked a topping. The user often missed the icon appearing above the toppings selection until he or she scrolled up. The two participants thought the half circle shapes were arrows and couldn’t figure out what those icons do until they actually clicked and saw the topping being placed on the 3D picture. Two participants said that having written descriptions might be helpful in understanding the icon better.

USERS’ VOICE

Only indication is by the picture. There is no actual verbal text to explain it. I’m down here. When I pressed on that (topping), I’d have to scroll back up to see what this icon was doing. I would have missed it. It (icon) was kind of small too.”

“I would think light or heavy toppings (what the icon means). That would be my first thought. I was thinking of them as arrows. To me when I saw that graphic, it wasn’t intuitive to me that it was initially half and half.”

Usability Finding 2

FINDING 3 (severity 4 + frequency 2 = critically 6)

The 3D picture showing what the customized pizza looks like doesn’t always reflect users’ specifications, which often confuses the users.

ANALYSIS

Two out of three participants saw that the 3D picture sometime didn’t match their specified options such as sauce and toppings. When a participant who chose to customize a set pizza—Spinach Alfredo—saw that the 3D picture of the pizza was shown without the Spinach Alfredo sauce, she thought she accidentally selected the wrong pizza. So then, she went back to the Menu page to re-select the Spinach Alfredo but again saw the same 3D picture. She was confused.

USERS’ VOICE

“I wish this picture would reflect what that original picture on a menu looks like. It just looks like a plain cheese so that kind of threw me off. I pressed on the wrong one?”

“So I already clicked no cheese…oh, we should have that graphic with no cheese because the graphic has cheese on it.”

Usability Finding 3

FINDING 4 (severity 3 + frequency 2 = critically 5)

Users find it slightly irritating that they’re taken back to the Menu page every time they’re done customizing a pizza. When they want to order another custom pizza, they would also have to scroll down to get to the customize section.

ANALYSIS

Two out of three participants indicated that it was somewhat frustrating to being taken back to the Menu page when they were done customizing a pizza. Instead of getting an option to stay on the customize page and proceed to creating another pizza, they would have to scroll down to get to the Create Your Own section again on the Menu page to customize another pizza.

USERS’ VOICE

“It’s kind of sucks that you kind of go back to the main menu thing.”

“Now I’m back at the beginning. It’s tiring.”

You do have to scroll down to get the second pizza. I’d like it if it says one more pizza right up front.”

Usability Finding 4

Task 2: Sign Up For Deals And Coupons Without Registering On Papa John’s

FINDING 5 (severity 5 + frequency 2 = critically 7)

Users don’t expect to be able to sign up for deals and coupons without registering with Papa John’s.

ANALYSIS

Two out of three participants had failed to accomplish this task. One participant was able to complete the task with a prompt from a moderator. The problem seemed to be that the participants tended to believe that they had to create an account in order to receive deals and coupons. Moreover, the link to sign up for deals and coupon was tucked all the way at the bottom right. In contrast, the “Sign In or Create an Account” and “Papa Rewards” buttons were placed at the top of a page where the participants can spot easily; hence, all participants initially clicked those buttons. Two participants said they would just google coupons instead.

USERS’ VOICE

“How does that work though. Usually you do have to…initially I would have signed in. I would create an account to get an email notifications of coupons. I think I would just google Papa John’s coupons and get it on sites like RetailMeNot. They probably would have deals that I don’t have to create an account with Papa John’s because I don’t want to get that kind of junk mail either through text or emails.”

Usability Finding 5

FINDING 6 (severity 3 + frequency 2 = critically 5)

Users are confused which fields on a form are required to fill out.

ANALYSIS

Whether it was the form for signing up for deals and coupons or creating an account with Papa John’s, the required fields weren’t indicated as required; hence, users were unsure of what they were supposed to do. Participants weren’t notified to fill out the required fields until they tried to proceed without filling out those fields.

USERS’ VOICE

Usually I’m expecting to see an asterisk next to the headers that are required for you to continue on to create this account...”

“It has a first name, last name, and email address field. It doesn’t tell you what is required.”

Usability Finding 6

Task 3: Contact The Corporate Office

FINDING 7 (severity 2 + frequency 2 = critically 4)

The corporate office number is too small and buried in other text; hence, it’s difficult to find.

ANALYSIS

Two participants expressed that the corporate office number is rather small and difficult to spot. Although all three participants were successful in finding the number, it was not standing out from other copies on the page and may be missed by busy users who may be in rush to find the number.

USERS’ VOICE

“It wasn’t very easy to find but I found it. Didn’t really stick out.”

“The way that the text is laid out, I don’t like it but that’s just me. Underneath all of that then there is a customer care team, which is an 877 number.”

Usability Finding 7

RECOMMENDATIONS

FINDING 1

Novice users have difficulty finding the option to customize a pizza.

RECOMMENDATION

Make the customize section more prominent and place in the area that’s more visible to users.

FINDING 2

The half and half toppings option often gets missed. Users find the half and half icon too small and not intuitive.

RECOMMENDATION

Place the half and half icon closer to each topping selection. Make the icon larger and more prominent. Revise the icon to make it more intuitive or provide written descriptions to indicate the half and whole toppings option.

FINDING 3

The 3D picture showing what the customized pizza looks like doesn’t always reflect users’ specifications, which often confuses the users.

RECOMMENDATION

Revise the 3D picture functionality to make the users’ specifications reflect on the picture.

FINDING 4

Users find it slightly irritating that they’re taken back to the Menu page every time they’re done customizing a pizza. When they want to order another custom pizza, they would also have to scroll down to get to the customize section.

RECOMMENDATION

After users are done customizing, provide them with the option to customize another pizza, continue shopping, or go to a shopping cart instead of taking them back to the Menu page.

FINDING 5

Users don’t expect to be able to sign up for deals and coupons without registering with Papa John’s.

RECOMMENDATION

Either combine the “Sign In or Create an Account” and “Sign up for Text and Email Offers” options as one option or make the “Sign up for Text and Email Offers” option stand out and provide better explanations to distinguish from the “Sign In or Create an Account” option.

FINDING 6

Users are confused which fields in a form are required to fill out.

RECOMMENDATION

Provide asterisks to the required fields to indicate they’re required.

FINDING 7

The corporate office number is too small and buried in other text; hence, it’s difficult to find.

RECOMMENDATION

Make the corporate number more prominent and maybe move to a more visible area of the page.

UI DESIGN

Ideate design solutions

Based on the recommendations, I sketched a series of wireframes that tackled the problems users faced in the usability testing. To further improve the UI, I also added new areas of design. Here are some of the key solutions to the usability problems:

  • To make the customizing pizza option easy to spot, I made the option large and prominent on a home page above fold. On the Menu page, I removed the Create Your Own Pizza selection from the Papa’s Pick category and made it as its own category on a local navigation.
  • To make the half and half topping icon more intuitive and apparent, I designed the icon larger, provided written descriptions, and placed closer to each topping selection.
  • To help make the process of customizing multiple pizzas easy, I provided a popup window offering the option to customize another pizza, add a set pizza, or go to checkout.
  • To help users easily find the sign up for deals and coupons option, I relabeled the option as Sign up for deals & coupons and made it available on the sign in form as well as at the footer area.