About Product Y:

Product Y is a Shopify app that helps to educate consumers about the hair products on a merchant’s website. Shoppers can interact with Product Y directly on the merchant’s product page. After a quick conversation, they receive a match score which informs them of how well the merchant’s product(s) is predicted to perform on their hair. Also within the app, is an admin panel that Shopify merchants use to view app activity and manage Product Y on their site.

Essentially, Product Y provides the experience of the Product X mobile app on e-commerce platforms.

Background:

The first set of wireframes for the consumer experience were created and in need of user input. To learn about how the consumer experience wireframes were designed, please click here. 

In order to get feedback, I attended an event called The Burnout at Vector90, a co-working space in South Los Angeles. At this event, multiple people from the surrounding area gather in one place to get work done for about 8-12 hours straight. I chose to attend because I saw this an opportunity to use the venue and event to conduct user testing sessions. This fit perfectly into our already limited budget.

Objective:

To get feedback on the wireframes, I took the following approach:

• Create and use test script consisting of an introduction to the product/company, warm-up interview questions, wireframe tasks, and concluding questions

• Have participants test on their own devices

Goals for testing:

Testing:

I tested the Shopify app wireframes with a total of 6 participants and each test lasted around 30 minutes. Prior to testing, I created a mobile and desktop wireframe prototype (using Sketch) which I’d prepared in the Marvel App. 4 of the participants tested on the mobile prototype and 2 tested the desktop screens.

Additionally, in order to get feedback on the score screen and whether or not participants preferred graphic information over text, I prepared a display that showed the previous score screen graphic. This graphic was shown alongside the wireframe.

After each test, the participants were awarded with a $10 gift card and were required to sign a receipt to indicate they’d received compensation. On this receipt, I added a small questionnaire in order to (1) receive permission to contact them in the future and, (2) assess their personal interest in the app.

Guerrilla [Preference] Testing

CTA
At the Vector90 event, I also set aside time to gather feedback on the CTA from individuals outside of the user testing participant group. I was able to gather input from 7 people about which wording they preferred and why using the kiosk mode on UsabilityHub’s platform. These were the options participants chose from:

• Get a Personalized Regimen

• Find What Works Best for You

• Shop Smarter with our Beauty Expert

• Digital Beauty Expert

• Personalize Your Shopping Experience

Product Page Button Position
To determine where the product page button should go, I created two wireframe designs of a mock website product page. One design had the button near the price while the other design had the button positioned in the upper-right corner of the main product image.

This test was performed first, prior to the user tests and other button tests since it informed the way ahead for the mock product page designs that I needed to include in the prototype.

The test was set up as a variation set of 2 within-subject, first-click tests. A total of 38 self-recruited participants provided feedback. In one test, the design with the button near the price was shown first and in the other, the design with the button on the product image was displayed first in order to reduce the chances of bias. 

For each test, I posed this scenario and question to the participants: 

You're not sure if this product is best-suited for your hair. Where would you click to find out?

Followed up by: 

What would you expect the "Will this work for me?" button to do?

Product Page Button Text
In order to get feedback on the product page button text, I created a click test on the UsabilityHub platform for 4 different text variations. The test had 38 self-recruited participants in total. For each variation, I posed this scenario and question to the participants:

You're not sure if this product is best-suited for your hair. Where would you click to find out?

These were the options participants chose from: 

• Will this work for you?

• Will this work for me?

• Find out if this will work for you

• Will this work well for you?

Each variation had either 9 or 10 participants (each of whom only saw and gave input on the single variation) who clicked on a specific part of the mockup screen they were shown. This click was recorded as well as the responses to the follow-up question: 

What would you expect the gray button (located beneath the price) to do after clicking it?

Major Findings: 

Following the testing sessions, I input all of the feedback gathered into a spreadsheet broken out into 3 main categories: Feature Set, UX Design, and Messaging (organized by each major feature). I recorded all positive and negative comments in the spreadsheet and tracked how many participants said the same/similar things. This allowed me to discover the most common issues that needed to be tackled immediately as well as other adjustments to possibly implement much later in the future. After analyzing the information, I created a feedback summary presentation for my team. Some of my findings included:

Product Page Button Position

• In both tests, over 60% of the participants chose to click on the button near the price first when compared to the less than 40% that clicked on the button on the image

• Several participants expected to either learn more about the product or provide more information about their hair after pressing the button 

Chat

• The chat posed no problems for the majority of participants

• One participant mentioned that she didn’t know her texture because no one had ever told her what it was

Match Score

• At a high level, most participants understood the significance of the match score and the implications of a high match score vs. a low match score

• Several participants mentioned wanting to know more about the reason for their score 

• Many participants mentioned wanting to know their score right away without having to take action to see it

• Participants expected to see information that described what the score meant in the score details description; the majority of participants wanted this information displayed visually, with some stating that they’d prefer both words and a graphic

Match Score Buttons

• Out of all of the participants, most participants expected the high match score button to be green because that signified a high match to them; some also mentioned blue or purple because they liked those colors

• The majority expected the low match score button to be red because that signified low, stop, and was attention grabbing

Regimen Requests

• Participants were excited about receiving a regimen and most indicated that they would be willing to submit their email address for that or a coupon

• Most participants understood what a regimen was and expected to see directions/step-by-step instructions on how products should be used in the email they received

Edit Profile

• The profile screen posed no problems for the majority of participants

CTA Text

• Results from guerrilla and user testing: Get a Personalized Regimen - because it was clear what pressing the button would do

• Runner up: Find What Works Best for You - because it was an action that the participant was interested in taking

Product Page Button

• Results from UsabilityHub: Will this work for me?

Low Scores

• The participants who encountered the low match score screen desired to see recommendations of other products on the site that had higher scores

Lessons Learned:

• It’s important to treat all components as a system

  • When setting up the testing for the CTA and product page button, I failed to treat them as if they were part of the same product. It is imperative that the design remains consistent; however, the text variations that I proposed and tested had different pronouns (i.e., “you” vs. “me”) when the pronouns should have either been the same or left out altogether

  • Additionally, the CTA and product page button are both components that were going to live on the merchant’s website and be seen by many. The merchants should have also been given a chance to provide feedback while this testing was taking place in order for us to know what they were comfortable with and if they had a strong preference

  • These realizations resulted in me having to backtrack and conduct another set of testing with merchant input. This also led me to start the process of developing a design system

• While the first-click test was helpful in some ways, it may have been better to set up the product page button test as within-subjects preference test. If there was a variation that participants thought of as better than what they saw in the test that only displayed one variation, that’s information that we now do not have

• Product page button text may have needed to be tested before the button position test, outside of the mockup. There may not have been a perfect way to do it at the time; however, testing the text first would have informed the copy that was displayed during the button position test

Next Steps (Future Iterations):

• Incorporate feedback into consumer experience wireframes where necessary

• Begin high-fidelity UI design of consumer experience wireframes

• Design and test merchant experience of the Product Y Admin Panel

To continue learning about the next steps taken on designing/testing the consumer experience UI and the admin panel, please click here.