About Product Y:

Product Y is a Shopify app that helps to educate consumers about the hair products on a merchant’s website. Shoppers can interact with Product Y directly on the merchant’s product page. After a quick conversation, they receive a match score which informs them of how well the merchant’s product(s) is predicted to perform on their hair. Also within the app, is an admin panel that Shopify merchants use to view app activity and manage Product Y on their site.

Essentially, Product Y provides the experience of the Product X mobile app on e-commerce platforms.

Background:

I learned a great deal when testing the wireframes of the consumer experience and moved on to designing the high-fidelity mockups. To learn more about how the wireframe tests were performed, please click here.

In order to assess the usability and get an idea of how users might naturally progress through the high-fidelity designs, I needed to conduct another round of testing. This page will focus on my contributions to user testing the consumer experience.

Objective:

Since I’d completed the first iteration of the high-fidelity mockups, I needed to ensure that users could successfully understand the details that had been added to the wireframes. My approach to best do this was as follows:

• Create a prototype in InVision

• Draft a list of directions and tasks to enter into UserTesting.com tasks

• Set up unmoderated UserTesting.com tests

• Release dry-run tests & edit as needed until confident with test

• Release improved tests one by one

• Review feedback and iterate where needed

Testing:

I tested the high-fidelity mockups (mobile only) using the UserTesting.com platform. I utilized the free trial, which allowed us to create 15 unmoderated tests total at a discounted price with limited access to special features.

In total (before my departure from the project), I created and received feedback for 8 tests. The tests took participants around 7-30 minutes, depending on their talkativeness and ability to progress/speed through the tasks.

I established the following testing goals:

• 100% of participants - initiate chat & make it through chat to receive a match score

  • In order to generate revenue, users needed to receive a score for the product(s) purchased; therefore, all testers must receive scores

• 80% of participants - learn more about their score

  • To maintain engagement with the app, most testers needed to continue exploring after receiving a score

• 60% of participants - see email submission screen

  • Email addresses are valuable to merchants; therefore, more than half of the testers needed to see that they could save high-matching products via email. (I determined that UserTesting.com would not provide us with a realistic journey that would show testers submitting an email without direction via a task, so I believed actually seeing the submission screen to be the next best thing)

Major Findings (from the first 2 dry runs):

• Participants read at different speeds

• Participants adhere closely to the tasks and will not progress through the entire product “naturally”

• The prototype tends to display the InVision screens differently on different mobile devices

• When a participant’s profile option did not match well with the product, they wanted a brief explanation with the reason as to why it didn’t match

• Neither user journeyed to the email submission screen

Adjustments/Design Updates:

Taking what was learned from the first 2 tests:

• I slowed down the timed portions of the prototype

• The scenario was adjusted to prompt testers to explore more of the app, thereby leading them to remain more active before moving on to the next task

• Tasks were added to ensure that we saw users use/give feedback on a certain feature even if they did not do so during the exploration scenario

• I adjusted the spacing in the prototype screens so that all content (that was not meant to be scrollable) would be visible on all devices without having to scroll

• Re: the brief explanation — this was a request we had heard before. Because it seemed to be important to several consumers, a short popover was added for all profile options that did not match well with a product

• I redesigned the score screen to include the email submission form on that same screen

Major Findings (from the next few tests):

• Consumers wanted to know more about the way the match score was calculated

• The ability to edit profile information was not being acknowledged

• Some testers weren’t tapping the “no match” icons to get more information

Adjustments/Design Updates

• Add general information on scoring

• Move the “Edit Profile” button up near the section showing the current profile

• Redesign icons to test if it influences behavior change

Major Findings (from the last few tests):

• 100% of the testers acknowledged the edit profile option

• 100% of the testers scrolled down and viewed the email submission form

• Some users still did not tap the “no match” icon to get more information

Additional Testing

Moderated Remote Test

I had the opportunity to test with a consumer via a moderated, remote testing session. This test lasted around 20 minutes and was conducted via Zoom. During the test, I had her angle her computer camera so that I could view her interaction with the consumer experience prototype on her phone.

Score Screen Preference Test

For the score screen “match” and “no match” icons, I’d designed 2 different options. When creating the second option, I reached out to 7 people in my network to get a sense of what they thought each screen (with the different icons) meant. The second option I decided on based on feedback is shown below:

Since both seemed like they could communicate the information accurately, we thought it might be best to create a between-subjects preference test to decide which design to move forward with.

During user testing, we started off by including the first design (with the checks ans x’s) in the prototype. Testers did not seem to have any trouble understanding what this design was communicating, so I deemed this preference test to be unnecessary; instead, we moved forward with the first option.

Observations From Testing

It seems that testers who tap to view more information about the “no match” icons are ones who actually want to see more information; for other testers, the icon indicating that it does not match well with the product is enough information for them to be satisfied.

Lessons Learned:

• Releasing 5 (or more) tests all at once after a dry run once all is fixed would be an ideal approach; however, when financial resources are limited, releasing the tests one or two at a time helped to lower the risk of faulty/unintended changes as edits from previous dry runs were made
  

• Testers on the UserTesting.com platform stick very closely to the tasks they’ve been given; expecting to see a true natural journey that one might take in reality may not be 100% possible
  

• It is a mistake to postpone testing until all mockups are complete

  • Testing different features will allow for a more thorough test and more valuable feedback. This would also help to keep the test time to a minimum, thereby reducing the chances of testing fatigue
    

• It’s important to stick closely to/speak up on behalf of my design process when the timeline seems too aggressive

  • At the start of this project, I believed that there was not enough time for me to test the high-fidelity mockups of the consumer experience, so I removed it from my plans. Later in the project, the Founder/CEO grew frustrated that this was not included in my plans. Instead of removing it, I should have strongly advocated for a later deadline — using this need for user testing as the reason. This would have put me in a better position from the start
    

• It’s incredibly helpful for developers to be involved in the design process

  • There were several instances during this project where I needed to backtrack and change or remove portions of a design due to their complexity and the amount of effort they would take to build. Having a developer involved earlier would have been more efficient and could even have led to more innovative ideas

Next Steps:

After completing the testing and iterating based on what was learned, I added the updated prototype and screens in addition to the screen flow, other design assets, and design notes in InVision. This was passed off to one of our contracted Shopify developers.

At this point, my time with Company X ended and I closed out of the project.

Accessibility, IxD and CCPA

Before passing my screens to be developed, I learned of accessibility guidelines, interaction design (IxD) best practices, and the California Consumer Protection Act (CCPA) that was going into effect January 2020. Since our product would be available and used via the web and the company was based in Southern California (and would have users from California), we thought it’d be best to update all screens to meet accessibility and CCPA guidelines in addition to following IxD best practices.

To continue learning how I updated all mockups to adhere to these guidelines, please click here.