About Product X:
Product X is an iOS app that helps beauty consumers find hair products that will most likely work well for them. With the help of this expert system, consumers answer 4 questions via chat and are able to scan or search for hair products to receive personalized predictions (known as match scores) on their performance.
An additional product, called Product Z — an analytics platform for brands and retailers — was also being built during this time. This product provided information on consumer activity when using Product X.
Background:
Immediately after coming on board full time in October 2018, I was responsible for working with the development team to salvage any major UX issues that could be tackled within a month before the beta app’s soft launch in early November, in addition to leading the UX efforts for additional updates quickly after launch. Please click here to learn more about my contributions prior to the app’s soft launch.
This page will focus on my contributions after the soft launch.
The Project Details
Timeline: ~2 months
Team: The Company X team consisted of a remote team of 6 people: myself, 1 in San Diego, CA, 1 in WA, and 3 in Costa Rica. I worked closest with the Founder/CEO, the Head Engineer, another Lead Engineer, and the Lead Quality Assurance Tester
My Role: Director of UX (User Research, User Testing, UX, UI, QA Tester)
Tools Used: Figma, Sketch, Google Slides, Asana, Test Flight
Objective:
Following the early-November launch, I was responsible for acquiring as much feedback as possible from users in order to determine design changes that would be required for future updates. This required me to gather feedback in creative ways due to a limited budget.
There were several iterations that were made based on the feedback received — this page will cover some of them. The development team, primarily located in Costa Rica, was near the end of their contract so the goal was to identify any and all major UX and/or design issues, prioritize them, and work with the engineers to implement what was most important.
In addition to finding necessary changes, there was also a need to align the consumer product (i.e., the beta mobile app) with the business-facing analytics platform, Product Z. We needed to ensure that the mobile activity was being captured and displayed accurately.
Below is a summary of our team’s main goals:
Real-time data capture
Foster a community of consumers who are excited to continually provide information
Create and maintain valuable, pleasant experiences
I focused primarily on understanding how users viewed their hair and their hair/hair product journeys in addition to how they felt about the mobile app while using it.
Solution:
I thought it was important to gather feedback on the app via conventional testing and through somewhat controlled observation since we believed the app would be more likely to be used in store. Because of this, I approached the objectives with 2 main plans in mind:
Interviews + moderated and unmoderated user testing
Interviews + Testing Approach
Observation/Testing event at Sally Beauty stores
Observation/Testing Approach
Immediately Post Launch:
We soft launched the mobile app while at the AfroTech (2018) conference in San Francisco. To gauge sentiments toward the app and to do some initial testing with real people, I set up an area inside the conference location for interested participants to test.
Initial user testing at AfroTech 2018.
Initial Research:
Since Product X was just launched, we did not have any existing users to enlist for providing feedback. This required me to get input from users who I hypothesized to be in our target market.
I started off by creating 3 user personas* at a high level. These personas were hypotheses derived from market research of the beauty/personal care industry.
Confused parent*
Beauty enthusiast*
Ingredient-conscious beginner*
*personas & terms altered for portfolio
Initial high-level user personas.
Because our budget was very limited, I needed to find a way to gather information from multiple individuals that didn’t require significant funding. I chose to explore different channels that typically receive high traffic each day in addition to hosting an event and interviewing people in our network. The process for utilizing Craigslist and Task Rabbit is detailed below:
Craigslist
Create screener survey to only accept participants in target market
Post screener on Craigslist
Schedule testing time in coffee shop
Interview + Test
Analyze & report feedback
Task Rabbit
Create Task Rabbit post & select users that appear to be in target market within budget
Schedule testing time in public area
Interview + Test
Analyze & report feedback
The participants acquired from Craigslist were no-shows. I adjusted our method of recruitment from Craigslist to Task Rabbit because of this and also due to the potentially unsafe environment that Craigslist can create since I was conducting this research by myself.
In order to prepare for each test, I created a test script that included the test agenda, hypotheses/goals, warm-up/customer discovery questions, and tasks. The warm-up and customer discovery questions served to provide more qualitative information that I could use to update the user persona hypotheses.
Sally Beauty Observation and Testing Event
In addition to gathering feedback, we used this event to test whether or not the app activity was successfully being recorded and shown on the analytics platform.
The testing event was mainly held to gain feedback from participants who lived in the LA area; however, other testers who wanted to participate in other areas were invited to. In total, we had 11 testers participate locally.
Note: the app producing a prediction, or “match score,” is dependent on the product existing in the app’s database. Users can request products that aren’t found; however, action needs to be taken on the admin side to approve. In order to ensure that the majority of the products were present in the database, I visited each of the Sally Beauty test locations and scanned and requested over 1000 products total.
Hair product aisle in Sally Beauty.
The process for the Sally Beauty events was as shown in the graphic below:
Throughout the event, I noted and recorded the participants’ questions and comments. The events took place in 3 Sally Beauty stores located in the Los Angeles metro area: in the Burbank area, Downtown LA/USC area, and the West LA area.
Testing participants browsing with Product X.
Immediately after each event, participants were encouraged to complete an online anonymous survey before leaving that focused on their general feelings toward the app.
The survey also included questions that evaluated:
Ease of use (0 = Not at all easy; 10 = Extremely easy)
Perceived value of the app (0 = Not at all valuable; 10 = Extremely valuable)
Overall experience (1 = Not at all pleasing; 5 = Extremely pleasing)
Net Promoter Score (NPS)
Feedback Analysis:
Following each event, I watched the recordings to create a spreadsheet of all of the feedback that was gathered. I also included the feedback received from the other methods of testing performed. I documented all things, positive and negative, that were captured in regard to the app and placed them in 1 of 3 categories: Feature Set, UX Design, or Messaging.
Once documented, the number of people who mentioned similar issues or made similar comments were noted. Statements/sentiments that occurred more often were labeled as more urgent to address.
Once everything was compiled, I placed my primary focus on issues that were specifically related to UX Design or that strongly impacted the UX. Then, I separated each of the issues into sections that were based on the app’s main functions. In doing this, I was able to align the main functions of the app to its most critical flow to determine the issues that took top priority.
Major Learnings:
The initial round of user testing at AfroTech helped us to understand that phone signal plays a huge role in the app’s performance. It also allowed me to gain more perspective on different use cases and information that the testers did and did not understand.
We gained enough information to quickly update the app and launch it publicly on November 30, 2018.
There were several major key findings that came out of testing the app in Sally Beauty:
Beauty consumers like to talk about hair issues and goals when they are getting help; not asking enough questions makes them feel less convinced that they’ll receive valuable help
The lack of instruction after users request a product is confusing
Users long for more information that details why they received a particular product match score
Setting “male” as the default gender in the profile settings solicits negative reactions since our primary target (at the moment) identify as female
Several testers do not know the professional term that describes their hair texture type and, therefore, found that question difficult to answer accurately
Loading time after users select their answers makes the app appear to be dysfunctioning
The “Search,” “List,” and “Rating” features were not immediately obvious or clear to participants and were rarely accessed
Testing Observations
Users really enjoy getting high scores for products; it’s almost like a game for some of them where they feel like “they’re winning.” When they receive low scores, especially multiple low scores in a row, they become a bit frustrated and long for a better recommendation.
Survey Results
The app received an NPS score of -9. The app’s ease of use centered at a median of 7.5 out of 10 and its perceived value at a median of 8 out of 10. The participants’ overall experience was moderately pleasing (a median of 3 out of 5). With these findings, I was able to establish benchmarks to compare with future results.
Major Takeaways (based on testing and AfroTech feedback):
Users need to be given direction at each step of their journey through the app
While 4 questions is sufficient for some, many users are willing to answer more questions to feel as though Product X “knows” their hair characteristics and goals
Visual feedback during times when information is loading is an essential part of the UX
Not everyone characterizes their hair textures with the same terms and many users consider themselves as having more than one texture type
Recommendations:
For next steps, I summarized and presented design recommendations for the Founder/CEO to review. In order to improve the UX of the app, several updates were required; however, I focused primarily on fixes that were of high value to the company (i.e., aligned with business goals) and those that did not take significant development effort:
Settings - change default gender setting to “female”
Product Requests - remove optional/confusing fields; add informational message that shortly explains what happens when requests are submitted that include direction to continue scanning other products
Onboarding - Notify users right away that they can edit their responses after completing the chat conversation
Onboarding - Provide visual feedback as user’s answer loads
Onboarding - Replace hair texture terms with photographs of different texture types
Score - Add additional score details to the visual match score; educate consumers on how scoring is calculated
Recommendations - Add a feature that provides high match recommendations
Education - Add content that directs users to take action on match score screen; re-work definitions provided for ingredient preference types to be more understandable
Lessons Learned:
This process highlighted the importance of creating a user flow or screen flow in addition to the design screens. It also highlighted the importance of conducting user research and user testing with designs prior to building the product. The team did not have personas, flows, or thorough prototypes to guide our work.
Although these are not things that I had control over (the app design was completed by a design agency and was already in the process of being developed when I joined the team), they still greatly affected my ability to do my job effectively and negatively impacted the team. There were several changes that I would have made to the content structure and UI just based on UX principles had I been involved earlier in the process.
Next Steps (Future Iterations):
Include users in as many steps of the design process as possible
Create design artifacts (e.g., personas, user stories, user flows) to keep team aligned
Implement recommendations that can be completed within the time of development contract