Experience Research Intern
12 weeks, May – Aug 2019
Guest Homes Strategic Supply Team
In summer 2019, I started my journey at Airbnb HQ as an Experience Research Intern on the Guest Strategic Supply team. The team aimed to merchandise long-term stay listings and hotel listings in H2 2019 to provide guests with more choices in their stays. Over the 12 weeks of internship, I worked on two projects with my cross-functional partners:
1. Long-term stay flexible date pivot research: 6 remote moderated usability testing sessions
2. Hotel vs. Home reviews research: 7 remote user interviews + IC panel survey
For both projects, I went through the whole research process from scoping the research plan, recruiting participants, executing research and sharing findings with stakeholders. I would like to spend more time focusing on the second project as it’s a foundational research that required me to explore the unknown field and deal with ambiguity. Through this research, I was able to use mixed research methods and tell a story of our guests with different data sources, and eventually influenced the product strategy at a higher level.
As Airbnb acquired HotelTonight in April 2019, the team hoped to better merchandize hotel listings on Airbnb. However, from Q1 report, we noticed that:
Hotels are reviewed less frequently than homes (hosts’ properties).
Hotels are less likely to get five stars.
Hotels receive lower scores overall.
The customer tickets gave us the numbers and great insights. Then we need to know how and why guests’ behaviors differ when they review hotels vs. homes, and subsequently give suggestions on how to better merchandise hotels.
There were so many questions related to the review and rating system that could be added to my research from previous reports. There were so many details on the design of the review section that our designers were curious about. There were so many OTA (online travel agencies) competitors doing different things that I wanted to know how users would react to them. There were so many potential research directions I could choose... Scoping a foundational research plan could be really hard when you had to strike the balance between what’s most valuable and immediate to your team and what’s helpful for the company in a long run.
To give myself more context in the realm of hotel stay experience, I participated in a design workshop at HotelTonight, where the marketing managers and HotelTonight team shared what they heard from guests about their expectations of hotel experience--I found it highly relevant and informative to my research because reviewing was about the matched/unmatched expectations. From there, I realized I need to understand how guests would evaluate hotels and homes based on their expectations.
To address designers’ concern on how they should redesign the review section and PM’s concern on how to better surfacing hotel listings, my research goals were to:
Understand how guests perceive reviews of hotels vs. homes
Identify hotel criteria that guests would find helpful in reviews
Give suggestions on how to better merchandise hotels with a focus on reviews
Researchers should be product-oriented and purposeful--always ready and resourceful to support the product team with research plans that would solve their uncertainties about the users. As a researcher, I learned the importance to include stakeholders’ goals and concerns into my research scope from the beginning. I shared my research proposal to stakeholders and refined my research questions to make sure I could get actionable insights from users’ answers.
Outlining research questions is a collaborative effort.
Since I wanted to understand how and why people interpret the reviews of hotels and homes differently, and a large part of hotel guests were international travelers, I decided to conduct remote 1:1 user interviews with participants from English-speaking country (US, UK, Australia, etc). I recruited 7 participants with mixed gender, age, trip type, room type, group size, length of stay, and price range. All of them wrote at least one review for their past trips.
Since one of the research goals was to know the review criteria for hotels, I decided to do a card-sorting exercise after interviewees talked about their experience and perceptions on hotel and home stays -- so they could refer to what they’ve said to make judgements. The preset options were based on competitor analysis of other OTAs and I would add new ones on the empty post-its if interviewees mentioned anything while they were sharing their experience.
From the qualitative research, I had some basic ideas on what criteria guests may use to evaluate hotel experience. But I was unsure if it could represent the majority of the hotel guests. To see the trend and preference over people’s criteria over hotel reviews, I followed up this research with a short survey, sent out to 238 global panelists through Qualtrics. Compared to a larger-scale survey, the panel survey could offer faster and higher-quality responses. Considering the time constraint of my research, I did not have enough time to launch a survey to the general audience and analyze a big amount of data. But I hoped to give some directions for design or for future research so I tried out the IC panel to know whether there was any regional difference for guests’ criteria for hotels in reviews.
When conducting interviews, it’s so easy to unconsciously ask for what we’re looking for. It’s hard to find a good way to direct the conversion without asking leading questions. But, practice makes perfect. By piloting with teammates (sometimes the first interviewee could also be a “tester”), I could come up with my strategy to go with interviewee’s flow and ask follow-up questions. Another tip was “playing dumb” and asking interviewees to elaborate more on their opinions.
When doing interviews, it’s a good time to engage designers or PMs to join the session and take notes, or at least I would post the live interview notes in the team slack channel. In this, I could keep them updated and build empathy in an effortless and effective way.
Coming up with the right research scope and methods was vital, but communicating the findings through a compelling way was equally important. When showing qualitative data, it’s always nice to have real quotes and video clips. When it came to quantitative findings, using the right graphs and data visualizations could be convincing and appealing.
Usually it’s a good idea to archive information I wanted to present in one full presentation. When I presented to different stakeholders, I might emphasize different parts and dive into different levels of details. Due to NDA, I was not supposed to share how I visualized and presented the findings here. But if you’re interested to know more, please feel free to contact me at email@example.com.
“Be a host!”--That’s the sentence written on our badges, reminding us to be accommodating and welcoming to everyone around us. I felt like, through these three months, I grew not only in my research and communication skills, but more importantly, in my mindset to be a more inclusive and caring human being.
From the first day, we were encouraged to believe in ourselves and our teammates, and I never felt more empowered because of all the support from my recruiters, intern peers, research fellows and teammates 💙.
Huge huge thanks to —
🌟Emily — my mentor who set me up for success, made me feel safe to talk about all of my thoughts and concerns, showed me how caring, responsible and knowledgeable a mentor could be, gave me enough space to grow and encouraged me to pursue the career path I believed in.
🏨My guest strategic supply/hotels team — Liz, Drew, Renee, Betty, Xiaowen, Chelsy, Matt —the Dream Team, always resourceful, inspiring, and genuine.
🔮Caitria, Nanako — thank you for the inspiring chats and making me start to think how to do research differently to bring more value to teams, and also to underrepresented user groups.