Market Insights Research Intern
12 weeks, Aug – Nov 2019
Partnerships Team, Market Analytics Team
Over the fall of 2019, I had an incredible opportunity to work as a Research Intern on the Market Insights & Analytics Team. This team focuses on customer and competitor research, and looks into web analytics to uncover customer needs and market opportunities for Mailchimp. By pulling together survey data, interviews, and market research, the team provides an ever-evolving view of the customers and the industry.
Over the fall semester, I worked cross-functionally with Analytics Team, UX Research Team and Partnerships Team to design a survey study to explore customers' preference and attitudes over integration tools, or adjacent platforms, on Mailchimp. The findings informed the direction of integration iteration and future partnership opportunities.
A quick view of the integration directory on Mailchimp.
Mailchimp aims to build an all-in-one marketing platform to empower SMBs (small/medium businesses) to do all marketing in one place. In that, it needs to provide an integrated app experience to help customers work smarter and grow faster. However, the current situation for small business owners is that they need to switch between platforms and applications, which could be tedious and time-consuming. That's why Mailchimp provides over 200 integrations that customers could use to transport data easily from platform to platform.
New tools and integrations are always coming out as customers keep coming up with new needs. In order to understand their needs, I ask:
What integrations are popular among users currently?
How satisfied are they with the current integrations?
What integrations do they want to add to Mailchimp?
To put myself into context and understand the integration landscape quickly, I started from existing competitor research reports and customer research. Building my knowledge on the secondary research, it's easier for me to know what to focus on while outlining survey questions.
These reports and dashboards gave me a good understanding of what we've already known, what we were still unsure, and what we should ask in the survey. By incorporating different sources of "truth", I'd like to see how the survey research could validate or contradict with previous findings. I'm a firm believer of using mixed-methods to minimize the biases of a single research method, and to convince myself and my stakeholders with both qualitative and quantitative data.
In order to understand the trend and preference, I decided to design a survey study to explore the current usage of integrations for each segment (types of customers, slightly similar to persona, but different) on Mailchimp and other competitor websites.
Survey is a great method to measure product performance in order to know people's attitudes, impressions and opinions around the product features if a product is already launched. Survey is also useful in the product planning phase when looking for new directions. In my case, I wanted to quickly assess the performance about current integrations and explore new integration opportunities. So, survey it is ❤️.
Source: When to Use Which User-Experience Research Methods, Nielson Norman Group. nngroup.com/articles/which-ux-research-methods/
With the bigger research question in mind - what integrations our customers are currently using or hope to add to Mailchimp, I outlined the questions and got early feedback from my team where I found designing research was an iterative process and practice.
Research sounds like an independent and isolated work before interacting with real users. However, it’s not the case in product-based companies. Research is throughout the development loop and always needs to incorporate cross-functional perspectives to be intentional, informative and impactful. Sharing the outline and draft with other researchers, Partnerships Team an Analytics Team allowed me to make quick changes to align our goals together, and at the same time got them engaged at an early stage. Designing research was for sure a collaborative effort and required me to proactively seek feedback from various stakeholders.
While compiling questions and options, I also thought about the logic of each question to put them into the right order, and add conditions accordingly. Putting notes before each question was super helpful and convenient for coding them in the survey software later on.
Language or the way of asking questions was also important. Questions need to be specific and use familiar words so recipients could understand them with little effort. Terms or jargons, if have to be used, need to be clearly defined or elaborated with examples. For instance, what I meant by “integration” was “mechanisms that connect two different tools so they can work together”. To get richer and less-biased data, it’s better to ask “how” and “why” questions instead of “yes/no” questions.
When writing options for close-ended questions, they need to be exclusive. And it’s always nice to have an “other” option where recipients could add self-reporting answers.
An example of how I used the tips to write survey questions
After transferring all the content from google doc to SurveyGizmo (survey tool used at Mailchimp) and making sure all the logic and conditions worked, I pulled the user contact list from Looker (data storage and analytics software) using SQL.
- Participant Pool: Considering the response rate (~1%), the survey would be sent out to 100k recipients to get enough data to do quantitative analysis.
- Active Users: The recipients should be active users who performed some activity in the past 6 months
- User Types: The recipients should cover both free and paid users, domestic and international users, evenly distributed.
- Skewing the Quota for Integration Users: Since I especially cared about users who used and had opinions on integrations, I added a quota sampling to include ~150 users who had at least one integration (API key) with Mailchimp.
The screener questions could also help to filter to get targeting users.
- Employment Status: The recipients should be full-time employees.
- Roles: The recipients need to have some knowledge, experience or interest in marketing.
- Revenue: The organization/company could make revenues.
Doing a pretest or a pilot for the survey was 100% necessary for quality check purpose. I emailed the survey through Mailchimp Campaign to myself and my team first - the same way as I would distribute it to the whole recipient group.
Things for looking for in the pilot test:
- Does the survey link work?
- Is the length of time it takes to complete the survey okay?
- Does the survey look okay on mobile devices?
- Does the flow of the questions make sense?
- Do the questions and options raise any confusions?
Seemed that everything was ready to...
Hold on... Before sending the survey to such a large group of audience, it’s necessary to get another round of quality test with real users. I soft launched it to 10% of the total recipient group (random sampling) who haven’t done any surveys from Mailchimp in the past 3 months.
Things to look for in the soft launch:
- Is the length of time it takes to complete the survey okay?
- Are there strange or unexplainable patterns in the data? Could it be caused by confusion around my questions?
- Am I able to answer my core research questions with the data?
Launching the survey to real users was definitely the most nervous and exciting moment ⭐ - that’s exactly why we need to do so many testings to make sure the questions were meaningful to both the users and the research goal ️.
Due to NDA, I should not share any specific data from my research. But if you’re interested to know more about how I analyzed and presented my research findings, feel free to hit me up at firstname.lastname@example.org.
SurveyGizmo was a great tool to provide quick insights and I was able to segment responses to see patterns on it. For further analysis and creating graphs, I mainly used Excel and Tableau to clean the data and style the graphs. It took me around 2 weeks to finish the share-out deck -- again, I got feedback as I was putting together the insights and graphs on it. I think the biggest lesson I learned from the previous internship experience was never waiting until I came up with a perfect and finalized work to get feedback. It’s risky, too much effort, and there’s no way to have a “finalized” version. And I benefited a lot from keeping all the stakeholders involved and on the same page, so there’s no unexpected surprise (or shock) at the end 😁.
Through this research I was able to inform and influence strategies on different teams:
- Partnership Team: Recognized potential partnership opportunities
- Social Team: Prioritized social campaigns and co-marketing efforts for undervalued integrations
- UX Team: Further user research directions and design opportunities
- Market Insights Team: Contributed to the customer research repository
My research internship ended but the research is never ending. I could see this survey as a starting point for further qualitative research to understand why people have such preferences and feelings, and also a continuous work since people's preferences are always changing and new tools are coming out everyday.
My highest moments came when I launched the survey to over 90k audience, and also when I saw people on different teams referring to my report in their decks -- it’s such a sense of achievement ☺️. This semester I returned to Mailchimp as a product marketing intern, with the intention to understand what other functional teams are doing. During the first week, I got a chance to present the integration research findings to my squad and triggered some great discussions. I will be working closely with the integration partnerships team to create a "playbook" to streamline future campaign planning. Can't wait to see how I can incorporate some learnings from previous research to this project 🗝.
I owed my thanks to my intern manager Brooke who helped me set up milestones, always let me take my time to decide how I wanted to approach things, and always being supportive and accommodating 💛. Also huge thanks to the whole Market Insights Team -- Robin, Rachel, June, Mike, Elliot, Laurie, Christy, Chelsea, my intern parents, Jasmine & Mitch, for your mentorship and encouragement. You helped me grow tremendously 💜.