For the past few months I've been working on a very interesting project at Wattpad around improving our discovery experience. It's been a great challenge to tackle and there have been so many great learnings that I thought I'd share with everyone in the case study below.
Challenge / Opportunity
Content discovery has always been a major challenge for large content networks. The challenge only gets larger when you consider that Wattpad is a network of user generated content with over 150 million story uploads and millions more being added every week. In a sea of stories about everything from dating the bad boy to sweeping travel journals to serial killer mysteries - how does a user find stories that are personally interesting to them?
The answer is usually more straightforward when there is intent. Somebody who knows what they're looking for can simply search for it. Oftentimes though, people don't know what they are looking for exactly - it's hard to know when there are so many potential options out there that they don't even know about. So often people want to browse a variety of things that could be interesting to them and pick something out.
For Wattpad this browsing experience is a critical element of helping people find what to read. It's so important that we knew it had to be incorporated into our home experience - the first place users land when they log in. For Facebook it's the feed, for Reddit it's the front page, for YouTube it's rows of video recommendations. For each of these platforms there is a vast amount of content and a social network of users. For each of these platforms, the home experience is all about you engaging with content by letting you browse a series of options, all of which have been deemed likely to interest you personally.
Wattpad has experimented with a variety of home experiences, from a newsfeed that focuses on your Wattpad social interactions, your 'library' of content that serves as a collection of stories you plan to read, and previews of popular stories in set categories, such as romance or science fiction. For all these options, we knew we could do a better job to help people find stories that mattered to them.
We began to rethink our home experience from the ground up, with the goal of making it a personalized experience that helped our users find stories relevant to them. We knew that in doing so we could improve user retention overall, thus driving growth.
This new home experience is the beginning of a machine learning system that evolves and learns based on each individual's actions. We wanted to create a series of modules that present different types of content recommendations based on different signals. Some modules could be based on which stories you voted on, some could show reading lists by profiles you follow, some could be based on what you'd read in the past. We knew that for different users, each of these may be more or less helpful, so we knew we'd have to build a scoring system that would allow us to tailor these modules based on each person's interactions with them.
Picking those initial modules was challenging. There are hundreds of potential ways we could show various types of content based on your behavior, or what's hot on Wattpad. We needed to start with something manageable that would meet the needs of the majority of our users. It would have to be a balance.There would need to be modules that were personalized as well as ones that were generic. The personalized ones would be very powerful in delivering the most relevancy, but the generic ones are critical for new users, for whom we have no information about.
New users for whom we had no information wouldn't get modules based on their reading behaviour or who they followed - but we could show them popular content on Wattpad. We also wanted to make sure that while we showed them popular content, we also showed them content that was diverse and representative of different types of stories so they could pick from a broader set of options (especially since we didn't know anything about what they were interested in). Exposing diverse content is a data challenge that we were very excited to tackle - and we were able to do that with sets of trending tags, profiles, and reading lists. These modules would also provide an opportunity for existing users who might typically only see a narrow range of content, to be exposed to other areas, as well.
Personalized modules can also be broken down into different types based on the signals we get from the user. We had two types of signals to work with: implicit and explicit. Implicit signals come from what we observe about our users, e.g. what they are reading. Explicit signals allow users to express overt interest in an author or topic, e.g. they follow R.L. Stine or a ChickLit Club. For any given user, the importance of each signal may vary. The more somebody uses Wattpad, the more information we collect (both implicit and explicit) and the better this home experience can become.
So our initial set of modules ended up being a balance between trending content we can show all users (including new ones), as well as personalized content based on both implicit and explicit actions of the user.
For more information on the engineering challenges of building this experience, check out this engineering case study here.
Testing the Waters
With such a big change to such a critical part of an experience that millions of users see every day, we had to be careful. We couldn't just launch it and hope for the best. We often A/B test small changes in our product to see impact on various factors including retention. In this case, we wanted to test a major change across all our platforms to make sure it didn't have any negative impact, and hopefully to prove that were were on the right track with our new home experience. We also planned to look at the results in a variety of ways - by age group, by existing vs. new users, and by platform (iOS, Android, Web) to understand exactly what the impact on retention would be for a variety of user types. We ran experiments initially with a small portion of users getting the new home experience.
In addition, we were careful to look at both quantitative and qualitative impact. In the past, when we've looked exclusively at qualitative results, we've sometimes missed that while users may tell us one thing, their behaviours reflect something else. And when we've focused exclusively on measuring behaviours quantitatively, we've missed sentiments that can create unanticipated changes in our community.
To assess the sentiment of our new home, we rolled the experience out to our various beta groups, and surveyed them for feedback. We also have a network of ambassadors at Wattpad (passionate users who help moderate the community) - great sounding boards whom we involved early on. This gave us a good sense of how users (particularly our passionate and vocal users) would respond ahead of time.
Results and Learnings
The results of the A/B experiment were incredibly positive. We found that retention increased overall by about 4%. The size of the retention increase varied depending on how we looked at it (new users vs. existing users, users under 25 vs. users over 25, and so on) but every single variation saw a sizable increase in retention. We also saw great validation in our qualitative feedback, with high satisfaction scores. 76% found it easy to do what they intended on the page, which was mostly about finding great stories and being able to engage quickly.
Given this validation, we were able to continue to hone the experience, and launch to all our users with confidence.
But we're not done yet! We will continue to iterate on this home experience, always making it easier for users to find great content, and making sure Wattpad can move the dial on retention each time.
Through this experience, we continue to learn more about how to help our users discover stories they'll love. We now know that the key lies in combining the value of curators in our network, with data-driven recommendations. Finding relevant content is increasingly challenging in a world with so much of it. The power Wattpad has to help make that easier for people lies both in its network and its highly personalized data. Finding the right balance between the too makes the engine that is our new home experience all the more powerful.
YOU ARE READING
Tales from a Product ManagerNon-Fiction
Ever wonder what it's like to be a product manager at Wattpad? Product management is an art and science and is ultimately a balancing act to ensure that we are building the right product to solve our users needs. In this story, I share my experienc...