This blog has been in the works for over 2 months. Want to know why?

Procrastination. Ah, an old friend 😌

Whenever I sat down to finish this blog, I invariably switched to Instagram and started watching reels.

No matter how many times I refreshed, I had new content to consume.

Similar to Instagram and I, our entrepreneurs are a constant fixture on our app. They log on to our app, place orders for others, add commissions and take e-commerce to folks who have never shopped online before.

Our entrepreneurs have never run out of products to recommend to customers. This is because of an array of products we offer.

'Fresh feeds' are crucial for a range of products, and we ensure there’s always something to sell. Our data science system empowers personalised feed for more than 100 million users on the platform 💪

How do we do it? Say hello to Unseen 👋

Humble beginnings 🐣

An important learning from User Research: on an average, an entrepreneur opens our app and browses for around 20 seconds. This happens multiple times a day.

The idea was to collect certain types of interaction that users had on the platform and then deprioritise these catalogues from the freshly computed feed.

How does Unseen work? 🤔

We have a complex ML system to generate personalised feed for all our users. However, we also need to ensure that we are not showing the same catalogue repeatedly to entrepreneurs on our platform.

Our ML models assign a score to each of the catalogue recommended for individual users.

To ensure feed freshness, we used to deprioritise some of the catalogues from the personalised feed for a respective user. Through this, we reduce the weightage of each catalogue based on certain types of interactions the user had earlier for the respective catalogue.

Why did we choose to deprioritise a catalogue, rather than removing it completely from the user's feed?

A complete removal of these catalogues would have created a vacuum. This would then be filled with some new catalogue.

Now since these catalogues were new, we didn’t have enough data to quantify the quality of these catalogues. This was a trade-off our Data Science team took to balance the quality and the freshness of our personalised feed.

It performed well — until we had sale days.

Technical snags 🤕

During normal days, Unseen worked flawlessly.

However, we found that Unseen wasn’t as effective during sale days! 😪

We ran experiments to observe its efficacy.

One explanation for this was that during sale days, buyers are focused on the most popular products on the platform — say, a highly rated t-shirt. They don't want to explore items; they want a quality product.

In their case, Unseen was actually deprioritising these popular products, since a buyer would have come across them due to our algorithm already.

Another accidental discovery occurred because of our constant innovations/ Between 2019 and 2022, we've launched a plethora of features on our platform.

These features — although targeted for own specific use case — indirectly seemed to solve the problem of providing a fresh feed. Some of these were:

  • Personalised Advertisements: It promotes certain catalogues to the users based on their interest.
  • CT-QCT: We pass newly launched catalogues through a series of quality checks and then add these to our feed. This adds a certain degree of novelty of our feeds. You can read more about this in Abhay Shukla's blog on Catalogue Discovery.
  • Non-personalised organic slot: Some catalogues with high rating and good feedback get added organically to feeds. This boosts the performance of great products on the platform, as well as draws attention from entrepreneurs.

Similarly, we're constantly working on hooks to keep our feed updated with the latest products.

At the same time, aggregating these user interactions at scale gradually became an operationally challenging task.

We realised that we couldn’t run Unseen in the long run due to the sheer amount of data collected.

We solved some of these issues instance-by-instance. But these were band-aid solutions and we had to figure out a long-term way to solve this problem of feed freshness.

The core of engineering is to automate a task. We wanted to set it and forget it.

You can see where this is going…

Survival test🧪

After running some A/B testing, we realised that there were minor differences with or without Unseen.

Microscopically, we saw a true uplift in some of our core business metrics.

The writing was on the wall.

Drawing conclusions🧑‍🏫

With a heavy heart, the Data Science team had to draw the curtains on Unseen.

We didn’t notice a huge variation with Unseen. Taking into account the resources it took to keep it working, we decided to shutter down the entire operation.

Unseen was enabled for 2 years, and was a constant source of attention from our Data Science team — both good and bad.

Here’s to you, Unseen. Have a grand old retirement!

Huge thanks to Abhijeet Raj and Rama Badrinath for helping out with the information needed for this blog.

If you want to work on a rapidly evolving landscape with the best Data Science talent in India, head to our careers page to check out our openings!