Meesho’s engineering efforts to create a single interface for all analytics requirements.

By Arjit Agarwal, Pratik Kumar, Joel Fernandes, Vibin Reddy, Sweekriti Dhawan and Ashok Lingadurai

Meesho is built on the premise of analyzing and understanding consumer expectations and satisfaction. By adhering to this principle, we continuously improve our product (app).

Always start with the user.

Our core value, “User First,” requires all teams to take the user’s psychological profile into account when designing new features.

In recent years, Meesho has grown exponentially, with millions of users regularly interacting with it. Every day, millions of interactions (or events) take place, which gives us the opportunity to better analyse user behavior and optimize their shopping experiences on our mobile app.

Capturing user interactions poses a variety of challenges. To help us overcome these challenges, we built our own in-house analytics tool called Meshlytics.🤩

What is Meshlytics?

Meshlytics is a single interface for all your analytics needs. It is a pure Kotlin library that allows your app to interact with all of your analytics platforms (like Mixpanel or Clevertap) through one single interface.

Fig 1:- Meshlytics Explained

Sounds simple! But Meshlytics is quite powerful and is packed with many features:

  • It is a pure Kotlin library with kotlin-multiplatform, and you can use the same library on Android, iOS, and the web.
  • It makes it super easy to integrate a new analytics platform or remove an existing analytics platform.
  • It allows you to define platform-specific events.
  • It makes onboarding existing events to a new platform very easy.
  • It allows you to define a regex for your keys and validate if events and property names follow the regex.
  • It allows you to have different naming patterns for different platforms without worrying about conversion while sending the event.

Why did we build Meshlytics?

As we were integrating new analytics platforms in our application, we faced various problems that made our development cycle more challenging and painful. For example, when we decided to integrate Clevertap and send an existing event App Open to it, we had to rewrite the same repetitive code that we had written for Mixpanel earlier. This problem was more serious when the new platform followed a different naming convention like Snake Case (app_open )

Let’s look at a code comparison for the following task: App Open event being sent to 2 different analytics platforms which follows different key conventions.

Left: Code before using Meshlytics | Right: Same code revamped using Meshlytics library

One the older code (left), if we decide to send the App Openevent to a new platform we have to write the same repetitive code once again. In short, we were facing four major problems:

  • Code pollution.
  • Difficulty in integrating a new platform or removing an existing platform.
  • Slow development cycle to onboard existing events to a new platform.
  • Erroneously sending different data to different platforms

Hence, we decided to create an aggregating layer that solves all the above problems and provides us a neat and extensible way to solve our analytics challenges. The idea came into realisation as Meshlytics.

In the above code snippet (right), the App Open event is still being sent to the same three platforms using the same key conventions as described earlier. Clearly, the code looks neater now, and no code addition is required here even if you decide to send this event to a new platform in the future.

How did we build Meshlytics?

We formulated the idea of Meshlytics to solve several problems that were specific to Meesho. However, we wanted to extend this idea so that others can leverage this library in solving their product-specific use cases.

Hence we decided to build Meshlytics in iterations, with each iteration focussing on a subset of problems. Ultimately, we created a pure Kotlin library that is extensible and can be customised heavily to solve product-specific problems. Our first principle was to make sure:

If a problem is generic then it should be part of the library, otherwise, it should be solved through extensibility.

For example, when we allowed our users to experience the app without logging in, we decided to prefix certain events with Anonymous. We felt that it was not a generic requirement, and hence we did not provide any provision to prefix events in the library when the user is not logged in.

To sum up, Meshlytics went through the following iterations:

Iteration 1

In its first iteration, we studied various analytics platforms, selected a common API to support via Meshlytics, and exposed a single interface to the app to interact with Meshlytics.

  1. Since the actual platforms depended on the consumer, we created an abstract implementation for a platform. For every platform, you need to write a concrete implementation of Dispatcher and register it to AnalyticsManager.
  2. Dispatcher supports generic common API. Hence the platform-specific API can be exposed via concrete implementation of Dispatcher.
  3. For defining dispatchers (or platforms) on which an event is going to be sent, we created DispatcherResolver which is again abstract, and the library’s consumer must provide the concrete implementation. See concrete implementation here.

Dispatcher Resolver makes onboarding events to a new platform very easy. For example, the following snippet sends App Openevent to Mixpanel, Clevertap, and Meesho.

Suppose, if you’ve written a new dispatcher for FIREBASEand you want to push the same event to it. Change the above code to:

Automatically, the library will start sending this event to Firebase too.

Iteration 2

In the second iteration, we mainly focused on the naming strategy. We used the title case (App Open ) for naming events and properties related to Mixpanel & Clevertap; and snake case (app_open) while sending events to our own analytics platforms (denoted by MEESHO).

Earlier, we decided to support only the above two naming conventions. We felt that down the line, someone may require support for app-open or maybe weirder app*open formats too. So, we took inspiration from Glide and decided to use transformers known as KeyTransformer. A key transformer takes a string and outputs a string after applying the transformation.

The library itself provides specific concrete implementations of KeyTransformer like LowerCaseTransformer to convert the key into lower case. As with Glide, you can use a combination of KeyTransformer. For example, to convert App Opento app_open you can use LowerCaseTransformer and PatternTransformer like below:

Fig 3:- Converting an event key to lower-case using KeyTransformer

You can register these transformations with the concrete implementation of Dispatcherand these will be applied automatically before sending events to the associated platform.

Iteration 3

In the third phase, we built validation over event data. Our default naming convention for events are title cases containing only alphanumeric characters, but during development, we didn’t follow this pattern strictly and ended up using strings like Button Clickedor Video Play/Pause Clicked.Although this works just fine, special characters might be a problem while building data pipelines, and it is recommended to follow a naming convention strictly.

So, we created a mechanism to validate incoming data. First, you need to register an acceptable regex with AnalyticsManager. Incoming keys are expected to honour this convention. For example,

You can try to experiment here, and check if your key name is valid for the above regex.

All the incoming data is checked against this regex, and you can configure the validation level as NONE, LOG_ERROR and ABORT as per your requirement. Since validation may slow down the actual process of sending the events, we recommend using LOG_ERROR in the debug variant and NONE in production. Now, even if you’ve accidentally broken the naming convention, the library will report an error, and you can fix it during development.

Our Learnings During the Implementation

Despite starting with a strong design foundation, we did face many challenges while implementing; so we kept revisiting the design again and again to make modifications wherever necessary.

Transforming Keys

Initially, we decided to support only two naming conventions; title case and snake case. However, over time we needed other transformations for reserved properties (for example, mixpanel accepts email under $email key) and requirement to add Anonymous prefix with certain events if the user is not logged into the app. Since these requirements were unexpected, this led to fragmentation in transformations and the Dispatcher API started to look more complex.

We solved this problem with KeyTransformer and now we can add any new transformation just by adding a transformer, and we don’t need to worry about where the Dispatcher actually uses it.

Dispatcher Resolution

Earlier, we used String as dispatcher id and so DispatcherResolver maintained a list of String for each event name to store dispatcher information. This was consuming a decent amount of memory. We reduced it to less than half, by making the dispatcher id as Byte. Since there is just one set bit in each dispatcher id (power of 2), we can use the bitwise ORoperation to store the same dispatcher information in a single Byte.

Left:- Dispatcher Id is String, and so we need to use a list of strings for each event in DispatcherResolverImpl | Right:- We use Byte as Dispatcher Id, and as we can see, each id is a power of 2. So we can use the bitwise or and store the same dispatcher information in a single Byte.

Optimising Dispatcher Resolution

At first, we resolved dispatchers by doing bitwise ANDwith each registered dispatcher (and sending events only if the bitwise ANDof resolved byte and dispatcher id is non-zero). Though shouldn’t be a problem, if you are using 64 dispatchers and sending most of the events only to 3 platforms. However, now we do it only for the set bits in resolved Byte. So, if an event is meant to be sent only to 3 platforms, its resolved Byte will have only 3 set bits. While you may have 64 registered dispatchers, the library will send the event to those 3 dispatchers without making any comparisons.

We made good use of bit-magic to do so.

Several rounds of learnings and iterations made made Meshlytics one of our super complex projects that required days of brainstorming andcollaboration. Considering the fact that social e-commerce is on the rise, our engineering projects like Meshlytics will serve as a backbone of Meesho in the upcoming months (even years !)

Interested in working with Meesho’s Android Engineering Team and changing the way India shops online? Take a look at our careers page for opportunities.