Litmus

Litmus is an experimentation platform designed for Gojek to try out new ideas and validate hypotheses. It helps the organisation in making data-informed decisions.
This case study is about how we evolved a product by taking a user-centric approach.

Background

In early 2019, a decision was made to build an in house experimentation platform so that people at Gojek can scientifically validate what works and what doesn’t. I got to know about this product on a fine morning when I was going through my emails and it was there in one of the product updates.

You are not going to believe me but bam! 2 years later I was asked to lead a project and it was Litmus. So, this story starts from there 🙈

I still remember Ravi Suhag (VP of Data Engineering) briefing me about the project. In short, the product was in really bad shape from an adoption perspective and he wanted the design to help shape the next 1-year vision for Litmus.

One thing was very clear from that conversation, many sleepless nights were knocking and we opened the door.

Knowing the unkowns

Without the pre-existing insights, we were in dangerous waters without any idea of in what direction we should take the product and where to start.

That’s when we decided to plan a research exercise to understand users existing way of working, challenges, their expectations from an experimentation platform.

We were hoping to get the answer to our biggest question -

“Why is this product at the edge of going down?”

Finding the right people and asking the right questions is what leads to the real insights and for the next 3-4 weeks we interviewed 10-15 users across Product Managers, Engineers & Data Analysts.

Vision - defining the road ahead

After spending one month on the discovery phase, we had lots of information on our plate and it was time to assemble the avengers.

We opened the great gates of Valhalla and invited the whole team to feast on the insights and to come up with strategies to butcher the problems we had in front of us.

After the exchange of diverse thoughts over a 2-hour zoom call, with a common understanding that the product needs a course correction, we come up with a plan to fight the odds over the next 1 year period.

To achieve this, we framed a few goals to guide us through this journey.

• The product should be efficient.
• The product should be inclusive.
• The product should be progressive.
• The product should be simple & useful.

While writing this I can still feel the energy we had while stepping out of that call, it was like we were going on a Viking raid with Ragnar Lothbrok.

Making it efficient

The most important resource to our users & business was time. It was a time of the pandemic and saving man-hour was definitely helping the company to reduce the cost.

With a mindset to optimize our platform, we started with our very first feature.

Easy Access

Can you imagine having a chat with the customer service desk before you can even start using a product, that’s a horrible experience and that’s exactly how our product was. Users have to ask the service desk to add them to the VPN list and then every time they have to open google authenticator, connect with VPN before they can see the first screen of Litmus.

Just by talking about this experience is making me feel frustrated, that’s why the very first thing we did was to remove VPN dependency and introduced sign in with google.

Access Management

Running an experiment is a team effort & require multiple stakeholders to play their part, but Litmus only allows the experiment creator to modify or take any actions on it.

So, users started raising tickets to the service desk for such tasks & had to wait for it to resolve before they could do anything else on the experiment.

This was creating an unnecessary workload on the service desk also. It was important to give users control over their experiment, for that we designed an access management feature on Litmus.

Users can decide who can have access to their resources and what kind of actions they can perform, which gives them control and confidence.

Making it inclusive

Good design keeps each individual in mind and makes sure to design a solution that serves everyone. Our insights had given clear points of exclusivity in our system and that’s exactly where we shifted our focus next.

Sample Size Calculator

Before starting any experiment you need to know how many people experiment need to meet the desired statistical constraints. Users had their own workaround to figure that out.

This has to be centralised so that for every experiment we can confidently say sample size was calculated with a trusted method. We designed a feature for users to do that on Litmus itself.

Experiment Reach

Only knowing the required sample size was not enough, once the experiment starts they need to track the reach of that experiment.

PMs or analysts regularly query the data and analyse it, that’s a repetitive and time-consuming task.

We aimed to automate this so that users can have visibility on experiment reach in real-time.

Making it progressive

The most crucial part of experimentation is the analysis and figuring out its effect. This part of the journey has huge potential of impact and waiting to be explored.

We knew working on a post-experiment use case will require huge effort but it was capable of bringing that change and helping us evolve experimentation at the organisation.

To make our product powerful and inspiring we worked on two major features.

Experiment Analysis

Once an experiment is finished, the product team request data analysts to analyse and create a report to know if they can make a decision statistically for their experiment.

This whole process of raising tickets, analysis and reporting take 6-10 days.

Imagine if we are able to scale experiments created at Gojek because that’s our goal, it will increase the data analyst workload also and that reporting time can go from 10 to 20 to 30 to unbearable workload in no time.

It was clear this can not be done manually and needed a solution before this problem hit us in the face and that’s exactly what we did.

Metric Performance

Any experiment requires analysis of multiple metrics, how they are behaving and affecting business or experiment hypothesis. This was another piece that was done manually with analysts dependency.

We designed a solution to automate this so that users can have a real-time view of the metrics performance.

Making it simple & useful

A product should satisfy users needs and function in a way users expect it to. It should help users in achieving their goals and eliminate all unnecessary elements from the system. It should be self-explanatory and at the same time aesthetically pleasing.

Segmentation Rules

It’s very important that the product is obvious in nature and users can easily perform tasks without any anxiety or friction.

Defining segmentation rules in Litmus was a real pain, they have to go through the documentation to learn the syntax and what all they add in there.

To make it even worse our system didn’t even provide error validation.

We simplify the experience so that this burden can be removed from users shoulders and the system can take care of it.

Impact

We noticed positive results with the above solutions in terms of quantitative and qualitative data. We started getting lots of feedback & feature requests from product groups to help them with running experiments.

• Increased CSAT score by 30% & NPS by 20%
• Saved 40k hours on experiment analysis in 6 months.
• Monthly Active Experiment increased by 32%.