Metrics for Startups: Tools & How-to (part 2)

January 2020 · 9 minute read

In the first post of this series, I tried to make the case for growth accounting: what it is, its most important metrics, and why startup people should get acquainted with it.

You can find it here, but the gist of the matter is the following:

  1. Growth accounting comes down to measuring product intent, product habit and product behaviour
  2. This means tracking a few key metrics: growth, average activity and events
  3. Growth accounting can help founders and investors alike, since its value lays in two things: the knowledge these metrics bring and ability to act upon them

It’s all very nice and well, but I see how people who never heard of such stuff can get a little bit lost. We’re lucky though: it became easier than ever to measure, understand and use these KPIs. It’s no longer a business for only nerds.

In-house vs outsourced

First, a few words must be spent on the trade-off between building analytics tools in-house and using outsourced products.

My general rule of thumb is the earlier you are, the more you should use ready-to-use products. Early-stage startups typically don’t have the skills to develop a reliable analytics system, nor the time to develop it, let alone the money to maintain it.

(For those interested, I’ve discussed the topic at length in another essay.)

Sometimes, however, you just have to build the whole thing by yourself, since you need more control than what you get from a standardized platform. This is the case when:

  1. Part of your product is hardware
    Most of the analytics product out there don’t provide REST APIs, so you won’t be covered
  2. Mobile and web carry share the same importance
    Data consolidation between web and mobile is tricky with pre-cooked tools, as you have to provide a way to identify users regardless of the platform they’re using
  3. You want to consolidate walled gardens
    It’s the case of ads-money spent (from Facebook Business Manager) in relation with user segments

Other good reasons for going in-house are financials (”we don’t want to pay for something we can make ourselves”) or legal (”we don’t want / we can’t share usage information with a third party”).

As a side note, internal data solutions go by the name of “ETL pipelines” (extract-load-transform).

Tracking engagement with Google (Firebase) Analytics

The most important tool in the analytics world is Google Analytics (”GA”), that comes under the umbrella of Firebase when used in complex products.

From what I’ve seen, Google tried to consolidate Google Analytics and Firebase Analytics under the same umbrella. They haven’t been able to pull it completely off though, and the bottom line is they’re still somewhat distinct products:

For such reason, I’m quite convinced that Firebase is a far better choice than Google Analytics for most startups.

Including Firebase in an app is trivial: just include the SDK and call it’s init method when your app is instantiated. After some time, numbers just start popping up by themselves.

Right in the dashboard tab, Firebase tells you about your active users, MAU, DAU e WAU:

Active Users in Firebase

Active Users in Firebase

In the Retention tab (Users in Google Analytics) you’ll find the user cohorts that can help you understand how much your product is getting used in a given point in time:

Retention Users in Firebase

Weekly retention in Firebase

Each row is a week. Each cell is how many users that you acquired in that specific week got back after a given period (say, a week). In the example above, in the third week of December that company acquired 4,000 users, 21% of whom used it again at least once 7 days later.

Estimating churn is tricky, because you have to decide a point in time after which you deem a user as lost. The retention tab in Firebase can help you with that: if you choose a different time scale in the filter options, you can see things “from the above” and understand how many users you lost after a few months:

Monthly retention in Firebase

Monthly retention in Firebase

In this case, out of those 2700 users that discovered the product in November, 7% used it again in January — 2 months later. We can therefore estimate the 60-days churn-rate at 93%.

Events

Both GA and Firebase have a dedicated section for events, where you can find all the events that you are tracking, how many times it has fired in a given time window, and how much it changed in % given the previous period.

It can be quite helpful to use this part of Firebase Analytics to understand user behaviour and power-users ratios — in particular for driving the product strategy. For instance, let’s take a look at this example:

Events Management in Firebase

Events Management in Firebase

In the last 28 days, there have been 12K pictures posted by ~1K users. 100K likes have been cast, a +50% increase.

What this story is telling me is that there’s a typical 1-99 ratio between creators (“posted a picture”) and consumers (“liked”). Also, it tells me that this product got really better at making people liking stuff (+50%), but only for those who were already liking stuff before — since they only slightly changed (+8%).

Therefore, if I want to have more people like pictures in this product, I should consider targeting those who never discovered the feature in the first place.

Acquisition & Attribution

In the web things are easy because we have query-string parameters. Almost all URLs have them: they come after a ?, they are in the form of key=value, separated by a &, and they are a pretty neat way to communicate between web-pages.

A web page alice.com can have a link embedded in it like bob.com/?who_called=alice, and Bob will know that the user came from Alice. Easy.

Google Analytics, specifically, uses a set of standard query-string parameters called UTM: utm_campaign, utm_source and utm_medium. If you include them in any link, they’ll be automatically tracked inside GA.

So, for instance, let’s say that you are paying an Instagram influencer to promote your website. You want her followers to go to cool-website.com/landing-page. She just has to include a link in her bio like cool-website.com/landing-page?utm_campaign=product-launch&utm_source=ig-model-name&utm_medium=influencer, and boom, done:

Google Analytics UTM

Google Analytics UTM

There you go, easy acquisition channel split.

In the mobile-first world we’re living in, unfortunately, things are not so simple. There’s no such thing as query-string parameters via which you can communicate between apps. And thus, you have to… adapt:

Attribution, a mobile nightmare

Attribution is pure hell in mobile. Where people are coming from is both a critical piece of the growth-accounting puzzle, as well as a nightmare to do if you have an app.

There are three sub-optimal options: 1. Pay for extremely expensive premium tools 2. Rely on a limited set of data, difficult to consolidate 3. Build your own ETL attribution system

1. Paid option: Adjust, Branch

There are companies out there, like Adjust and Branch, who are selling products that solve the mobile attribution.

It’s rather simple: you embed their SDK inside your app, and they’re able to tell you where users are coming from. In particular from Ad network (data you wouldn’t be able to get in any other way).

Adjust Dashboard Attribution

Adjust Dashboard Attribution

They even give you the channel split for user cohorts, so that you’re able to tell which channel is bringing the best users:

Adjust Dashboard Cohorts

Adjust Dashboard Attribution

Problem is that these tools are bloody expensive. Adjust comes at a starting price of 100€/m, while Branch is free for up to 50K MAU and then jumps straight to ~0.01€ per MAU (big money, trust me).

They’re both OUTSTANDING platforms, and I had the chance to use both. But I’m not really sure they’re worth that money if you just started out.

2. App Stores data

Both the Android Play Store and the Apple App Store have sections dedicated to acquisition and attribution.

The Play Store is very strong in this regard. It supports the UTM standard, it gives a split of all the most meaningful keywords that drove impressions and install, and you can potentially use your app name to infer what was word of mouth.

Play Store Acquisition Overview

Play Store Acquisition Overview

There are three problems with using only this approach though:

  1. Cool data is not there for iOS
  2. It neglects what doesn’t pass through the search field of the store: you’re not able to understand if a user came from an Instagram ad or a Twitter one
  3. It’s scattered data, very difficult to consolidate because both stores don’t provide APIs to programmatically fetch it

I’d still suggest keeping an eye on these views in any case. They can tell a lot.

3. Go DIY

The third approach is to do it yourself.

Firebase, among the plethora of services present in its portfolio, offers a tool called Dynamic Links.

It’s a very dev-heavy tool that lets you create your own system for sending links to your app. It works on multiple platforms, and whether or not the app is already installed.

This ultimately means that you can effectively create your own attribution system. Architecturally speaking, it’s easy:

  1. You create your own domain, say share.example.com
  2. Your domain created the Firebase Dynamic Link, embedding a source keyword in it; say: share.example.com/?source=influencer
  3. Firebase takes care of telling your app, after being installed, that someone invited someone else via that URL; you count +1 on your internal database on that keyword

That’s what I did in Uniwhere for more than a year. There are quite some problems though: - it costs a lot to develop and maintain - it breaks a lot (let me stress again: mobile links are a nightmare) - you still cannot track paid installs.

On this last point: Firebase will tell you that you can, but you can’t. Facebook offers the chance to embed a deep link in your ads, but it works only on Android. All the other ad network I tried (Google Ads, Twitter Ads, Snapchat Ads) don’t even have the option.

Long story short, going DIY you’ll end counting paid installs as organic. If you don’t do many ads, that’s ok, but if you do, just pay Adjust.

Recap

TL;DR version:

  1. Use Google Analytics if you have a publishing business/e-commerce running on the web
  2. For everything else: Firebase
  3. All the analytics solutions out there show basic engagement metrics: MAU/WAU/DAU, retention curves, churn rates
  4. Attribution on the web is easy: just use query-strings
  5. Attribution on mobile is hell: you have to choose the lesser evil; if you’re an ad-heavy company, just stick to Adjust

Closing notable mentions: Facebook Analytics is an actually pretty good and pretty solid product, but very very data-hungry (be very mindful of what you’re sharing with them and be sure of including it in your privacy policy)

Follow up

Thank you for reading. If you have any questions, feel free to shoot me an email at blog@giansegato.com or tweet them @giansegato.

I've a non-lame 0-bullshit newsletter on startups, tech-enabled scalability and data-driven impact. Sign up to stay up to date and to keep this conversation open