Today's question comes from Benedict:
Do you track events in Buttondown and how do those feed back into product development? How have you changed what you track over time, and how you interpret it?
— Benedict Fritz (@benedictfritz) February 18, 2023
This is a great question!
Let me start by outlining my mental model on data-driven decision making in a technical organization:
Collecting, maintaining, and analyzing data in a rigorous and epistemologically honest manner is a way of exchanging resources (time, sometimes money) for visibility.
And like any trade, sometimes the deal is worth it; sometimes it is not.
You may not be surprised to learn that, for instance, I don't find myself in need of much visibility when it comes to feature development or prioritizing between various product-level efforts. This is for a number of reasons:
User Personae.docx
floating around anywhere.) and there's a lot of prior art floating around to piggyback off of. All of which is to say there's no function or process in Buttondown that goes, roughly, potential_projects + data => critical_path
...
...but I still use data! Mostly in what I think of as boring ways. Like:
All of this feeds into visibility, and implicitly into my roadmap, but it's hard to point to specific circumstances where I explicitly made a decision based on the data and nothing else; it's more of a reinforcement, or a dossier to consult when trying to reconcile qualitative feedback ("FooCorp wrote in that her p95 performance on /v1/subscribers
has been slipping... is this an isolated issue or something systemic?") or brainstorming long-tail work (see last week on SEO.) But 95% of the time, it's vibes-based development. That's a conscious choice: it's easier to lean into the vibes if your roadmap deliberately minimizes surface area.
The point I want to end on: one way to improve the outcome of a trade is to give up less in the trade. If you're spending 20% of your resources maintaining your data posture, you better be getting a hell of a lot of value from it; conversely, if you're spending 2% of your resources on it, then the threshold for usefulness becomes much lower.
Whenever possible, I just use operational data. Buttondown has no analytics database, just a read-only replica; Buttondown's operational database has no data used exclusively for analytics:
I get to cheat a little bit here, from a philosophical perspective: not every app requires an "analytics" view, and therefore not every app gets to approach data engineering as an end-user experience question.
Thanks again Benedict, for asking!