A brief love letter to Logtail

I’m keeping this somewhat brief because I want to expand my thoughts into a longer and more cogent blog post, but I finally got myself into a place where I feel good about Buttondown’s logging. This was a combination of a few separate things:

1. I finished onboarding to structlog, which makes every logline structured (in JSON for production use cases, but not locally since it’s very annoying to visually parse JSON)
2. I found a log sink that handles my use cases — Logtail, which lets me write SQL against my logs.

I’ve put off caring about logs for a while, since there aren’t that many times where I’ve thought to myself “boy, I wish I had better logs.” Most of Buttondown’s shenanigans that require deep introspection (missing subscribers, odd rendering quirks, and so on) can be handled at the database level.

I performed a bit of high-pressure grape surgery this week with exactly one production issue (a far cry less than I was worried about slash expecting) — all drafts are now emails. What that means: where there was once a “drafts” table for, well, drafts, and an “emails” table for things that were emails, there is now one single table with a status enum (draft, sent, in flight, et cetera) combining them.

And with that, one of the last poor architectural decisions of 2016 Buttondown has been erased.

This is one of those technical debt things that is probably hard to glean the value of from outside the codebase. It has become increasingly obvious over the past few years that “drafts” and “emails” should be the same thing:

• The interface for editing a sent-out email is extremely janky because the big, fancy writing interface can only really interact with drafts;
• a common desire is to be able to preview the true web archive of a draft before its published, which is impossible since that rendering pipeline only happens on emails and not drafts;
• adding undo support for publishing drafts was tricky because history had to be tracked across multiple tables;
• and so on.

A brief excursis on working

A little less about the what and more about the how — I’ve been slowly triangulating on what a really nice schedule for my day-to-day is going to be now that I’m shifting from “sabbatical mode” to working on Buttondown in earnest.

One of the things that has been a very surprising change to me is how difficult and tiring craftsmanship can be. If you’ve been reading these notes for a while, you likely remember me complaining often about how difficult it was to find time for the ‘serious’ work, since my schedule (and Buttondown’s growth) only permitted time for reactive work during the week.

That is no longer the case! This is great. Most mornings start out with an hour or so of operational and reactive work (emails, dashboards, you know the drill) and then I get to shift to actually use my creative energy. It is exhilarating to actually be able to spend the preponderance of my time improving Buttondown, rather than just keeping my head above water.

Multi-factor authentication

I joke often about my product prioritization heuristic for Buttondown being “work on the thing that I’m most embarrassed at not having.” This heuristic has led me to my work for the past week: support for multi-factor authentication (now live for Authenticator apps!)

This is long overdue (and perhaps a little accelerated by a recent enterprise sell whose conversation soured when I said it was merely ‘on the roadmap’.) It’s also been, frankly, pleasant to work on — I get a lot of decision fatigue when it comes to designing relatively complicated UX flows from whole cloth, and I don’t really have to worry about that here because:

1. we As A Society have pretty well narrowed in on what the right MFA registration & input flow should be, and leaning into that is a good thing;

2. Buttondown’s design system now has rigorous and easily extendible answers for “how should I display a multi-step modal?” and “how should I display a rich table of primitives”?

ProseMirror is neat!

I went down a bit of a rabbit hole this week, to nobody’s surprise.

Last year Buttondown swapped out a very old javascript package for fancy text editing (TinyMDE) for a much newer one (tiptap). This was one of the more fraught roll-outs in Buttondown’s existence, not for any explicit technical issues but because lots of people are understandably persnickety about core workflows changing. A valuable lesson learned!

The main reason I wanted to shift to tiptap was not out of any one specific use case but being able to go from an abandonware piece of software to something nice and powerful and extensible, so I could invest in a better writing experience down the line.

A bit more about the new marketing site

I mentioned last week that I am tooling around with a new marketing site. The reasons for this, in no particular order:

- getting to extricate the marketing site from the core application codebase

- better SEO (and outright performance)

I wrote a few months ago about a critique by Brian Lovin (the best thousand dollars I’ve spent on Buttondown by a country mile!) and how it was dovetailing with some of my efforts to refresh Buttondown’s overall app scaffolding. That work has been slow but ongoing, and I wanted to chat about it a bit. First, the screenshots:

• I started out with a pretty mobile-inspired concept of dual actions: core CTA on the bottom right, ancillary actions on the bottom left. I think pinning core actions makes sense, but I’m not sure this scales; I’ll likely need to have a generic “actions” button in the bottom left to keep things uncluttered.
• One thing that makes this work a little harder than I’d like is that Buttondown’s front-end is split between a bunch of legacy SCSS and Tailwind. I’d love to wave a magic wand and get rid of all the SCSS (and I’m trying to make sure any files I touch get SCSS-purged), but this is one of the perils of trying to design a relatively old (and in some places, crufty) app in the codebase as I go.
• The elevation of the header to a three-tier thing (core header, universal search, and hamburger menu) is small. I go back and forth on whether or not having the search bar as such a prominent affordance is useful; it’s tempting to just have an icon that pops up a modal, which would certainly make the design quieter.

There is a lot left to be done. I want to make the sidebar look — how should I phrase this? — less janky; I want to finally unshackle the app from its default full-width constraints; I want to make sure this all plays nicely on mobile viewports, which have been hitherto underdeveloped. But it’s closer to being shipped than it is to being started, so to speak, and I wouldn’t be surprised if I get it across the finish line by the end of the month.

Now that a full month has passed, I can actually take a look at how the new pricing plan is working as compared to the old one with some level of statistical rigor. I expected and/or hoped for three things:

1. A better conversion rate from “activated user” to “paying user”, because the pricing was clearer and anchored lower for a non-trivial portion of folks;
2. A slightly worse conversion rate from “prospect” to “registered user”, because the free tier’s efficacy was diminished;
3. A net improvement in overall conversion from “prospect” to “paying user”, meaning that point 1) would outweight point 2)

A couple caveats and points of elaboration:

Zombie branches

Sometimes you have zombie branches.

Zombie branches are little projects that are large enough to be difficult to completely set aside a day and ship but either important or approachable enough that you can chip away at, slowly, such that it never feels like you should just close the branch and call a sunk cost a sunk cost.

One of my goals this month is to try and get things as tabula rasa as I can for whatever a long-term schedule looks like in June. This of course means trying to cull the zombie branches, one way or another.

May 20, 2022

Two pieces of followup from last week, to kick things off:

First, repricing work is officially done. “Officially” meaning that I no longer have a working Things project for it; I’ve gated the last few pieces of functionality, and the stuff that I should do as fast-follows (lifecycle emails to prompt people to bump up for an annual plan; exit surveys for folks who churn; etc.) are all hiding away in the roadmap to be tackle independently. Yay! The numbers — which I’ll dig into after next week, as it’ll have been a full month which feels like adequate level of statistical significance — are better than even my positive-case scenario.

Second, I talked about trimming costs a little bit, and I’m running around three hundred dollars a month cheaper than I was a week ago. (This is a reminder to myself that I need to update my running costs page, which is a few months out of date.) This was fairly easy to do:

The big news

I led last week’s note with a teaser, and here’s the reveal: I’m working full-time on Buttondown.

Okay, “full-time” is not quite yet the case. I’m taking the month of May to relax and recharge and enjoy the trappings of a slightly more lackadaisical lifestyle for a bit.

But Buttondown is now the main thing. This is exciting! And long overdue.

(Sorry for missing last week! I promise you will learn why next week. It’s a pretty good reason.)

Repricing is freakin’ live

If you were to sign up for a Buttondown account this very second, you’ll be on the new pricing structure! It’s been live for about a week now, and I’m very proud that nothing has exploded. It’s too small a sample size to see what the numbers “actually” are in terms of conversion relative to the old pricing structure, but I’ll take pride in the fact that it’s non-zero.

In exactly six days, I’ve had one $79/mo plan, one $29/mo plan, and six $9/mo plans come through. I am most proud of that last bucket: as I’ve chatted about (and as I’ll make public in an upcoming blog post), the primary goal of this work was not to increase ARPU in of itself but to ‌decrease the relative burden of the free plan, and five of those six relatively small customers would have been free under the old pricing scheme. This makes me very, very happy, and points towards a much more sustainable usage pattern for new folks.

Repricing: it’s finally happening

After what feels like years of punting, the repricing work is finally going live in a non-trivial way! This week marks a legitimate milestone in the project: I have the first few production subscriptions to the new pricing plans. (This is kind of a cop out because I manually generated and assigned invoices for those folks as they have atypical payment structures, but it counts — it means the feature gating logic and all of that is working and active in production.)

I’m getting to the point where the end might not be in sight — I don’t have a firm burndown list of everything that needs to be done — but certainly seems much closer than it did. Big, amorphous projects kind of go through an explore/exploit phase — you have broad, unbounded work until you get to the point where you’ve mapped out the state space of the full project and then you burn it down. I’m not in that latter space yet but I think I’m pretty close: there’s going to be a long tail of marketing copy that I haven’t yet itemized and a lot of pretty boring plumbing to update, but if I was told I absolutely needed to get this all done by the end of the month I certainly could.

A little aside: when I told a friend about these changes and about how many authors would in fact be paying less (if you’re paying for Buttondown for Professionals and have 4,000 subscribers, under “old” pricing you’d pay around $54/mo and under “new” pricing you’d pay a flat $29/mo) and he warned me to run the numbers to make sure I wasn’t killing my ARPU. This was good advice, but the panic was unfounded: if I were to simply project the exact same userbase that Buttondown has today into the new pricing scheme, MRR would approximately double. (Of course, you can’t assume that all cows are spherical and all that, but my point is that the cohort economics are valid.) So that’s encouraging.

The case of the missing scheduler, part 3

It would be lovely (albeit unsatisfying) if this was an issue that evaporated on its own, like so many transient hardware-ish issues I’ve dealt with in the past. In the meantime, I’m just grateful I haven’t been woken up by any pages.

By leading off with that quote, you might be able to surmise the followup: absolutely no progress, and also no crashes. At this point my Spidey sense is leaning more towards “this was some esoteric hardware issue that I should have better alerting on but was ultimately exogenous to my code”.

Sigh.

The case of the missing scheduler, part 2

I was really hoping that I would have some sort of juicy insight or breakthrough to share here. I… do not.

The good news: it appears to be happening somewhat less frequently than last week. I can’t point to any reason why that might be the case: I’ve added some additional instrumentation and logging to the scheduler dyno, but haven’t actually changed anything. (To wit: the scheduler’s crashed exactly once in the past four days, as opposed to roughly once every eighteen hours last week.)

It would be lovely (albeit unsatisfying) if this was an issue that evaporated on its own, like so many transient hardware-ish issues I’ve dealt with in the past. In the meantime, I’m just grateful I haven’t been woken up by any pages.

March 23, 2022

The case of the crashing scheduler

This week’s tricky operational issue is one that is still a little in-progress. Buttondown runs on Heroku, and uses a bunch of dynos. Think of dynos as akin to containers: everything on Heroku uses the same application bundle and the same mess of code, but each dyno is meant to correspond with a specific function or executable. For instance, Buttondown’s got the following Procfile, which is Heroku’s nomenclature for defining dynos:

web: bin/web
worker: bin/worker
checkerworker: bin/checkerworker
scheduler: bin/scheduler

Hello after a month of chaos! I am writing for you for the first time from my new house in Richmond, Virginia. In the past 28 days since my last missive, lots of non-Buttondown things have happened: a cross-country move, an unpack (178 boxes in total, because of course we counted), a whole lot of Telemachus playing fetch (and gaining a solid two pounds thanks to his grandparents spoiling him), and Buttondown has — well, there’s been less forward progress and more staying afloat.

A whirlwind tour of the highlights (and lowlights):

• Overall, things were… pretty stable? Growth continues apace and I am much worse at inbox management than I’d like to be (it’s been a while since I’ve hit inbox zero, but I’m fairly steadily at inbox ~thirty). No major incidents besides…
• Heroku went down. That was annoying, even if I’m a little pleased that I found out before their status page was updated. Still, one hour of downtime in ninety days is not worth seriously considering accelerating my AWS migration plans.
• I had some Redis scaling issues. These were boring scaling issues: if roughly ~500,000 emails were queued up to send then my instance would cap out and deadlock and what do you know, that happened. This was solved trivially by bumping up my instance to the next size, but that’s only delaying the problem’s event horizon to the next order of magnitude. I’ve been fantasizing about migrating off of Redis and using postgres-based queuing (RQ, which I use as a queuing library, is not particularly wonderful even if the API is nice) and this accelerated things somewhat.
• February’s roadmap was ambitious, and… did not go as planned. There was a lot going on, and the snatches of time I had to work on Buttondown were spent furiously going through email and keeping the lights on. Despite that, I shipped one big thing that I haven’t quite blogged about yet (and will next week!) — an entirely new subscribe form. Given that this is an input that a cool ten thousand or so people convert through every day, the fact that I was able to roll out the new experience with no fanfare and a huge amount of uplift is tremendously exciting.
• March’s roadmap, on the other hand, is much more reasonable — two small features (already shipped!) and two bigger ones which are roll-forwards from February. The vast majority of the moving part of moving is behind me, but it’s still going to be a busy month: I’ve got two and a half trips lined up in the next three weeks.

Hello hello! This past week was a quiet one: I mentioned that I was headed to the cabin with some friends, and cabin time was unproductive — which is to say it was time well spent. This week has been productive, but it’s also been start-of-the-month-productive: there’s been a lot of invoices being sent for processes that I’ve been a little too lazy to automate, there have been tweet threads to write and various marketing efforts to kick off, that kind of thing. (Today is perhaps emblematic of the entire week: four hours spent, which is pretty high, but two of those on operations, one on marketing, and one on actual development efforts.)

Two ‘big’ things on my mind:

First, I was lucky enough to score some time with Brian Lovin before he wised up and raised his prices. If you are interested enough in Buttondown to read this ol’ newsletter every week, I couldn’t recommend enough reading through his design critique — I was furiously nodding along through the entire thing, and have a slew of things to fix (both big and small) thanks to what he surfaced.

One of the things that he really brought to light for me, at a bit of a meta level, is how lost I’ve gotten in the day-to-day of working on things. I probably spend.... 7%? Of my time thinking about Buttondown as a gestalt, since so much is firefighting and keeping the lights on — which I mean in a positive way, since Buttondown has enjoyed such unalloyed growth, but it still means reaction rather than action. In particular, I think he pushed me in a really good way for some of the design system-y things that I’ve been thinking about, and the long-ballyhooed scaffolding work I haven’t merged yet is… going to remain unmerged. It is (cheerfully, since I didn’t share it with Brian) very much in line with where he was going, but I think the right thing to do is to keep it under wraps and keep iterating on it a bit longer.

I wrote last week about three painful queries that I needed to futz with: exporting all email events, aggregating client statistics, and time-series aggregations.

This slightly more methodical approach worked pretty well! To start with the good stuff:

1. Exporting email events for a big newsletter now takes O(seconds-to-minutes) instead of O(minutes-to-hours). This in of itself was the single most urgent thing: I can stomach analytics not working for certain newsletters, but the idea of someone not being able to get their data out of Buttondown was anathema. ¹
2. Client-level aggregations are now live and the “inbox statistics pane” loads quite fast. This is one of those lovely performance changes that actually have product-level implications, too — it’s now feasible for me to pull out specific clients used by a subscriber and display them live in the subscriber detail page, whereas previously that would have been a pretty expensive call.

Time-series aggregation is still…pretty painful. If you’re a newsletter with, say, twenty thousand subscribers and a year’s worth of emails on Buttondown, I’m no longer timing out when you try and get your analytics for the past quarter (yay!) but I’m definitely still timing out when you try to get your analytics for the past year (boo!)

Buttondown had its first planned partial outage of the year. (Of 2021, too, if my memory isn’t failing me — but, you know, new year and all that.)

This was to upgrade Buttondown’s database: a measly but stalwart db.t2.medium running on RDS, first spun up in 2017 and humming along ever since. ¹ The general parlance when thinking ab out relational databases is “people tend to underestimate how strong and scaleable a small, boring Postgres box can be” and I think that is true: this cute lil $29/month instance has brought me very few complaints over the past few years.

The upgrade was necessitated by two factors:

First, the database (cleverly named buttondown) was running Postgres 9.6, which Amazon was planning on formally deprecating and forcing upgrades. They’d pushed back the required upgrade window twice already, but I did not love the idea of waking up at 4am to a slew of pages because AWS decided when to take the database down.

Welcome to the first Weeknote of 2022! I’m fairly excited about this year (as much as one can be excited about any year) — the slate of work feels fun and well-formed, I’ve got some exciting personal stuff happening (moving!), and for the first time in a while I was able to actually spend the last two weeks decompressing to some extent (a marked difference from December 2020, which was quite fraught and stressful.)

My slate of January work is much more meager than my Q4 roadmap, which was — let’s be honest — overly ambitious. Here it is, in total:

• Launch new scaffolding
• Blog about the new scaffolding
• Add checker dashboard
• Remove slug uniqueness assumption
• Allow changing subscriber addresses
• Add recovery flow for spammy subscribers
• Allow both banning and unsubbing

That new scaffolding is almost ready for prime-time, and there’s a 60% chance or so that by next week it’ll be live, which will be the single biggest change to Buttondown’s interface since…ever? I think?

(Apologies for the off weeks for the past two weeks! Travel, et cetera.)

Checker, revisited

As the year winds down and I try not to alarm myself with the lack of progress on my Q4 goals (only two out of the six items I wanted to ship have shipped!), I amuse myself by reading through older issues of this newsletter and admiring the delta.

One such item was the checker infrastructure that I built in early April. I wrote about this as if it was dessert work: a fun challenge with low ‘real’ ROI:

A belated note about public archives

In my excitement and/or delirium last week, I forgot to mention that I managed to carve out and ship one of my six Q4 goals — a nicer public archive list! This was absolutely the easiest one of the six in terms of scope (just messing around with HTML and CSS for a while) as well as the most fun (just messing around with HTML and CSS for a while).

The rebuild itself was not particularly noteworthy: the pre-existing design was , and so the bar was low, but I mostly just spent five hours tweaking things until they looked particularly nice. A couple more interesting points:

Good on my promises.

I vowed last week to push out the list view changes, and I did! They’re now live. There are going to be some hiccups in making sure the design scales out to the bevy of use cases, but no big surprises or all-hands-on-deck-emergency-commits; it was as painless as I can hope from a substantial refactor of that nature, which I’m pretty pleased about. There will undoubtedly be a long tail of tiny fixes and tweaks over the next month, but that’s good — all of those will apply to the actual design system, rather than one-off components.

This week was split in twain.

I spent the first half in Seattle, as I tend to do, plodding away on the ListView work that I’ve spent the past two weeks chatting about.

First, I suppose, the good news:

Email filtering, such as it is, is done. My goal with the item is complete! My DataTable component has expanded and I now have a component that takes a bunch of stuff, and the page at exists solely as a wrapper around the . (Plus I added filtering, the whole point of the exercise — and the most trivial one.) There are a couple things I need to do before merging, like better type information and fixing where the dropdown menu appears, but it is Done.

I started my first piece of the Q4 roadmap yesterday. It might not look like much yet, but I am very excited about the following screenshot:

No big news, since I spent most of this past week trying to clean up the slate for the Q3 -> Q4 transition. A smattering of updates this week, then:

• I got the big subscriber views transition over the wire this past weekend! Around 85% of Buttondown’s pageviews are going through the new, HTML-based views: that remaining 15% is attributable to folks on custom domains (which had a now-fixed bug) and folks with custom CSS (who I need to manually ask to audit their CSS and then migrate over.) I am going to probably be very negligent at driving down the latter camp to 0 so I can finally unship the old code.

I whined last week about having a fairly unstructured Q3 despite my lofty goals, and I think Q4 is going to (hopefully?) be a bit better. You can take a look at my (unpublished/unfinalized, but mostly all there) Q4 roadmap — repricing and then a whole bunch of small follow-up work.

It feels a little bit like cheating to shift my gaze from Q3 to Q4 when I've got a handful of things that aren't quite done yet. The subscriber-facing views are technically "live" now, but I need to start actually ramping up the flag and prompting folks to update.

Where have I been for the past two months, you ask?

I have been active, I promise! I’ve been around!

I write a lot about the “good type of busy” and the “bad type of busy”. This was mostly the good type of busy: lots of new folks to onboard, lots of fires to put out.

It is funny, thinking about fires: I was reminded, cleaning up a part of the codebase from 2017, about the sheer I had the first time I had someone reliably sending out 10,000-subscriber emails on a weekly cadence. Some of the terror was, frankly, rational: one bad send with a few hours’ delay and I would have terrible blood pressure for a week. Email is tricky: it is temporal, it is irrevocable, and it is a scary thing to screw up.

Hello from the East Coast! I’ll be here for the next four weeks, which will be an interesting stress test on how productive I can be without my routines & contexts (to say nothing of the whole 34” curved monitor and desk setup.)

A week later, and I have written… well, I’ve written some content for the docs site. I have made good progress in general on the site, and I think it’s about ready to launch even if it’s not ready to call done — all of Notion’s content has been ported over, I’ve added the last bits of scaffolding such as a table of contents and stronger search metadata (as discussed last week), and it’s in a good place to just start iterating on.

I’m working on the hairiest part, technically speaking, of the documentation site — which is search functionality! This is slightly trendy at the moment — all the coolest kids are adding FTS to their docs — but is also tremendously useful (and replicates one of the really nice things about Notion hosting).

There are… a lot of options in terms of powering client-side search.

• . There’s something to be said for home-rolled search functionality; hell, that’s what I’m doing in Buttondown proper, though that’s largely because it’s a split between client side search (for paths) and server side search (for per-user objects). Still, the killer thing here is full-text search (since the documentation is going to be relatively long), and I think it’s much harder to do that than tokenized search.

It is OKR season at my day job and so it is OKR season in Buttondown as well. I think I am actually making a pretty good go of it, this time around:

Two meaty projects (maybe a month each?), two smaller (maybe one-to-two-weeks each?) projects.

Anyway, my goal for this week: pushing out some customer portal changes! My goal for the end of the week is to allow all premium subscribers to pause, upgrade, and downgrade their subscriptions: mostly this means a lot of transactional emails and web hook unit tests left to write.

I did it!

And, uh, not too much else. It was a week of emails (things are still fairly downbeat, but it happened to be a particularly voluminous week, the kind where the raw count is static but the “amount of thinking per email” spiked up), setting up a new laptop (I love my Mac mini to death, I really do, but the M1 MacBook Air feels almost comically performant in comparison. I bought it on a lark knowing that it was hard to justify, and yet I think I’ll be spending more time coding on it than on my Mac mini!), getting some nice press (I landed a fairly meaty mention in the , which is the kind of thing that I don’t know impacts me in a serious way besides SEO juice but certainly feels good!).

This coming week, there’s something vaguely time-sensitive on the horizon! Heroku is changing how they handle DNS registration for custom domains, which means I need to send out a bunch of relatively urgent (“take this action by July 31st…”) comms to everyone using custom domains. I’ll also use it as an opportunity to tweak how I’m handling the Heroku flow, since it relies on a couple bad assumptions (that get invalidated after this Heroku change) and is flaky in a painful way. So that (plus Roadmap V2, as aforementioned) should be on the ship list this week: though it’s going to be a long weekend and the start of a new month, which means I’ll be a little more spread thin than usual.

Hey, I did both of these things!

First, the Heroku changes. These were largely boring and good; I got to clean up some tech debt and improve the custom domain onboarding interface as I built it out, which is always nice. Exogenous event as impetus for tech debt pay down is rare, but I have to celebrate it when it happens.

That being said, it’s time for some dessert work! Maybe I’ll treat myself to cleaning up some of my dropdown CSS or messing around with the search bar UI, which has somehow already becoming stale. Either way, I’m going to grant myself a bit of a reprieve from the weekly goal: as long as I push the roadmap changes live, that’ll be good enough.

Well, it is 6.20pm on Sunday as I write this, and I have two items left on my list:

It turns out the trick to completing a daunting project is roughly threefold:

1. Whine about it a bit in various public forms
2. Ignore all outside stimuli (and by “all outside stimuli” I expressly mean Twitter and emails and not, say, a corgi mewling for attention)

Sorry for the radio silence last week. A double-dosed Weeknotes this time around, as recompense, focusing on two fairly divergent workstreams — one that has been occupying my coding time and one that has been occupying my thinking time.

Vue migration.

I’m desperate to onboard to Headless UI, a very useful set of components that let me unship a lot of cruft (I have home-baked popovers that are awful, I have home-baked modals that are awful, etc. etc.)

(Fool that I am, I wrote this on Sunday and checked it off of my todo list without actually scheduling the dang thing. Apologies!)

All of that is to say, I think the DNS work is a worthy goal for the week ahead. Once that lands, I think I’ll be in a good spot to spend May on dessert work: JavaScript exfiltration, support for attachments, that sort of thing.

This coming week, though — bug fixes. For real this time: I need to ameliorate some of the incoming customer service and bug fixes are the way to go. Honest! I’m building out a Things project and everything!

Hey, I did this! I picked out the five most noxious bugs (in terms of “time I have to spend answering questions around them” and also “embarrassment that they are extant bugs”) and by Sunday I had fixed four out of them. The fifth is — perhaps, dear reader, you are familiar with this one! — the whole “cursor randomly jumps to the top of the textarea” bug, which I am still as of yet unable to reproduce. There is even an entire about this of bug, which was both heartening (I am not the only one!) and disheartening (this may remain forever unsolved!)

I wrote last week that my Big Goal for this week was to take care of the three most noxious bugs I deal with and to, well, stay sane. Neither of these things happened! It was a very busy week — lots of support requests to answer, lots of day job work, lots of accumulated fatigue — and I mostly did my best to keep my head above water.

When I have tough weeks, I fall back into a familiar holding pattern: working on things that seem neat rather than important or urgent. This week’s subject of neatness is at least one that I’ve been discussing and thinking about at length lately:

On the face of it, this didn’t feel like a productive week. I did get two things of note done:

1. I shipped the standalone checker infrastructure I mentioned. I punted on some of the trickier (and less important) architectural questions, like how to declare reactions to check failures; I realized it was more important for me to just get the infrastructure live and working and that more nuanced iterations would only come with experience. (To wit: there have been a lot of fast-follows, such as me immediately adding the ability to track how long a check has failed or being able to re-run checks through the admin interface.)

Pay-what-you-want is live! There are still some fast-follows that I want to add, as alluded to in the blog post, but overall I’m satisfied with things. (In the…48 hours since public launch, I’m sitting a little shy of $5,000 being committed through PWYW, which is higher than I expected!)

It is always tricky to know when to call it a day here. I could have spent this week working on fast-follows, and the fact that so many of them have been explicitly called out in questions I’ve received post-launch is a sign that maybe I should have spent the extra day documenting upsell CTAs or adding more details to the post-upsell email. All of those tasks feel granular and snackable, though: they don’t require the dedicated level of flow and focus that actually hunkering down and shipping PWYW did.

I got a surprising amount of work done in the past week on pay-what-you-want! This is good. This kind of feature work is not what I am particularly good at lately: the ratio of “staring and thinking” to “coding” weighs heavily in the favor of the former, and I am guilty of getting bored of staring and thinking and trundling off to some well-scoped problem that I can just crank out in a few hours.

In fact, if you wanted to, you could enable PWYW right now — it’s live in Paid Subscriptions. This is also the kind of feature where me saying “oh I’m done!” presages the majority of the work. Wiring together the billing code and the settings code is the easy part: then it’s like, oh right I need to update four FAQ pages and write a blog post and most of the subscription interface is built on the assumption that everyone is on the same price so I need to add more details to the NotesModal and also the subscription confirmation email and I’ve been to revamp the premium subscription email to look nice and have GIFs so I might as well roll that into this work and oh this means I need to start slurping up custom plans from Stripe to populate the various modals and emails so — [exhales].

There are quite a few apocryphal quotes around “overnight success taking ten years”, and that’s kind of what this past week felt like: it took me I think around a year and a half to get my first $1,000 MRR on Buttondown (still one of my proudest moments as a developer ever!) and then things happen and I get a marginal $1,500 MRR in the span of a single week. Phrasing this as “MRR went up nearly 25%” is less visceral and more directionally accurate, but also much less fun.

Mostly what that meant was randomization: I wrote last week about my humble ambitions of puttering around on some small paper cuts and tech debt tasks while I set my brain to work thinking about a revised analytics stack in the background. Neither parts of that came to pass: I spent a tremendous amount of time on the operational and onboarding side (pulling in archives is easy; matching existing Stripe subscriptions is less so) and what little time remained was on figuring out what a new revised roadmap would look like.

“Change your roadmap entirely based on an influx of one specific customer demographic” is kind of a giant warning sign, and I don’t think I did anything too drastic: the on my new list are things that I slated for this half anyway, they just seem suddenly new and more important.

Last week, I wrote:

There is some bug bashing that needs to take place, but I think finally tackling multiple-newsletter creation in a less janky way than “log out, create a new newsletter, and paste that API key in” feels like a good candidate. (And no, of course that’s not because I received two support emails about it this morning. Why would you even ask that?)

And I appear to be in the position now for the second week in a row that a chunk of work I pegged at being a solid week’s worth of work ended up being about an hour’s. When I thought about it in the abstract, “newsletter creation from the perspective of an already-registered user” felt hairy and complex and prone to lots of yak-shaving. When I actually sat down to work on it and start figuring out the data modeling, it turns out it’s quite simple: newsletters need a name and a username to be created, and that’s it. The rest can be handled just by the existing settings interfaces (which, don’t get me wrong, need some TLC, but that’s an orthogonal problem) — there’s no point recreating all of those form inputs just to have them in a multi-stage modal or whatever.

Last week, I mentioned that I was gearing up for a week of frustrating (and hopefully rewarding) Heroku spelunking: dusting off the proverbial (and I guess literal — I had to update a couple dependencies) toolbox and going to town.

…

It was a one-liner, I think.

Not even a one-liner, the kind of one-line change that you arrive at with a sense of exhausted satisfaction after many hours down in the Performance Mines: a stupid, idiotic one-liner, a “oh my god of it’s this issue, why didn’t I just look at this sooner”. The in this case was a timeout set by , the web server I run around Django — it was set to kill any requests after , whereas Heroku had a hard timeout of .

The coy way to phrase how this week went is that I blew my Heroku budget out of the water. I woke up on Friday morning to discover that I had been Fireballed, something that my 20-year-old self would have been much more thrilled about but is still none-the-less very cool and very valuable. Getting a low-orbit cannon of traffic thrown at the site is useful in two dimensions:

This has been quite the week!

I’m in the final stages of a pretty fun high touch sales deal. I spent twelve hours assembling IKEA furniture in my basement. I shipped Importing V2. My dog got neutered. I’m in the final stages of delivering on a very fun and philosophically well-placed partnership with an analytics provider.

(Okay, some of those aren’t related to Buttondown.)

Buttondown’s newest mascot.

My latest of reasons for being bad at writing is the most adorable yet: Telemachus, my ten-week-old corgi puppy. He is a very sweet dumb fluffball, with some strengths: