Support

We are slowly and steadily making our way back down to inbox zero on the support side, having spent most of the past month and a half playing catch-up. This is largely uninteresting (the backlog, not the tickets themselves!): support is, at the end of the day, fairly easy to model under queuing theory — if inflow outpaces outflow, things are bad, if outflow outpaces inflow, things are good.

Anita has been starting to label and categorize our tickets so as to triage hotspots. Our three most common types of issue are Paid subscriptions, Deliverability, and Billing/Pricing, which kind of makes sense — they’re all open-ended, they’re all subject to weird external factors: much of “Paid subscriptions” is really “Cosplaying as Stripe support.

Mise

I spent a few hours playing around with Mise, which we're already using for managing some of our dependencies in the monorepo. It almost pains me to divulge, albeit slightly prematurely, that we're going to migrate off of Just and use Mies instead for our task runner. Just is still a great tool — but some of the things that I want to be able to do, such as cross-task dependency caching and better logging, Mies handles out of the box, whereas Just, by design, does not. I don't think this is a case where Mies is flatly better, and I think their alternatives guide is pretty reasonable and fair comparison of the two tools.

Incidents

I moved the incident postmortems that we had, which is thankfully only a handful, from BetterStack onto our blog as part of a slightly larger initiative behind moving and consolidating some of our more random bits of content in a place where they're easy to maintain and surface.

Rendering

Every quarter, I gather a handful of bug reports we've received around email rendering issues in esoteric clients and try and fix as many as I can in one fell swoop.

This is the unsexy side of running an email business. You spend more time in Outlook 2003 than you would otherwise ideally spend.

We are officially in Q4 — the time has flown. I'm reading through the changelog entries from this time last year, and it feels like a foreign country. It's also interesting to note which items stuck versus which things did not: building out email filtering was a foundational piece of technology that we now get to reuse in a bunch of different places; meanwhile, our emphasis on integrations and various one-off things barely recouped their implementation cost. (Will I learn any lessons from this? Only time will tell.)

Planning

I have historically been anti-deadline because I think the manufacture of false urgency does little more than occlude and dilute real urgency when it appears. Still, I also know two things: I am prone to snacking style work at the expense of really meaningful, ambitious projects, and I also know that you can't bench the bar all the way to three plates, so we're bringing back some lightweight project planning, by which i mean just scoping out issues on a month-by-month basis:

Buttondown has been around long enough, and the core set of assumptions around which we've laid our foundations have shifted and mutated so much, Ship of Theseus style, that sometimes we get to wake up one day and realize we can pull out an entire chunk of annoying logic without actually causing any issues downstream.

Here's one example: Stripe launched Payment Links a few years ago as basically a no-code way to spin out checkout sessions. This was kind of perfect for us because we had just launched paid subscriptions and had built out some of the logic for the base case, but nothing else (upsells; currency conversion; etc.) and we were able to just automatically spin up payment links on behalf of our users instead of having to really try and dig in and handle all of the edge cases and persistence.

That worked for a while, but over time, the number of things we needed to do in-house grew. We had a handful of customers who wanted esoteric tax ID setups; we had a number of customers who wanted to do pay-what-you-want and metered billing and things that payment links didn't support. The list grew and grew and grew, and we finally got to a point where we basically had a CheckoutSession creation flow that was at parity and in fact exceeded the functionality of what you get out of the box from Payment Links themselves. At the same time, we were still stuck with an orchestration layer to sync Payment Links, and these two things could get subtly out of sync with one another. For instance, if someone were to create a payment link in the Stripe dashboard or remove some existing metadata from one of the ones we created, this caused noise (our invariants have suddenly become variant!) and angst.

And Ben finally suggested, "Hey, can we just get rid of payment links?" The answer was, resoundingly and satisfyingly, yes. And we got to embark on a sequence of PRs that are so rare: pure deletions with no actual change to functionality and a cleaner state machine as a result.

Lord forgive me but it’s time to go back to the old me.

— Bugs Bunny

I can think of no better way to revive a years-dead newsletter than with a little bit of navel-gazing, so strap in because your appetite for five minutes of me dithering and rhapsodizing about content strategy and identity will be a useful bellwether for the things to come.

One of the interesting things about Buttondown as a product is that we've really elevated a use case for which I'm still trying to find the right phrase, but for now I call it "headless mode." What this means is you can basically set up Buttondown, connect an RSS feed to your newsletter, fire up a good old-fashioned form tag, and then you're sort of done. Buttondown will work away in the background, sending a digest email every week to your growing subscriber base. This is really nice and, in many ways, the acme of what I originally wanted out of the tool when I started it so many years ago.

The problem, of course, is that you don't log into the tool and you don't use the tool (or at least "the tool" in a very conventional sense of the word), and suddenly your touch sense of the quirks and rough edges and places where users of all ilk stub their toes dims and fades away. This is pernicious, especially because headless mode—aka the way I use Buttondown through jmduke.com, a very lovely personal web page, which you should still subscribe to if you haven't already—is how I primarily interact with the product. For the vast majority of users, the two biggest product surfaces are the editor and the archives. These are also the two surfaces that I do not use on a day-to-day basis.

Well, gang, thanks for letting me soft-launch surveys last week. The results are in, and I promise to play by the rules: this week, a technical deep dive and next week a war story.

So, a bit of a technical dive, courtesy of Matt:

Today's question comes from Benedict:

This is a great question!

I have been spending some time lately on long-tail SEO: comparison pages ("Buttondown vs. FooCorp"), integration pages ("how you use Buttondown with Bar.js"), and so on. This is, frankly, boring but valuable work — the kind I tend to shy away from unless I gamify it into a streak or something similar.

One model of "how to think about long-tail SEO" is the Crusonia plant, an economic plot device of a crop that, once planted, grows at a stable and monotonic rate with unceasing free yield.

I wanted to spend some time in January setting up some systems for both writing and working that would carry through the rest of the year: I succeeded (more about that in a second), but at the cost of being able to bang out these lovely little updates. Thank you for your patience!

So! What was I up to in January?

Well, first, I hit my monthly goals.

I’m stealing the concept of monthly goals from Michael Lynch, a bootstrapped founder who I’ve been following for a while and who has been kind enough to provide some high-level meta-direction.

I took a break from the mistletoe chaos of the dead zone between Christmas and New Year’s to do two things, both portrayed below:

• Triage Buttondown’s roadmap from 325 items (at time of writing this; by the time this goes out, it’ll likely be 330) to ‌under a hundred.
• Stack rank the top ten things I want to get done in Q1.

I wanted to write about annual planning, and how this year is going to differ from previous ones, but I have (in true fashion) procrastinated finalizing my annual plan a little bit. So that'll be next week, and this week I would like to write about... monorepos.

Buttondown is now in a monorepo. There's two big advantages of this approach (and these advantages are not particularly new to Buttondown):

1. Code sharing is much easier. I now have a lovely OpenAPI spec that powers the documentation page (and is published to its own repository, though I'll get to that in a minute); rather than having to perform a bit of a shell game keeping that spec in sync across a number of repositories, I can have a single source of truth.
2. Concomitant changes are much easier. The ideal feature launch involves not just changing some stuff in the main application but writing a release note, adding a changelog entry, adding documentation — timing the release of all of these PRs was frustrating.

Revue and growth

I mentioned a few weeks ago that my cockroach-esque strategy of "just stay alive while competitors flame out" had led to another recent success, this time coming from Revue (and Twitter in general)'s general cacophony over the past six weeks.

This framing — one that has deeply impacted my product approach — I owe to idlewords aka Maciej Ceglowski aka Pinboard. I remember reading Pinboard's blog literally a decade ago and marveling at the general independent SaaS shtick and thinking it was beyond my wildest capabilities, and how there had to be some mysterious trick beyond "wait for competing applications to stumble and/or fail".

To wit, this quote of his from a few years ago has always rung true:

Sorry for the week off! I spent it in Florida with my soon-to-be in-laws (it was relaxing, in the way that Florida always is) and am digging myself out of the various administrative holes that ensue. In lieu of a traditional narrative update for these weeknotes — and perhaps a little inspired by the holiday festivities — I wanted to highlight three little things that have made me happy this past week.

Django-ninja

Django-ninja is a very good set of ideas with an unfortunate name. (Seriously, I cringe to see the word ‘ninja’ in my pyproject.toml.)

I am knee-deep in the process of replacing django-rest-framework in my public-facing API with django-ninja. DN offers two main advantages over DRF:

The challenge/luxury/eccentricity of working independently on a big piece of software is that you make and own all of the design decisions. There is no designer to iterate on mockups; you are the designer. There is no UX researcher to organize A/B testing sessions and wireframing feedback; you can do that, if you'd like, but it often means sacrificing velocity.

I've found that the best work I do comes when I can leverage jumping from design details to implementation details with frightful speed & abandon; this requires a comfort level with making snap decisions very quickly, but often can lead to really, really strong results.

Unfortunately, the reverse is also true. The worst work comes from me tunnel-visioning my way into a design or interaction that 'feels good' not because I think it resonates well with users but because it fits my pre-conceived notions or my desired architecture.

“Data onboarding” is a slightly bureaucratic name for a very boring and important process — bringing data from Service A to Service B when a customer joins Service B having used Service A. You have probably done data onboarding many times in your life, whether its uploading a list of contacts to a new social app or whatever.

Buttondown has two tranches of data onboarding:

1. importing subscribers. This is, all things said, pretty easy, thanks to the unimpeachable portability of email addresses. There are occasional edge cases: heavy use of metadata, esoteric paid subscriptions, that kind of thing, but Buttondown’s current solution here (a bespoke CSV parser with some bulk validation at the backend) is pretty good, and the number of times I have to really do something manual here is fairly low.
2. importing archives. HTML is, while portable, very gross, and in general folks import archives at a rate of (according to an SQL query I hammered out just now) one tenth of the rate at which they import subscribers. This is for a number of reasons: some people don’t care that much about bringing their archives over from a previous service, some people use a blog or other website as a ‘source of truth’ for their archives, whatever.

The combination of “not many people do this” and “it’s very annoying to make this better” is generally a sign that I neglect whatever that thing is, and that heuristic holds true for archive imports. Some services (Substack, Wordpress, Mailchimp) have fairly standardized exports that include archives, which makes my importers for them pretty reasonable and durable. Some services — like, say, Revue — do not, and force me to crawl + parse them to get something approaching a reasonable result. Or, in Revue’s case, a pretty unreasonable result: Buttondown’s ability to import archives from Revue is frankly pretty bad.

This is a blessed Monday — the first one since early July that I am not sitting on a bloated diff that I’m antsy to merge.

I merged in the editor changes Thursday, and I am thrilled with how they turned out. (Thank you everyone who submitted feedback + bug reports over the past few weeks!)

What’s next? is a question I’ve been asking myself over the past few days, and the answer is, I think, boring — the rest of the year is going to be focused on stability, marketing investments, paying down some technical debt, and operational excellence:

• I mentioned a few weeks ago that I spent a few days babysitting. One of the big takeaways I had from that experience was that Buttondown has a lot to go before it’s as turnkey as I’d like it to be: I imagine a world not too far off from this one where Buttondown requires around five hours a week of engineering labor and five hours a week of operations/services labor, both of which I can hand-off to contractors I trust. While Buttondown’s roughly there from a time standpoint, it’s definitely not from a documentation standpoint. There are a lot of warts and weird corners that make sense to me and my brain but would be incomprehensible to a theoretical outsider. I’d like to pay down some of that knowledge debt: clean up parts of the codebase, automate things that take me three minutes but would take an outsider an hour, write more things down.
• I mentioned last week that I’m deep in some refactors on enabling teams & SAML support. Those are going well, and also perfect ‘cozy work’; it’s fairly rote and I get to do some cleaning as I go, and there’s very little meta-work I need to do around it.
• Marketing pages. So many marketing pages. (At least the changelog was fun.)

(A long overdue weeknote. The last two weeks featured a wedding and an impromptu trip down to Florida, so I haven’t had much bandwidth — thanks for your patience!)

The settings and sidebar are now live 🥰. The new writing experience is absolutely ready to be live — I am hovering my cursor over the metaphorical deploy button — but I’m holding off a little bit longer just to make sure nothing catastrophic appears on the settings or sidebar fronts.

The good news: nothing catastrophic has appeared yet. I am thrilled! It is very hard to change five-year-old code used by literally every user without imagining some very grim possibilities. There were some pain points, to be sure (I did a poor job of testing on some mobile breakpoints and in Firefox), but the vast majority of the feedback was positive and the most pointed critiques of the changes (“it is hard to jump between drafts” and “search is harder to get to quickly” are both things that I anticipated and will be fixed with future iterations.)

The funniest second-order outcome thus far has been an uptick in the number of people asking me to migrate them onto the new pricing from earlier this year; I didn’t actually change any logic here, but I moved the notice that has been live for a while from the Billing view to the top of the omnibus Settings view, which of course gets more traffic. This has overall been positive in terms of MRR, albeit in an unintended way; I am now even closer to unshipping the legacy pricing code, and tempted to carve out some time to just manually beseech the remaining few-dozen users to let me shift them over.

A very late note this week — I was babysitting my nine-month-old niece, who is very sweet and also imparted upon me a newfound respect for folks who manage to find time and space for deep work while being an active parent.

The settings work I discussed last week is ready for launch (yay!) but I didn't want to babysit the launch while also literally babysitting (both due to lack of focus and, as I was soon to discover, lack of sleep), so I found a bit of low-priority work that would be fun to noodle on between stroller & crib sessions.

This was bulk actions, a primitive that I am fairly confident every SaaS has reimplemented. Buttondown already has a kind of bulk action, which you can see when you select a bunch of subscribers and/or emails:

I have ranted a few times about how much I like the metaphor of a technology tree as a representation of a technical product roadmap. It does a good job of representing a number of unintuitive but important ideas: the role of invisible foundational work to enable downstream functionality; the seams and overlaps between various features and surfaces; the choices one must make between wildly orthogonal areas of investment. I have long wanted to make a technology tree for Buttondown, and at some point I should just dedicate the afternoon to doing so.

One pain point in any sort of planning (using a video game-y tech tree or otherwise) is that the map is not the territory — you can make a bunch of implicit and explicit decisions based on well-reasoned and charitable assumptions that end up being incorrect.

When I started the increasingly-not-tongue-in-cheek "Buttondown 2.0" effort two months ago, my thought was roughly: "I want to build a new authoring experience, which will require a bunch of refactoring changes to the current authoring experience & email abstractions. Once that is done, I can continue with other significant scaffolding changes to the IA and the settings." Basically, I assumed that the core changes I wanted to ship to improve the writing experience were completely independent of the other changes I had in mind.

Reality — and user testing, thanks again to those who provided feedback — reveals otherwise! As I touched on last week, the big piece of feedback needing redress was an improvement in switching between open drafts. "Okay," I thought. "Time to start working on some of those IA changes and bring them in to shift from a big ol' dropdown to a more conventional collapseable sidebar." I have a WIP branch of that and I really like how that turned out (a drawer of recent drafts that you can click through to easily swap):

A rapid fire of three topics, since I’m still feeling a little bouncy from my pre-workout:

I.

I, somewhat unprompted, launched my new marketing site. It is… pretty unremarkable, intentionally so — a new font face, bigger text, but otherwise the same corpus and approach and all of that. I wrote about the marketing site during my initial efforts with it back in late Spring, but it fell under the wayside for — well, for no reason in particular. It was a high-importance, low-urgency project in the Eisenhower sense; I wanted to get the marketing site off of the core Django application that serves the actual app, but didn’t have any big timeline associated with it and so it lingered at 90% done for…eight weeks or so. I told a friend about this and they asked (rightly!) “so are you just four hours of work away from finishing it, in perpetuity?” and that got me sufficiently off my ass to finish porting the remaining pages. There are a couple things I want to do still, but it’s nice having them be small additions as opposed to increments on a very big change.

(Also, if you’re interested in adding your newsletter to the rotation of social proof right above the main CTA, let me know!)

First, to lead with something like a call to action or exciting news: want to play around with the new writing interface? You can! Head over to beta.buttondown.email. (The interface will warn you, as will I: this is a real production environment with a different clientside skin, so don't send any emails you aren't meaning to send.)

The new writing interface is going well, and if I was surreptitiously delivered truth serum before sitting down to write this note I would probably describe the status as "totally production-ready but I'm a little anxious about the sweeping change so I'm currently manufacturing reasons to delay pushing it to full production for another week or two." Around a hundred people are using it actively; around half of them have sent actual emails.

I started this workstream with a bunch of work that barely looked relevant at the time: unwinding some spaghetti code, breaking out nested modals into routes, a bunch of Serious Work that had zero impact on the actual look and feel of the writing interface. Now, in the home stretch of the workstream, I find myself engaged in something like the opposite of that work: a multitude of cosmetic changes that are important not because they touch the writing interface itself but because they emphasize the jarring disconnect between the interface patterns of the interface and the administrative side of the application.

Some of these things are pretty trivial: the new emails list (that's right! there's no more separate list of drafts versus scheduled emails versus archived emails any more, there's just a single glorious list) has a different filter button than the subscribers list, and it's all of fifteen minutes of work to change that.

This was slated to be an uneventful week for Buttondown, and in most cases it was: I was traveling for a combination of wedding prep and visiting my new nephew, and there wasn't a lot of bandwidth for work beyond keeping on top of emails & fires.

Instead, I got to spend Tuesday evening hastily swapping over DNS records to try and recover from what ended up clocking in as a seven-hour DNS incident from Heroku. A deep extension of gratitude to Hacker News user rsweeney21 who suggested the approach I and many other folks went with, which was temporarily rerouting to the historical IP addresses via A records to restore service ahead of time — while mucking around in DNSimple on my friends Ethan and Laura's roof was not the way I imagined spending my time, I was grateful that there was at least some agency I could exercise. (DNS stuff is scary.)

Hello from the skies, and sorry for not checking in too much in August. Between a very nice vacation and a not very nice stomach bug, I was not availed of too much dedicated writing time. (I spent most of my time keeping on top of emails and working in fits and starts on the new writing interface.)

The end of August roughly marks the end of three months since having Buttondown be the most important professional activity in my life, and maybe ~one month since I started seriously ramping up my time on it. It has been really delightful (in no small part to the cohort of folks who upgraded to higher-end plans and emailed me to say it was explicitly to support my work — those of you, you have my undying thanks!)

I've shipped features at a higher cadence in the past three months than any time since Buttondown started collecting paid users back in 2017, and it's felt quite nice. That being said, I very much feel like it's time to spend my time less on features and more on core improvements, using this time and energy to make long-overdue improvements to the most important parts of the existing functionality. (I'm referring to this in my head with the appropriately overblown title of Buttondown 2.0, which of course does not refer to any actual new codebase or anything.)

The short list of what I have in mind here:

A brief love letter to Logtail

I'm keeping this somewhat brief because I want to expand my thoughts into a longer and more cogent blog post, but I finally got myself into a place where I feel good about Buttondown's logging. This was a combination of a few separate things:

1. I finished onboarding to structlog, which makes every logline structured (in JSON for production use cases, but not locally since it's very annoying to visually parse JSON)
2. I found a log sink that handles my use cases — Logtail, which lets me write SQL against my logs.

I've put off caring about logs for a while, since there aren't that many times where I've thought to myself "boy, I wish I had better logs." Most of Buttondown's shenanigans that require deep introspection (missing subscribers, odd rendering quirks, and so on) can be handled at the database level.

I performed a bit of high-pressure grape surgery this week with exactly one production issue (a far cry less than I was worried about slash expecting) — all drafts are now emails. What that means: where there was once a “drafts” table for, well, drafts, and an “emails” table for things that were emails, there is now one single table with a status enum (draft, sent, in flight, et cetera) combining them.

And with that, one of the last poor architectural decisions of 2016 Buttondown has been erased.

This is one of those technical debt things that is probably hard to glean the value of from outside the codebase. It has become increasingly obvious over the past few years that “drafts” and “emails” should be the same thing:

• The interface for editing a sent-out email is extremely janky because the big, fancy writing interface can only really interact with drafts;
• a common desire is to be able to preview the true web archive of a draft before its published, which is impossible since that rendering pipeline only happens on emails and not drafts;
• adding undo support for publishing drafts was tricky because history had to be tracked across multiple tables;
• and so on.

A brief excursis on working

A little less about the what and more about the how — I’ve been slowly triangulating on what a really nice schedule for my day-to-day is going to be now that I’m shifting from “sabbatical mode” to working on Buttondown in earnest.

One of the things that has been a very surprising change to me is how difficult and tiring craftsmanship can be. If you’ve been reading these notes for a while, you likely remember me complaining often about how difficult it was to find time for the ‘serious’ work, since my schedule (and Buttondown’s growth) only permitted time for reactive work during the week.

That is no longer the case! This is great. Most mornings start out with an hour or so of operational and reactive work (emails, dashboards, you know the drill) and then I get to shift to actually use my creative energy. It is exhilarating to actually be able to spend the preponderance of my time improving Buttondown, rather than just keeping my head above water.

Multi-factor authentication

I joke often about my product prioritization heuristic for Buttondown being “work on the thing that I’m most embarrassed at not having.” This heuristic has led me to my work for the past week: support for multi-factor authentication (now live for Authenticator apps!)

This is long overdue (and perhaps a little accelerated by a recent enterprise sell whose conversation soured when I said it was merely ‘on the roadmap’.) It’s also been, frankly, pleasant to work on — I get a lot of decision fatigue when it comes to designing relatively complicated UX flows from whole cloth, and I don’t really have to worry about that here because:

1. we As A Society have pretty well narrowed in on what the right MFA registration & input flow should be, and leaning into that is a good thing;

2. Buttondown’s design system now has rigorous and easily extendible answers for “how should I display a multi-step modal?” and “how should I display a rich table of primitives”?

ProseMirror is neat!

I went down a bit of a rabbit hole this week, to nobody’s surprise.

Last year Buttondown swapped out a very old javascript package for fancy text editing (TinyMDE) for a much newer one (tiptap). This was one of the more fraught roll-outs in Buttondown’s existence, not for any explicit technical issues but because lots of people are understandably persnickety about core workflows changing. A valuable lesson learned!

The main reason I wanted to shift to tiptap was not out of any one specific use case but being able to go from an abandonware piece of software to something nice and powerful and extensible, so I could invest in a better writing experience down the line.

A bit more about the new marketing site

I mentioned last week that I am tooling around with a new marketing site. The reasons for this, in no particular order:

- getting to extricate the marketing site from the core application codebase

- better SEO (and outright performance)

I wrote a few months ago about a critique by Brian Lovin (the best thousand dollars I’ve spent on Buttondown by a country mile!) and how it was dovetailing with some of my efforts to refresh Buttondown’s overall app scaffolding. That work has been slow but ongoing, and I wanted to chat about it a bit. First, the screenshots:

• I started out with a pretty mobile-inspired concept of dual actions: core CTA on the bottom right, ancillary actions on the bottom left. I think pinning core actions makes sense, but I’m not sure this scales; I’ll likely need to have a generic “actions” button in the bottom left to keep things uncluttered.
• One thing that makes this work a little harder than I’d like is that Buttondown’s front-end is split between a bunch of legacy SCSS and Tailwind. I’d love to wave a magic wand and get rid of all the SCSS (and I’m trying to make sure any files I touch get SCSS-purged), but this is one of the perils of trying to design a relatively old (and in some places, crufty) app in the codebase as I go.
• The elevation of the header to a three-tier thing (core header, universal search, and hamburger menu) is small. I go back and forth on whether or not having the search bar as such a prominent affordance is useful; it’s tempting to just have an icon that pops up a modal, which would certainly make the design quieter.

There is a lot left to be done. I want to make the sidebar look — how should I phrase this? — less janky; I want to finally unshackle the app from its default full-width constraints; I want to make sure this all plays nicely on mobile viewports, which have been hitherto underdeveloped. But it’s closer to being shipped than it is to being started, so to speak, and I wouldn’t be surprised if I get it across the finish line by the end of the month.

Now that a full month has passed, I can actually take a look at how the new pricing plan is working as compared to the old one with some level of statistical rigor. I expected and/or hoped for three things:

1. A better conversion rate from “activated user” to “paying user”, because the pricing was clearer and anchored lower for a non-trivial portion of folks;
2. A slightly worse conversion rate from “prospect” to “registered user”, because the free tier’s efficacy was diminished;
3. A net improvement in overall conversion from “prospect” to “paying user”, meaning that point 1) would outweight point 2)

A couple caveats and points of elaboration:

Zombie branches

Sometimes you have zombie branches.

Zombie branches are little projects that are large enough to be difficult to completely set aside a day and ship but either important or approachable enough that you can chip away at, slowly, such that it never feels like you should just close the branch and call a sunk cost a sunk cost.

One of my goals this month is to try and get things as tabula rasa as I can for whatever a long-term schedule looks like in June. This of course means trying to cull the zombie branches, one way or another.

May 20, 2022

The big news

I led last week’s note with a teaser, and here’s the reveal: I’m working full-time on Buttondown.

Okay, “full-time” is not quite yet the case. I’m taking the month of May to relax and recharge and enjoy the trappings of a slightly more lackadaisical lifestyle for a bit.

But Buttondown is now the main thing. This is exciting! And long overdue.

(Sorry for missing last week! I promise you will learn why next week. It’s a pretty good reason.)

Repricing is freakin’ live

If you were to sign up for a Buttondown account this very second, you’ll be on the new pricing structure! It’s been live for about a week now, and I’m very proud that nothing has exploded. It’s too small a sample size to see what the numbers “actually” are in terms of conversion relative to the old pricing structure, but I’ll take pride in the fact that it’s non-zero.

In exactly six days, I’ve had one $79/mo plan, one $29/mo plan, and six $9/mo plans come through. I am most proud of that last bucket: as I’ve chatted about (and as I’ll make public in an upcoming blog post), the primary goal of this work was not to increase ARPU in of itself but to ‌decrease the relative burden of the free plan, and five of those six relatively small customers would have been free under the old pricing scheme. This makes me very, very happy, and points towards a much more sustainable usage pattern for new folks.

Repricing: it's finally happening

After what feels like years of punting, the repricing work is finally going live in a non-trivial way! This week marks a legitimate milestone in the project: I have the first few production subscriptions to the new pricing plans. (This is kind of a cop out because I manually generated and assigned invoices for those folks as they have atypical payment structures, but it counts — it means the feature gating logic and all of that is working and active in production.)

I'm getting to the point where the end might not be in sight — I don't have a firm burndown list of everything that needs to be done — but certainly seems much closer than it did. Big, amorphous projects kind of go through an explore/exploit phase — you have broad, unbounded work until you get to the point where you've mapped out the state space of the full project and then you burn it down. I'm not in that latter space yet but I think I'm pretty close: there's going to be a long tail of marketing copy that I haven't yet itemized and a lot of pretty boring plumbing to update, but if I was told I absolutely needed to get this all done by the end of the month I certainly could.

A little aside: when I told a friend about these changes and about how many authors would in fact be paying less (if you're paying for Buttondown for Professionals and have 4,000 subscribers, under "old" pricing you'd pay around $54/mo and under "new" pricing you'd pay a flat $29/mo) and he warned me to run the numbers to make sure I wasn't killing my ARPU. This was good advice, but the panic was unfounded: if I were to simply project the exact same userbase that Buttondown has today into the new pricing scheme, MRR would approximately double. (Of course, you can't assume that all cows are spherical and all that, but my point is that the cohort economics are valid.) So that's encouraging.

The case of the missing scheduler, part 3

It would be lovely (albeit unsatisfying) if this was an issue that evaporated on its own, like so many transient hardware-ish issues I’ve dealt with in the past. In the meantime, I’m just grateful I haven’t been woken up by any pages.

By leading off with that quote, you might be able to surmise the followup: absolutely no progress, and also no crashes. At this point my Spidey sense is leaning more towards “this was some esoteric hardware issue that I should have better alerting on but was ultimately exogenous to my code”.

Sigh.

The case of the missing scheduler, part 2

I was really hoping that I would have some sort of juicy insight or breakthrough to share here. I... do not.

The good news: it appears to be happening somewhat less frequently than last week. I can't point to any reason why that might be the case: I've added some additional instrumentation and logging to the scheduler dyno, but haven't actually changed anything. (To wit: the scheduler's crashed exactly once in the past four days, as opposed to roughly once every eighteen hours last week.)

It would be lovely (albeit unsatisfying) if this was an issue that evaporated on its own, like so many transient hardware-ish issues I've dealt with in the past. In the meantime, I'm just grateful I haven't been woken up by any pages.

March 23, 2022

The case of the crashing scheduler

This week’s tricky operational issue is one that is still a little in-progress. Buttondown runs on Heroku, and uses a bunch of dynos. Think of dynos as akin to containers: everything on Heroku uses the same application bundle and the same mess of code, but each dyno is meant to correspond with a specific function or executable. For instance, Buttondown’s got the following Procfile, which is Heroku’s nomenclature for defining dynos:

web: bin/web
worker: bin/worker
checkerworker: bin/checkerworker
scheduler: bin/scheduler

Hello after a month of chaos! I am writing for you for the first time from my new house in Richmond, Virginia. In the past 28 days since my last missive, lots of non-Buttondown things have happened: a cross-country move, an unpack (178 boxes in total, because of course we counted), a whole lot of Telemachus playing fetch (and gaining a solid two pounds thanks to his grandparents spoiling him), and Buttondown has — well, there’s been less forward progress and more staying afloat.

A whirlwind tour of the highlights (and lowlights):

• Overall, things were… pretty stable? Growth continues apace and I am much worse at inbox management than I’d like to be (it’s been a while since I’ve hit inbox zero, but I’m fairly steadily at inbox ~thirty). No major incidents besides…
• Heroku went down. That was annoying, even if I’m a little pleased that I found out before their status page was updated. Still, one hour of downtime in ninety days is not worth seriously considering accelerating my AWS migration plans.
• I had some Redis scaling issues. These were boring scaling issues: if roughly ~500,000 emails were queued up to send then my instance would cap out and deadlock and what do you know, that happened. This was solved trivially by bumping up my instance to the next size, but that’s only delaying the problem’s event horizon to the next order of magnitude. I’ve been fantasizing about migrating off of Redis and using postgres-based queuing (RQ, which I use as a queuing library, is not particularly wonderful even if the API is nice) and this accelerated things somewhat.
• February’s roadmap was ambitious, and… did not go as planned. There was a lot going on, and the snatches of time I had to work on Buttondown were spent furiously going through email and keeping the lights on. Despite that, I shipped one big thing that I haven’t quite blogged about yet (and will next week!) — an entirely new subscribe form. Given that this is an input that a cool ten thousand or so people convert through every day, the fact that I was able to roll out the new experience with no fanfare and a huge amount of uplift is tremendously exciting.
• March’s roadmap, on the other hand, is much more reasonable — two small features (already shipped!) and two bigger ones which are roll-forwards from February. The vast majority of the moving part of moving is behind me, but it’s still going to be a busy month: I’ve got two and a half trips lined up in the next three weeks.

Hello hello! This past week was a quiet one: I mentioned that I was headed to the cabin with some friends, and cabin time was unproductive — which is to say it was time well spent. This week has been productive, but it's also been start-of-the-month-productive: there's been a lot of invoices being sent for processes that I've been a little too lazy to automate, there have been tweet threads to write and various marketing efforts to kick off, that kind of thing. (Today is perhaps emblematic of the entire week: four hours spent, which is pretty high, but two of those on operations, one on marketing, and one on actual development efforts.)

Two 'big' things on my mind:

First, I was lucky enough to score some time with Brian Lovin before he wised up and raised his prices. If you are interested enough in Buttondown to read this ol' newsletter every week, I couldn't recommend enough reading through his design critique — I was furiously nodding along through the entire thing, and have a slew of things to fix (both big and small) thanks to what he surfaced.

One of the things that he really brought to light for me, at a bit of a meta level, is how lost I've gotten in the day-to-day of working on things. I probably spend.... 7%? Of my time thinking about Buttondown as a gestalt, since so much is firefighting and keeping the lights on — which I mean in a positive way, since Buttondown has enjoyed such unalloyed growth, but it still means reaction rather than action. In particular, I think he pushed me in a really good way for some of the design system-y things that I've been thinking about, and the long-ballyhooed scaffolding work I haven't merged yet is... going to remain unmerged. It is (cheerfully, since I didn't share it with Brian) very much in line with where he was going, but I think the right thing to do is to keep it under wraps and keep iterating on it a bit longer.

I wrote last week about three painful queries that I needed to futz with: exporting all email events, aggregating client statistics, and time-series aggregations.

This slightly more methodical approach worked pretty well! To start with the good stuff:

1. Exporting email events for a big newsletter now takes O(seconds-to-minutes) instead of O(minutes-to-hours). This in of itself was the single most urgent thing: I can stomach analytics not working for certain newsletters, but the idea of someone not being able to get their data out of Buttondown was anathema. ¹
2. Client-level aggregations are now live and the "inbox statistics pane" loads quite fast. This is one of those lovely performance changes that actually have product-level implications, too — it's now feasible for me to pull out specific clients used by a subscriber and display them live in the subscriber detail page, whereas previously that would have been a pretty expensive call.

Time-series aggregation is still...pretty painful. If you're a newsletter with, say, twenty thousand subscribers and a year's worth of emails on Buttondown, I'm no longer timing out when you try and get your analytics for the past quarter (yay!) but I'm definitely still timing out when you try to get your analytics for the past year (boo!)

Buttondown had its first planned partial outage of the year. (Of 2021, too, if my memory isn’t failing me — but, you know, new year and all that.)

This was to upgrade Buttondown’s database: a measly but stalwart db.t2.medium running on RDS, first spun up in 2017 and humming along ever since. ¹ The general parlance when thinking ab out relational databases is “people tend to underestimate how strong and scaleable a small, boring Postgres box can be” and I think that is true: this cute lil $29/month instance has brought me very few complaints over the past few years.

The upgrade was necessitated by two factors:

First, the database (cleverly named buttondown) was running Postgres 9.6, which Amazon was planning on formally deprecating and forcing upgrades. They’d pushed back the required upgrade window twice already, but I did not love the idea of waking up at 4am to a slew of pages because AWS decided when to take the database down.

Welcome to the first Weeknote of 2022! I'm fairly excited about this year (as much as one can be excited about any year) — the slate of work feels fun and well-formed, I've got some exciting personal stuff happening (moving!), and for the first time in a while I was able to actually spend the last two weeks decompressing to some extent (a marked difference from December 2020, which was quite fraught and stressful.)

My slate of January work is much more meager than my Q4 roadmap, which was — let's be honest — overly ambitious. Here it is, in total:

• Launch new scaffolding
• Blog about the new scaffolding
• Add checker dashboard
• Remove slug uniqueness assumption
• Allow changing subscriber addresses
• Add recovery flow for spammy subscribers
• Allow both banning and unsubbing

That new scaffolding is almost ready for prime-time, and there's a 60% chance or so that by next week it'll be live, which will be the single biggest change to Buttondown's interface since...ever? I think?

(Apologies for the off weeks for the past two weeks! Travel, et cetera.)

Checker, revisited

As the year winds down and I try not to alarm myself with the lack of progress on my Q4 goals (only two out of the six items I wanted to ship have shipped!), I amuse myself by reading through older issues of this newsletter and admiring the delta.

One such item was the checker infrastructure that I built in early April. I wrote about this as if it was dessert work: a fun challenge with low ‘real’ ROI:

A belated note about public archives

In my excitement and/or delirium last week, I forgot to mention that I managed to carve out and ship one of my six Q4 goals — a nicer public archive list! This was absolutely the easiest one of the six in terms of scope (just messing around with HTML and CSS for a while) as well as the most fun (just messing around with HTML and CSS for a while).

The rebuild itself was not particularly noteworthy: the pre-existing design was , and so the bar was low, but I mostly just spent five hours tweaking things until they looked particularly nice. A couple more interesting points:

Good on my promises.

I vowed last week to push out the list view changes, and I did! They're now live. There are going to be some hiccups in making sure the design scales out to the bevy of use cases, but no big surprises or all-hands-on-deck-emergency-commits; it was as painless as I can hope from a substantial refactor of that nature, which I'm pretty pleased about. There will undoubtedly be a long tail of tiny fixes and tweaks over the next month, but that's good — all of those will apply to the actual design system, rather than one-off components.

On the "edge cases" front, I was reminded that Buttondown's getting to the point where I really can't fit the universe of use cases and integrations in my head. I flagged the last cohort of folks into the new archives (which means I can unship the old archives soon, thank God!) and someone pointed out that they had issue numbers disabled, which — after running the numbers — less than 0.1% of active newsletters do. Which is fine for them! But man, it's just too hard to game-day out the very complex state space of newsletter configurations. Part of this is by design; part of this is perhaps by bad design. (Not that there's anything wrong with this particular setting, but that I need to get a bit more disciplined about enumerating them in the future, and potentially tamping down on existing usages.)

Another quixotic venture.

This week was split in twain.

I spent the first half in Seattle, as I tend to do, plodding away on the ListView work that I’ve spent the past two weeks chatting about.

It is… going well! The painful scope creep I mentioned last week has mostly been dealt with; I’m dealing with the long tail of various features that needed to be port over, and I’m firmly in the ‘downhill’ part of a scope creep-y project — when the punch list might be long, but I’m confident it’s finite, and the hours of “checking off one item will inevitably spawn six more” are behind me.

First, I suppose, the good news:

Email filtering, such as it is, is done. My goal with the item is complete! My DataTable component has expanded and I now have a ListView component that takes a bunch of stuff, and the Archive page at /emails exists solely as a wrapper around the ListView. (Plus I added filtering, the whole point of the exercise — and the most trivial one.) There are a couple tiny things I need to do before merging, like better type information and fixing where the dropdown menu appears, but it is Done.

The problem, of course, lies in an email from Anne I received in response to last week's weeknote:

This looks nice, but it's different than the current look and feel of the other pages, right? How do you think about rolling this out and not having it be consistent?