The big news

I led last week’s note with a teaser, and here’s the reveal: I’m working full-time on Buttondown.

Okay, “full-time” is not quite yet the case. I’m taking the month of May to relax and recharge and enjoy the trappings of a slightly more lackadaisical lifestyle for a bit.

But Buttondown is now the main thing. This is exciting! And long overdue.

(Sorry for missing last week! I promise you will learn why next week. It’s a pretty good reason.)

Repricing is freakin’ live

If you were to sign up for a Buttondown account this very second, you’ll be on the new pricing structure! It’s been live for about a week now, and I’m very proud that nothing has exploded. It’s too small a sample size to see what the numbers “actually” are in terms of conversion relative to the old pricing structure, but I’ll take pride in the fact that it’s non-zero.

In exactly six days, I’ve had one $79/mo plan, one $29/mo plan, and six $9/mo plans come through. I am most proud of that last bucket: as I’ve chatted about (and as I’ll make public in an upcoming blog post), the primary goal of this work was not to increase ARPU in of itself but to ‌decrease the relative burden of the free plan, and five of those six relatively small customers would have been free under the old pricing scheme. This makes me very, very happy, and points towards a much more sustainable usage pattern for new folks.

Repricing: it’s finally happening

After what feels like years of punting, the repricing work is finally going live in a non-trivial way! This week marks a legitimate milestone in the project: I have the first few production subscriptions to the new pricing plans. (This is kind of a cop out because I manually generated and assigned invoices for those folks as they have atypical payment structures, but it counts — it means the feature gating logic and all of that is working and active in production.)

I’m getting to the point where the end might not be in sight — I don’t have a firm burndown list of everything that needs to be done — but certainly seems much closer than it did. Big, amorphous projects kind of go through an explore/exploit phase — you have broad, unbounded work until you get to the point where you’ve mapped out the state space of the full project and then you burn it down. I’m not in that latter space yet but I think I’m pretty close: there’s going to be a long tail of marketing copy that I haven’t yet itemized and a lot of pretty boring plumbing to update, but if I was told I absolutely needed to get this all done by the end of the month I certainly could.

A little aside: when I told a friend about these changes and about how many authors would in fact be paying less (if you’re paying for Buttondown for Professionals and have 4,000 subscribers, under “old” pricing you’d pay around $54/mo and under “new” pricing you’d pay a flat $29/mo) and he warned me to run the numbers to make sure I wasn’t killing my ARPU. This was good advice, but the panic was unfounded: if I were to simply project the exact same userbase that Buttondown has today into the new pricing scheme, MRR would approximately double. (Of course, you can’t assume that all cows are spherical and all that, but my point is that the cohort economics are valid.) So that’s encouraging.

The case of the missing scheduler, part 3

It would be lovely (albeit unsatisfying) if this was an issue that evaporated on its own, like so many transient hardware-ish issues I’ve dealt with in the past. In the meantime, I’m just grateful I haven’t been woken up by any pages.

By leading off with that quote, you might be able to surmise the followup: absolutely no progress, and also no crashes. At this point my Spidey sense is leaning more towards “this was some esoteric hardware issue that I should have better alerting on but was ultimately exogenous to my code”.

Sigh.

The case of the missing scheduler, part 2

I was really hoping that I would have some sort of juicy insight or breakthrough to share here. I… do not.

The good news: it appears to be happening somewhat less frequently than last week. I can’t point to any reason why that might be the case: I’ve added some additional instrumentation and logging to the scheduler dyno, but haven’t actually changed anything. (To wit: the scheduler’s crashed exactly once in the past four days, as opposed to roughly once every eighteen hours last week.)

It would be lovely (albeit unsatisfying) if this was an issue that evaporated on its own, like so many transient hardware-ish issues I’ve dealt with in the past. In the meantime, I’m just grateful I haven’t been woken up by any pages.

March 23, 2022

The case of the crashing scheduler

This week’s tricky operational issue is one that is still a little in-progress. Buttondown runs on Heroku, and uses a bunch of dynos. Think of dynos as akin to containers: everything on Heroku uses the same application bundle and the same mess of code, but each dyno is meant to correspond with a specific function or executable. For instance, Buttondown’s got the following Procfile, which is Heroku’s nomenclature for defining dynos:

web: bin/web
worker: bin/worker
checkerworker: bin/checkerworker
scheduler: bin/scheduler

Hello after a month of chaos! I am writing for you for the first time from my new house in Richmond, Virginia. In the past 28 days since my last missive, lots of non-Buttondown things have happened: a cross-country move, an unpack (178 boxes in total, because of course we counted), a whole lot of Telemachus playing fetch (and gaining a solid two pounds thanks to his grandparents spoiling him), and Buttondown has — well, there’s been less forward progress and more staying afloat.

A whirlwind tour of the highlights (and lowlights):

• Overall, things were… pretty stable? Growth continues apace and I am much worse at inbox management than I’d like to be (it’s been a while since I’ve hit inbox zero, but I’m fairly steadily at inbox ~thirty). No major incidents besides…
• Heroku went down. That was annoying, even if I’m a little pleased that I found out before their status page was updated. Still, one hour of downtime in ninety days is not worth seriously considering accelerating my AWS migration plans.
• I had some Redis scaling issues. These were boring scaling issues: if roughly ~500,000 emails were queued up to send then my instance would cap out and deadlock and what do you know, that happened. This was solved trivially by bumping up my instance to the next size, but that’s only delaying the problem’s event horizon to the next order of magnitude. I’ve been fantasizing about migrating off of Redis and using postgres-based queuing (RQ, which I use as a queuing library, is not particularly wonderful even if the API is nice) and this accelerated things somewhat.
• February’s roadmap was ambitious, and… did not go as planned. There was a lot going on, and the snatches of time I had to work on Buttondown were spent furiously going through email and keeping the lights on. Despite that, I shipped one big thing that I haven’t quite blogged about yet (and will next week!) — an entirely new subscribe form. Given that this is an input that a cool ten thousand or so people convert through every day, the fact that I was able to roll out the new experience with no fanfare and a huge amount of uplift is tremendously exciting.
• March’s roadmap, on the other hand, is much more reasonable — two small features (already shipped!) and two bigger ones which are roll-forwards from February. The vast majority of the moving part of moving is behind me, but it’s still going to be a busy month: I’ve got two and a half trips lined up in the next three weeks.

Hello hello! This past week was a quiet one: I mentioned that I was headed to the cabin with some friends, and cabin time was unproductive — which is to say it was time well spent. This week has been productive, but it’s also been start-of-the-month-productive: there’s been a lot of invoices being sent for processes that I’ve been a little too lazy to automate, there have been tweet threads to write and various marketing efforts to kick off, that kind of thing. (Today is perhaps emblematic of the entire week: four hours spent, which is pretty high, but two of those on operations, one on marketing, and one on actual development efforts.)

Two ‘big’ things on my mind:

First, I was lucky enough to score some time with Brian Lovin before he wised up and raised his prices. If you are interested enough in Buttondown to read this ol’ newsletter every week, I couldn’t recommend enough reading through his design critique — I was furiously nodding along through the entire thing, and have a slew of things to fix (both big and small) thanks to what he surfaced.

One of the things that he really brought to light for me, at a bit of a meta level, is how lost I’ve gotten in the day-to-day of working on things. I probably spend.... 7%? Of my time thinking about Buttondown as a gestalt, since so much is firefighting and keeping the lights on — which I mean in a positive way, since Buttondown has enjoyed such unalloyed growth, but it still means reaction rather than action. In particular, I think he pushed me in a really good way for some of the design system-y things that I’ve been thinking about, and the long-ballyhooed scaffolding work I haven’t merged yet is… going to remain unmerged. It is (cheerfully, since I didn’t share it with Brian) very much in line with where he was going, but I think the right thing to do is to keep it under wraps and keep iterating on it a bit longer.

I wrote last week about three painful queries that I needed to futz with: exporting all email events, aggregating client statistics, and time-series aggregations.

This slightly more methodical approach worked pretty well! To start with the good stuff:

1. Exporting email events for a big newsletter now takes O(seconds-to-minutes) instead of O(minutes-to-hours). This in of itself was the single most urgent thing: I can stomach analytics not working for certain newsletters, but the idea of someone not being able to get their data out of Buttondown was anathema. ¹
2. Client-level aggregations are now live and the “inbox statistics pane” loads quite fast. This is one of those lovely performance changes that actually have product-level implications, too — it’s now feasible for me to pull out specific clients used by a subscriber and display them live in the subscriber detail page, whereas previously that would have been a pretty expensive call.

Time-series aggregation is still…pretty painful. If you’re a newsletter with, say, twenty thousand subscribers and a year’s worth of emails on Buttondown, I’m no longer timing out when you try and get your analytics for the past quarter (yay!) but I’m definitely still timing out when you try to get your analytics for the past year (boo!)

Buttondown had its first planned partial outage of the year. (Of 2021, too, if my memory isn’t failing me — but, you know, new year and all that.)

This was to upgrade Buttondown’s database: a measly but stalwart db.t2.medium running on RDS, first spun up in 2017 and humming along ever since. ¹ The general parlance when thinking ab out relational databases is “people tend to underestimate how strong and scaleable a small, boring Postgres box can be” and I think that is true: this cute lil $29/month instance has brought me very few complaints over the past few years.

The upgrade was necessitated by two factors:

First, the database (cleverly named buttondown) was running Postgres 9.6, which Amazon was planning on formally deprecating and forcing upgrades. They’d pushed back the required upgrade window twice already, but I did not love the idea of waking up at 4am to a slew of pages because AWS decided when to take the database down.

Welcome to the first Weeknote of 2022! I’m fairly excited about this year (as much as one can be excited about any year) — the slate of work feels fun and well-formed, I’ve got some exciting personal stuff happening (moving!), and for the first time in a while I was able to actually spend the last two weeks decompressing to some extent (a marked difference from December 2020, which was quite fraught and stressful.)

My slate of January work is much more meager than my Q4 roadmap, which was — let’s be honest — overly ambitious. Here it is, in total:

• Launch new scaffolding
• Blog about the new scaffolding
• Add checker dashboard
• Remove slug uniqueness assumption
• Allow changing subscriber addresses
• Add recovery flow for spammy subscribers
• Allow both banning and unsubbing

That new scaffolding is almost ready for prime-time, and there’s a 60% chance or so that by next week it’ll be live, which will be the single biggest change to Buttondown’s interface since…ever? I think?

(Apologies for the off weeks for the past two weeks! Travel, et cetera.)

Checker, revisited

As the year winds down and I try not to alarm myself with the lack of progress on my Q4 goals (only two out of the six items I wanted to ship have shipped!), I amuse myself by reading through older issues of this newsletter and admiring the delta.

One such item was the checker infrastructure that I built in early April. I wrote about this as if it was dessert work: a fun challenge with low ‘real’ ROI:

A belated note about public archives

In my excitement and/or delirium last week, I forgot to mention that I managed to carve out and ship one of my six Q4 goals — a nicer public archive list! This was absolutely the easiest one of the six in terms of scope (just messing around with HTML and CSS for a while) as well as the most fun (just messing around with HTML and CSS for a while).

The rebuild itself was not particularly noteworthy: the pre-existing design was , and so the bar was low, but I mostly just spent five hours tweaking things until they looked particularly nice. A couple more interesting points:

Good on my promises.

I vowed last week to push out the list view changes, and I did! They’re now live. There are going to be some hiccups in making sure the design scales out to the bevy of use cases, but no big surprises or all-hands-on-deck-emergency-commits; it was as painless as I can hope from a substantial refactor of that nature, which I’m pretty pleased about. There will undoubtedly be a long tail of tiny fixes and tweaks over the next month, but that’s good — all of those will apply to the actual design system, rather than one-off components.

This week was split in twain.

I spent the first half in Seattle, as I tend to do, plodding away on the ListView work that I’ve spent the past two weeks chatting about.

First, I suppose, the good news:

Email filtering, such as it is, is done. My goal with the item is complete! My DataTable component has expanded and I now have a component that takes a bunch of stuff, and the page at exists solely as a wrapper around the . (Plus I added filtering, the whole point of the exercise — and the most trivial one.) There are a couple things I need to do before merging, like better type information and fixing where the dropdown menu appears, but it is Done.

I started my first piece of the Q4 roadmap yesterday. It might not look like much yet, but I am very excited about the following screenshot:

No big news, since I spent most of this past week trying to clean up the slate for the Q3 -> Q4 transition. A smattering of updates this week, then:

• I got the big subscriber views transition over the wire this past weekend! Around 85% of Buttondown’s pageviews are going through the new, HTML-based views: that remaining 15% is attributable to folks on custom domains (which had a now-fixed bug) and folks with custom CSS (who I need to manually ask to audit their CSS and then migrate over.) I am going to probably be very negligent at driving down the latter camp to 0 so I can finally unship the old code.

I whined last week about having a fairly unstructured Q3 despite my lofty goals, and I think Q4 is going to (hopefully?) be a bit better. You can take a look at my (unpublished/unfinalized, but mostly all there) Q4 roadmap — repricing and then a whole bunch of small follow-up work.

It feels a little bit like cheating to shift my gaze from Q3 to Q4 when I've got a handful of things that aren't quite done yet. The subscriber-facing views are technically "live" now, but I need to start actually ramping up the flag and prompting folks to update.

Where have I been for the past two months, you ask?

I have been active, I promise! I’ve been around!

I write a lot about the “good type of busy” and the “bad type of busy”. This was mostly the good type of busy: lots of new folks to onboard, lots of fires to put out.

It is funny, thinking about fires: I was reminded, cleaning up a part of the codebase from 2017, about the sheer I had the first time I had someone reliably sending out 10,000-subscriber emails on a weekly cadence. Some of the terror was, frankly, rational: one bad send with a few hours’ delay and I would have terrible blood pressure for a week. Email is tricky: it is temporal, it is irrevocable, and it is a scary thing to screw up.

Hello from the East Coast! I’ll be here for the next four weeks, which will be an interesting stress test on how productive I can be without my routines & contexts (to say nothing of the whole 34” curved monitor and desk setup.)

A week later, and I have written… well, I’ve written some content for the docs site. I have made good progress in general on the site, and I think it’s about ready to launch even if it’s not ready to call done — all of Notion’s content has been ported over, I’ve added the last bits of scaffolding such as a table of contents and stronger search metadata (as discussed last week), and it’s in a good place to just start iterating on.

I’m working on the hairiest part, technically speaking, of the documentation site — which is search functionality! This is slightly trendy at the moment — all the coolest kids are adding FTS to their docs — but is also tremendously useful (and replicates one of the really nice things about Notion hosting).

There are… a lot of options in terms of powering client-side search.

• . There’s something to be said for home-rolled search functionality; hell, that’s what I’m doing in Buttondown proper, though that’s largely because it’s a split between client side search (for paths) and server side search (for per-user objects). Still, the killer thing here is full-text search (since the documentation is going to be relatively long), and I think it’s much harder to do that than tokenized search.

It is OKR season at my day job and so it is OKR season in Buttondown as well. I think I am actually making a pretty good go of it, this time around:

Two meaty projects (maybe a month each?), two smaller (maybe one-to-two-weeks each?) projects.

Anyway, my goal for this week: pushing out some customer portal changes! My goal for the end of the week is to allow all premium subscribers to pause, upgrade, and downgrade their subscriptions: mostly this means a lot of transactional emails and web hook unit tests left to write.

I did it!

And, uh, not too much else. It was a week of emails (things are still fairly downbeat, but it happened to be a particularly voluminous week, the kind where the raw count is static but the “amount of thinking per email” spiked up), setting up a new laptop (I love my Mac mini to death, I really do, but the M1 MacBook Air feels almost comically performant in comparison. I bought it on a lark knowing that it was hard to justify, and yet I think I’ll be spending more time coding on it than on my Mac mini!), getting some nice press (I landed a fairly meaty mention in the , which is the kind of thing that I don’t know impacts me in a serious way besides SEO juice but certainly feels good!).

This coming week, there’s something vaguely time-sensitive on the horizon! Heroku is changing how they handle DNS registration for custom domains, which means I need to send out a bunch of relatively urgent (“take this action by July 31st…”) comms to everyone using custom domains. I’ll also use it as an opportunity to tweak how I’m handling the Heroku flow, since it relies on a couple bad assumptions (that get invalidated after this Heroku change) and is flaky in a painful way. So that (plus Roadmap V2, as aforementioned) should be on the ship list this week: though it’s going to be a long weekend and the start of a new month, which means I’ll be a little more spread thin than usual.

Hey, I did both of these things!

First, the Heroku changes. These were largely boring and good; I got to clean up some tech debt and improve the custom domain onboarding interface as I built it out, which is always nice. Exogenous event as impetus for tech debt pay down is rare, but I have to celebrate it when it happens.

That being said, it’s time for some dessert work! Maybe I’ll treat myself to cleaning up some of my dropdown CSS or messing around with the search bar UI, which has somehow already becoming stale. Either way, I’m going to grant myself a bit of a reprieve from the weekly goal: as long as I push the roadmap changes live, that’ll be good enough.

Well, it is 6.20pm on Sunday as I write this, and I have two items left on my list:

It turns out the trick to completing a daunting project is roughly threefold:

1. Whine about it a bit in various public forms
2. Ignore all outside stimuli (and by “all outside stimuli” I expressly mean Twitter and emails and not, say, a corgi mewling for attention)

Sorry for the radio silence last week. A double-dosed Weeknotes this time around, as recompense, focusing on two fairly divergent workstreams — one that has been occupying my coding time and one that has been occupying my thinking time.

Vue migration.

I’m desperate to onboard to Headless UI, a very useful set of components that let me unship a lot of cruft (I have home-baked popovers that are awful, I have home-baked modals that are awful, etc. etc.)

(Fool that I am, I wrote this on Sunday and checked it off of my todo list without actually scheduling the dang thing. Apologies!)

All of that is to say, I think the DNS work is a worthy goal for the week ahead. Once that lands, I think I’ll be in a good spot to spend May on dessert work: JavaScript exfiltration, support for attachments, that sort of thing.

This coming week, though — bug fixes. For real this time: I need to ameliorate some of the incoming customer service and bug fixes are the way to go. Honest! I’m building out a Things project and everything!

Hey, I did this! I picked out the five most noxious bugs (in terms of “time I have to spend answering questions around them” and also “embarrassment that they are extant bugs”) and by Sunday I had fixed four out of them. The fifth is — perhaps, dear reader, you are familiar with this one! — the whole “cursor randomly jumps to the top of the textarea” bug, which I am still as of yet unable to reproduce. There is even an entire about this of bug, which was both heartening (I am not the only one!) and disheartening (this may remain forever unsolved!)

I wrote last week that my Big Goal for this week was to take care of the three most noxious bugs I deal with and to, well, stay sane. Neither of these things happened! It was a very busy week — lots of support requests to answer, lots of day job work, lots of accumulated fatigue — and I mostly did my best to keep my head above water.

When I have tough weeks, I fall back into a familiar holding pattern: working on things that seem neat rather than important or urgent. This week’s subject of neatness is at least one that I’ve been discussing and thinking about at length lately:

On the face of it, this didn’t feel like a productive week. I did get two things of note done:

1. I shipped the standalone checker infrastructure I mentioned. I punted on some of the trickier (and less important) architectural questions, like how to declare reactions to check failures; I realized it was more important for me to just get the infrastructure live and working and that more nuanced iterations would only come with experience. (To wit: there have been a lot of fast-follows, such as me immediately adding the ability to track how long a check has failed or being able to re-run checks through the admin interface.)

Pay-what-you-want is live! There are still some fast-follows that I want to add, as alluded to in the blog post, but overall I’m satisfied with things. (In the…48 hours since public launch, I’m sitting a little shy of $5,000 being committed through PWYW, which is higher than I expected!)

It is always tricky to know when to call it a day here. I could have spent this week working on fast-follows, and the fact that so many of them have been explicitly called out in questions I’ve received post-launch is a sign that maybe I should have spent the extra day documenting upsell CTAs or adding more details to the post-upsell email. All of those tasks feel granular and snackable, though: they don’t require the dedicated level of flow and focus that actually hunkering down and shipping PWYW did.

I got a surprising amount of work done in the past week on pay-what-you-want! This is good. This kind of feature work is not what I am particularly good at lately: the ratio of “staring and thinking” to “coding” weighs heavily in the favor of the former, and I am guilty of getting bored of staring and thinking and trundling off to some well-scoped problem that I can just crank out in a few hours.

In fact, if you wanted to, you could enable PWYW right now — it’s live in Paid Subscriptions. This is also the kind of feature where me saying “oh I’m done!” presages the majority of the work. Wiring together the billing code and the settings code is the easy part: then it’s like, oh right I need to update four FAQ pages and write a blog post and most of the subscription interface is built on the assumption that everyone is on the same price so I need to add more details to the NotesModal and also the subscription confirmation email and I’ve been to revamp the premium subscription email to look nice and have GIFs so I might as well roll that into this work and oh this means I need to start slurping up custom plans from Stripe to populate the various modals and emails so — [exhales].

There are quite a few apocryphal quotes around “overnight success taking ten years”, and that’s kind of what this past week felt like: it took me I think around a year and a half to get my first $1,000 MRR on Buttondown (still one of my proudest moments as a developer ever!) and then things happen and I get a marginal $1,500 MRR in the span of a single week. Phrasing this as “MRR went up nearly 25%” is less visceral and more directionally accurate, but also much less fun.

Mostly what that meant was randomization: I wrote last week about my humble ambitions of puttering around on some small paper cuts and tech debt tasks while I set my brain to work thinking about a revised analytics stack in the background. Neither parts of that came to pass: I spent a tremendous amount of time on the operational and onboarding side (pulling in archives is easy; matching existing Stripe subscriptions is less so) and what little time remained was on figuring out what a new revised roadmap would look like.

“Change your roadmap entirely based on an influx of one specific customer demographic” is kind of a giant warning sign, and I don’t think I did anything too drastic: the on my new list are things that I slated for this half anyway, they just seem suddenly new and more important.

Last week, I wrote:

There is some bug bashing that needs to take place, but I think finally tackling multiple-newsletter creation in a less janky way than “log out, create a new newsletter, and paste that API key in” feels like a good candidate. (And no, of course that’s not because I received two support emails about it this morning. Why would you even ask that?)

And I appear to be in the position now for the second week in a row that a chunk of work I pegged at being a solid week’s worth of work ended up being about an hour’s. When I thought about it in the abstract, “newsletter creation from the perspective of an already-registered user” felt hairy and complex and prone to lots of yak-shaving. When I actually sat down to work on it and start figuring out the data modeling, it turns out it’s quite simple: newsletters need a name and a username to be created, and that’s it. The rest can be handled just by the existing settings interfaces (which, don’t get me wrong, need some TLC, but that’s an orthogonal problem) — there’s no point recreating all of those form inputs just to have them in a multi-stage modal or whatever.

Last week, I mentioned that I was gearing up for a week of frustrating (and hopefully rewarding) Heroku spelunking: dusting off the proverbial (and I guess literal — I had to update a couple dependencies) toolbox and going to town.

…

It was a one-liner, I think.

Not even a one-liner, the kind of one-line change that you arrive at with a sense of exhausted satisfaction after many hours down in the Performance Mines: a stupid, idiotic one-liner, a “oh my god of it’s this issue, why didn’t I just look at this sooner”. The in this case was a timeout set by , the web server I run around Django — it was set to kill any requests after , whereas Heroku had a hard timeout of .

The coy way to phrase how this week went is that I blew my Heroku budget out of the water. I woke up on Friday morning to discover that I had been Fireballed, something that my 20-year-old self would have been much more thrilled about but is still none-the-less very cool and very valuable. Getting a low-orbit cannon of traffic thrown at the site is useful in two dimensions:

This has been quite the week!

I’m in the final stages of a pretty fun high touch sales deal. I spent twelve hours assembling IKEA furniture in my basement. I shipped Importing V2. My dog got neutered. I’m in the final stages of delivering on a very fun and philosophically well-placed partnership with an analytics provider.

(Okay, some of those aren’t related to Buttondown.)

Buttondown’s newest mascot.

My latest of reasons for being bad at writing is the most adorable yet: Telemachus, my ten-week-old corgi puppy. He is a very sweet dumb fluffball, with some strengths:

Customer service, that old devil.

I was relatively sunny and awash with optimism last week. I owe a lot of this to a new strategy I was trying out to preserve focus, which was to ignore my inbox except in the mornings and evenings: rather than keep the inbox open in a tab all day and respond reactively (and get randomized) by emails, I was going to just batch-respond twice a day and ignore my email otherwise.

This was working very well until Thursday morning, on which I awoke to....72 emails, what I think is a new record. These weren’t all bad emails — some were pleasant compliments, some were error notifications, some were interview requests. But a lot of them were serious and required work, and it was all a little overwhelming.

And for those of you reading this email who might have been in the cohort of 72, I promise you: I love you, and don’t bedrudge you at all! This is a me problem, not a you problem.

Logging and monitoring misadventures.

Buttondown has been slow and flaky recently. I feel pretty confident that this is due to increasing volume: the ‘odd’ HN hug has become a daily occurrence, and the load coming from Mailgun/SES, while spiky, scales linearly with traffic. This is manifesting in a painful ways: spats of timeouts and dropped connections.

Previously, this has been a problem I’ve scaled my way out of: Buttondown sits on a half-dozen Heroku boxes and then I bump that up to a dozen until things stop complaining. That steady state has shifted, though: instead of going from 6 to 12, I’m finding myself going from 12 to 24, and even then there’s a lot of pain.

This is not uncommon, and there’s a playbook for this:

Roadmap angst.

So, first, the exciting news. (Well, exciting news for me): I’m taking next week off!

This, of course, means I’ll be spending more time than usual on Buttondown, retreating to the figurative backyard shed.

I was poking around my roadmap finding a couple interesting tasks to work on with my time. I think I’ll be able to fully ship internationalization and the remaining marketing work, but wanted a nice meaty project for the back half of the week: revised billing? Multiple users per account? A redesigned writing interface?

Marketing, launched!

For once I was true to my word, and I launched.

In no real order, some takeaways thus far:

Marketing, hovering over the “merge” button.

It is so tempting:

Marketing V3, almost ready.

Feast upon these glorious screenshots:

Apologies for the late weeknote. This has been a low-energy, high-email week, where I would look at the end of my to-do list, see “write Weeknote” alongside three other things, and sigh to myself as I opt to do precisely none of them. Sometimes you need to have low-bandwidth weeks!

I’m reminded of one of my favorite pieces by Patrick McKenzie, my coworker:

First: hello from my new iPad keyboard. I’ll spare you the requisite bloviating about yet another piece of Apple hardware, but I quite like this one, and it means I can be a little more efficient shifting more and more non-development work away from my desktop. Notion works well; Mail works well; Zendesk works well. And thanks to Juno, I can even do some limited diagnostics/data exploration work!

Here’s a thing that I’ve known to be true for a while but have found to be particularly true in the past few weeks, as all element of my life bleed together: two hours is not the same thing as four chunks of thirty minutes.

I spent this weekend refactoring how Buttondown handles archival imports!

First, what are archival imports?: if you bring your newsletter over from Mailchimp or Tinyletter (or, now, Substack, but we’ll get to that later) I have some janky logic to import your archives. This was, prior to this weekend, done in the span of an HTTP request, which was, uh, non-ideal: if you had a large (or even a not-so-large) archive, your import would probably time out and/or silently fail, leading to a customer email, leading to me importing your archives manually, which was pretty easy but definitely frictional.

A good rule of thumb I learned a number of years ago is to get the number of “things” in an HTTP request as low as possible: “things” are database calls, sure, and third-party requests, and file reads, and pretty much . HTTP requests should be granular, instant, and fail very rarely — all the important stuff associated with them should happen after the fact. My archival imports broke this rule with wild abandon.

Reckless spending.

Buttondown has a new Twitter handle, which also marks my first (and, if I were to bet on it, last) foray into social media account purchasing.

I had had for a while and it was, you know, . The success or failure of an enterprise is not determined by the concision of its social media identifier, you know? But, because I am foolish, I wanted , and had a lingering to-do item to try and track down the owner.