I was really hoping that I would have some sort of juicy insight or breakthrough to share here. I… do not.
The good news: it appears to be happening somewhat less frequently than last week. I can’t point to any reason why that might be the case: I’ve added some additional instrumentation and logging to the scheduler dyno, but haven’t actually changed anything. (To wit: the scheduler’s crashed exactly once in the past four days, as opposed to roughly once every eighteen hours last week.)
It would be lovely (albeit unsatisfying) if this was an issue that evaporated on its own, like so many transient hardware-ish issues I’ve dealt with in the past. In the meantime, I’m just grateful I haven’t been woken up by any pages.