2015: A Ghost of Fastmail Future

25 Dec 2015

We don’t usually talk about upcoming plans, because they are always subject to change, and with programming brand new features it’s particularly difficult to know how long a task will take, because it depends how many unknown pitfalls you hit along the way.

Looking into the future

In two different ways! I’m looking into the future of the Cyrus IMAP server, as it looks into the future for tasks to do.

For my part I’m confident that what I’m about to describe will be functioning by next Christmas – or something better. It’s the right time, and I know it is because I almost didn’t go in the office on December 8th – between my daughter’s school orientation in the morning and a house inspection in the afternoon I only had 2 hours time in the office.

I’m glad I did though, because we were all a buzz about the closing of Mailbox. We had been hoping that they would support generic IMAP and work with Fastmail, but I guess it was never to be.

One of their most popular features was snoozing emails. We’ve been talking about how to do that, as well as delayed send (also undo send, which is just some sugar on top of a very short delayed send), and we grabbed a whiteboard and started working through ideas. We know we need to implement all those things for JMAP with its concept of an outbox folder for sending messages anyway.

At least once

For our push notifications it doesn’t matter if the occasional notification is lost, because the next one will sync up state just fine. So losing one in every ten thousand is no problem.

For snoozed emails or delayed send, that’s not acceptable.

In a distributed system (like the Fastmail platform) it is impossible to guarantee that a message will be processed exactly once. Email via SMTP handles this by each server not deleting its copy until it has positive delivery confirmation from the next server. In failure cases, it may deliver the same message twice, and the receiver is responsible for deduplication.

So our problem: how do we guarantee that the entire task has finished sucessfully before we mark an email as no longer needing to be delivered? The notify channel is not robust against all possible failures, even before it goes through the external pusher at Fastmail.

The answer – store the event and keep sending it until something (the final processing code) marks the task as completed.

Leveraging synergies

Behind the buzzword bingo of someone else’s language there is often some sense (we don’t have out-of-touch bosses at Fastmail, so we only use management buzzwords ironically while we’re waiting for our coffee).

We spent a lot of effort trying to work out how we might extend the replication protocol to track messages that had been processed, and what failover might look like. But the answer was to look at what we already have. We’ll create a top level mailbox called #events (per store), and inject all events related to messages that need to be moved back to Inbox or sent later into there, with an INTERNALDATE flag set to the action time.

Reading events from this mailbox can be as simple as an external admin client running IDLE on the mailbox, with a search for “BEFORE {now}” to find messages to process immediately. Once they are successfully processed, the external task just deletes them.

This is blindingly simple, low-tech, and most of all robust. In the worst case, the message has been sent or moved to Inbox, and it gets sent or moved again. That’s just a duplicate message which all SMTP receivers already have to deal with.

Show me the code

It’s Christmas, OK – so I don’t have much yet. Just a couple of commits which, in my testing, generate a message in #events listing the Outbox message and with the INTERNALDATE correctly set to the INTERNALDATE of the actual message in Outbox – so an append works. Commits are here:

Look forward for more work with future event support in Cyrus in the new year, and then features in Fastmail built on top of this support after that. And now I’m going to sleep.

Merry Christmas!