Over the last 24 hours, we’ve rolled out a new POP/IMAP server version for all users. This new server is the result of months of great work by Bron and includes many improvements and fixes. Not that many of the fixes are currently user visible changes, but they are significant internal improvements that help improve reliability, conformance and performance, and will allow us to build some future features we’re looking at.
Email replication improvements
Email replication has been made much more efficient and reliable.
The format includes
auto-integrity checking features, so that any unexpected mismatches
between both ends are automatically detected and fixed. It can also
recover automatically from unclean shutdowns or machine crashes
where “split brain” has occurred, automatically fixing up mailboxes
and messages. The format has also been made future extensible,
allowing more features to be added without compatibility problems.
Performance and integrity improvements
The internal mailbox format used to store emails has been
significantly reworked. The new format has reliable locking
semantics to remove all race
conditions. It also
stores and checks
CRCs on all
record data and cache data, and SHA1 checks on all message files.
This ensures that any corruption in any data is detected early and
can be dealt with. By moving around some of the data (such as the
user seen state), and only lazily opening files as needed, the new
format also improves performance in many common cases.
Strict MODSEQ, QRESYNC support and full IMAP test suite conformance
Recent extensions to IMAP allow clients to more quickly synchronise
data between the server and the client (eg.
QRESYNC). While the
server has supported CONDSTORE/MODSEQ for a while, unfortunately it
was a bit buggy in some situations, causing message seen state to
get out of sync. The server now correctly and accurately support
CONDSTORE/MODSEQ, and also supports the current QRESYNC standard
that will allow clients that support it to sync even faster. We also
now correctly pass detailed IMAP stress tests.
Major code cleanups
All of these improvements have also been done with major internal
code cleanups. This will allow us to continue building additional
functionality and features more easily in the future, and to more
easily fix and debug any other issues that are encountered.
Unfortunately no good deed goes unpunished, and even though we’ve been testing this code ourselves and on a sub-set of users for weeks with continuous improvements, unfortunately some bugs did get through when we finally rolled out to all users. Then in the attempt to fix these issues as quickly as possible, we also introduced some other issues. The net result was that for about 12 hours, there was a sequence of small but potentially annoying bugs that would have affected different sets of users.
- On first access, we upgrade a mailbox to the new format. During the
upgrade, we found some existing caches had allowed invalid data to
enter them, causing corruptions on upgrading which caused problems
when accessing these mailboxes. These cases are now caught and new
cache data is built from the underlying message files
- While reconstructing the mailboxes that had been incorrectly
upgraded by the above code, a quota error caused some peoples quota
to temporarily be double their actual used amount. This has been
fixed now. If this bug sent a user over quota temporarily, it
shouldn't be a problem. When a user is over quota, we return a
temporary 4xx error, which means no messages should have been lost,
the other side should just have re-delivered when they were back
- IMAP IDLE wasn't returning new messages, only updating existing
messages, causing pushing of new messages to most email clients to
- Mail App has a bug with parsing IMAP IDLE unsolicited fetch
responses that contain more than flags information. We've added a
workaround for this Mail App bug
- The IMAP COPYUID response was producing a non-conformant result,
which caused some programs to report an error (Outlook 2010)
- POP3 was using an optimised mode if a mailbox was empty.
Unfortunately the code to mark a mailbox as "non empty" wasn't
working properly when messages were delivered, but was working for
IMAP logins. This meant that messages delivered wouldn't be
downloaded by POP until you did an IMAP or web login
- The POP3 TOP command wasn't working, causing some programs (Outlook
in POP mode) that download email headers to fail
- The POP3 UIDL command with a message ID was producing a
non-conformant result, which was parsed incorrectly by some
programs. This caused some POP programs to download the same message
more than once, or to delete off the server before it should have
- Update: An update to UID sequence handling caused the mailbox
status command to report unread messages as read and vice-versa,
causing the unread count on folders to actually be the read count
for a short while.
- Update: The XLIST extension wasn't working. This has been added
back, so client that support it will automatically pick the right
Sent Items, Drafts, Trash, Junk Mail folders when setting up a new
- Update: NOOP on Mac Mail. Like the bug above with Mac Mail and
IDLE, this was affecting the NOOP command as well
- Update: Storing the \Seen flag + another flag on a message that
already had the \Seen flag would cause \Seen to actually
disappear. This mostly manifested as when deleting a message, it
would cause it to become marked as "unread" again
All these issues have now been fixed, and we’re closely monitoring all the server logs to see if there’s any other issues, but at this stage we believe that the new server and code is working correctly for all cases we’re aware of and for all clients, IMAP and POP.
All this new code is part of the open source project cyrus, and we’ll be pushing this code back to the main cyrus code base, which will eventually form the basis for a new cyrus version 2.4. For those interested in technical details, Bron will post to the cyrus mailing lists when he’s had a bit of time to compile all the documentation and technical details.