Over the last 24 hours, we’ve rolled out a new POP/IMAP server version for all users. This new server is the result of months of great work by Bron and includes many improvements and fixes. Not that many of the fixes are currently user visible changes, but they are significant internal improvements that help improve reliability, conformance and performance, and will allow us to build some future features we’re looking at.

  • Email replication improvements

    Email replication has been made much more efficient and reliable.
    The format includes
    auto-integrity checking features, so that any unexpected mismatches
    between both ends are automatically detected and fixed. It can also
    recover automatically from unclean shutdowns or machine crashes
    where “split brain” has occurred, automatically fixing up mailboxes
    and messages. The format has also been made future extensible,
    allowing more features to be added without compatibility problems.

  • Performance and integrity improvements

    The internal mailbox format used to store emails has been
    significantly reworked. The new format has reliable locking
    semantics to remove all race
    . It also
    stores and checks
    CRCs on all
    record data and cache data, and SHA1 checks on all message files.
    This ensures that any corruption in any data is detected early and
    can be dealt with. By moving around some of the data (such as the
    user seen state), and only lazily opening files as needed, the new
    format also improves performance in many common cases.

  • Strict MODSEQ, QRESYNC support and full IMAP test suite conformance

    Recent extensions to IMAP allow clients to more quickly synchronise
    data between the server and the client (eg.
    QRESYNC). While the
    server has supported CONDSTORE/MODSEQ for a while, unfortunately it
    was a bit buggy in some situations, causing message seen state to
    get out of sync. The server now correctly and accurately support
    CONDSTORE/MODSEQ, and also supports the current QRESYNC standard
    that will allow clients that support it to sync even faster. We also
    now correctly pass detailed IMAP stress tests.

  • Major code cleanups

    All of these improvements have also been done with major internal
    code cleanups. This will allow us to continue building additional
    functionality and features more easily in the future, and to more
    easily fix and debug any other issues that are encountered.

Unfortunately no good deed goes unpunished, and even though we’ve been testing this code ourselves and on a sub-set of users for weeks with continuous improvements, unfortunately some bugs did get through when we finally rolled out to all users. Then in the attempt to fix these issues as quickly as possible, we also introduced some other issues. The net result was that for about 12 hours, there was a sequence of small but potentially annoying bugs that would have affected different sets of users.

  • On first access, we upgrade a mailbox to the new format. During the
    upgrade, we found some existing caches had allowed invalid data to
    enter them, causing corruptions on upgrading which caused problems
    when accessing these mailboxes. These cases are now caught and new
    cache data is built from the underlying message files
  • While reconstructing the mailboxes that had been incorrectly
    upgraded by the above code, a quota error caused some peoples quota
    to temporarily be double their actual used amount. This has been
    fixed now. If this bug sent a user over quota temporarily, it
    shouldn't be a problem. When a user is over quota, we return a
    temporary 4xx error, which means no messages should have been lost,
    the other side should just have re-delivered when they were back
    under quota.
  • IMAP IDLE wasn't returning new messages, only updating existing
    messages, causing pushing of new messages to most email clients to
    not work
  • Mail App has a bug with parsing IMAP IDLE unsolicited fetch
    responses that contain more than flags information. We've added a
    workaround for this Mail App bug
  • The IMAP COPYUID response was producing a non-conformant result,
    which caused some programs to report an error (Outlook 2010)
  • POP3 was using an optimised mode if a mailbox was empty.
    Unfortunately the code to mark a mailbox as "non empty" wasn't
    working properly when messages were delivered, but was working for
    IMAP logins. This meant that messages delivered wouldn't be
    downloaded by POP until you did an IMAP or web login
  • The POP3 TOP command wasn't working, causing some programs (Outlook
    in POP mode) that download email headers to fail
  • The POP3 UIDL command with a message ID was producing a
    non-conformant result, which was parsed incorrectly by some
    programs. This caused some POP programs to download the same message
    more than once, or to delete off the server before it should have
  • Update: An update to UID sequence handling  caused the mailbox
    status command to report unread messages as read and vice-versa,
    causing the unread count on folders to actually be the read count
    for a short while.
  • Update: The XLIST extension wasn't working. This has been added
    back, so client that support it will automatically pick the right
    Sent Items, Drafts, Trash, Junk Mail folders when setting up a new
  • Update: NOOP on Mac Mail. Like the bug above with Mac Mail and
    IDLE, this was affecting the NOOP command as well
  • Update: Storing the \Seen flag + another flag on a message that
    already had the \Seen flag would cause \Seen to actually
    disappear. This mostly manifested as when deleting a message, it
    would cause it to become marked as "unread" again

All these issues have now been fixed, and we’re closely monitoring all the server logs to see if there’s any other issues, but at this stage we believe that the new server and code is working correctly for all cases we’re aware of and for all clients, IMAP and POP.

All this new code is part of the open source project cyrus, and we’ll be pushing this code back to the main cyrus code base, which will eventually form the basis for a new cyrus version 2.4. For those interested in technical details, Bron will post to the cyrus mailing lists when he’s had a bit of time to compile all the documentation and technical details.