From Cyrus
Cyrus Murder Mupdate Details
Cyrus Murder Mupdate Details
Please note that this document is not an authoritative MUPDATE protocol reference. For that, you should refer to RFC 3656.
The Mailbox Update (MUPDATE) protocol is the backend that allows the CyrusMurder to function with a unified mailbox namespace, despite having multiple frontend and backend servers. This document is to serve as a reference guide to that protocol.
Past implementations of the aggregator made use of ACAP as the mailbox backend protocol. MUPDATE is designed to be more lightweight than ACAP while at the same time allowing the database to be distributed across multiple servers.
1.0 Participants
Within an IMAP aggregator, there are several types of MUPDATE participants.
- MUPDATE Master - This component maintains the master copy of the MUPDATE database and allows operations on the database to be atomic.
- MUPDATE Slaves - These components connect to the master and maintain a replica of the MUPDATE database. They also accept connections from clients, though these connections are of a read-only nature. Note that there are short periods of time between an update occuring on the master and it being reflected across all of the slaves.
- MUPDATE Clients - These components query the master and slaves to obtain information from the mailbox database.
2.0 Data Formats
MUPDATE uses data formats similar to those used in [IMAP]. That is, it uses Atoms and Strings. All commands and tags in the protocol are transmitted as atoms. Every other piece of data contained within the protocol is a string, and must be quoted as such.
2.1 Atoms
An atom consists of one or more alphanumeric characters.
2.2 Strings
Just like [IM (and a ready response is not necessary), a close brace ("}"), and CRLF. If the plus sign is omitted, then the recieving side MUST send a "+ go ahead" response.
A quoted string is a sequence of zero or more 7-bit characters, excluding CR and LF, with double quote (<">) characters at each end.
The empty string is represented as either "" (a quoted string with zero characters between double quotes) or as {0} followed by CRLF (a literal with an octet count of 0).
3.0 Server Responses
There are three responses that the server may issue to a client for every command, and 2 responses that are only valid in response to a FIND, LIST, or UPDATE, and 1 response that is only valid for UPDATE. All responses are tagged by the same tag as the command that caused them.
3.1 Response: OK
A tagged OK response indicates that the operation completed successfully. There is a mandatory implementation-defined string attached with the OK response that may be used to give the user additional information.
3.2 Response: NO
A tagged NO response indicates that the operation was denied by the server. There is a mandatory implementation-defined string attached with the NO response that may be used to explain the cause of the denial.
3.3 Response: BAD
A tagged BAD response indicates that the command from the client could not be parsed. There is a mandatory implementation-defined string attached with the BAD response that may be used to explain the cause of the failure.
3.4 Response: RESERVE
A tagged RESERVE response includes two parameters, the name of the mailbox and a location identifier in the format "server!partition". This response is valid for FIND, UPDATE, and LIST commands. It indicates that the mailbox name is currently reserved in the namespace, however the mailbox does not currently exist.
3.5 Response: MAILBOX
A tagged MAILBOX response includes three parameters, the name of the mailbox, the location identifier (as with RESERVE), and the ACL of the mailbox. This response is valid for FIND, UPDATE, and LIST commands. IT indicates that the mailbox named is currently active on the given server with the given ACL.
3.6 Response: DELETE
A tagged DELETE response includes one parameter, the mailbox name. It indicates that the given name has been freed from the namespace and is no longer available. It is valid only in response to an UPDATE command.
3.7 Server Initial Response
The inital response from the server is a two line format. The first line MUST start with "* OK MUPDATE" and be followed by three strings: the server's hostname, an implementation-defined string, and the version of the implementation (also implementation-defined).
The second line of the inital response begins with "* AUTH" and is followed by a space-separated list of [SASL] mechanisms that the server will accept.
4.0 Commands
The following are the valid commands that a client may send to the MUPDATE server: AUTHENTICATE, ACTIVATE, DELETE, FIND, LIST, LOGOUT, NOOP, RESERVE, and UPDATE. Only AUTHENTICATE and LOGOUT may be issued before a successful authentication has occured. Only LOGOUT and NOOP may be issued after UPDATE has been successfully issued. Only one successful AUTHENTICATE command may be issued per session.
4.1 AUTHENTICATE Command
The AUTHENTICATE command initiates a [SASL] negotiation session between the client and the server. It has 2 parameters, the first is mandatory and is the string for the authentication mechanism desired. The second is the contents of the optional client first send. All SASL blobs sent across the wire must be in base64 encoded format, and followed by a CR and LF combination. Clients may cancel authentication by sending a * followed by a CR and LF.
4.2 ACTIVATE Command
The ACTIVATE command takes three parameters, a mailbox name, a location in "server!partition" format, and an ACL for the mailbox. It will tell the server to insert into its database the given mailbox in the given location with the given ACL. An OK response indicates success, a NO response indicates some sort of failure occured. This is not a valid command to issue to a slave.
4.3 DELETE Command
The DELETE command takes one parameter, a mailbox name. The server should delete the given mailbox from its namespace, and return OK. A NO response indicates that either the session is not currently authenticated or the given mailbox does not exist. The delete command should immediately be sent to all slave databases. This is not a valid command to issue to a slave.
4.4 FIND Command
The FIND command takes a single paramter, a mailbox name. The server then responds with the current record for the given mailbox, if any and an OK response. A NO response is suitable if the connection is not currently authenticated.
4.5 LIST Command
The LIST command is similar to running FIND across the entire database. The LIST command takes an optional parameter, which is a prefix to match for the location field. Without the parameter, LIST returns results for all mailboxes in the database. For each mailbox that matches, it issues a MAILBOX or RESERVE response to the client. When all responses are complete, it issues an OK response.
4.6 LOGOUT Command
The LOGOUT command tells the server to close the connection. Its only valid response is an OK response.
4.7 NOOP Command
The NOOP command takes no parameters. An OK response indicates success. If this command is issued after an UPDATE command, then the OK response also indicates that all pending transactions have been sent to the listening slave.
4.8 RESERVE Command
A RESERVE command takes two parameters: the mailbox name and a location in "server!partition" format. It will first check for an existing reservation or activation of the given mailbox name, if there does exist such an entity, a NO response is returned, otherwise a reservation entry is put into the database, and an OK response is returned on success. NO is also returned if the connection is unauthenticated. This is not a valid command to issue to a slave.
4.9 UPDATE Command
The UPDATE command is how a slave initiates an update stream from the master (though it is vaild to issue the command to a slave as well). In response to the command, the server returns a list of all mailboxes (as a LIST command with no parameters) and then an OK response. From this point forward whenever an update occurs to the master's database, it will stream the update to the slave (that is, it will send a RESERVE, MAILBOX, or DELETE response as the updates happen to its database). Only NOOP or LOGOUT are valid after this command is issued. This command may only be issued in the authenticated state.
5.0 Database Synchronization
A distributed database protocol such as mupdate must address the issues of synchronization. In our case, there are 2 places where the database can become out of sync if a connection is not dropped cleanly. For one, the master mupdate database can reflect the existance of mailboxes in the namespace that do not exist (or vice versa), and the slaves need to ensure that their local database is up to date with the master whenever they come up.
5.1 Synchronization of Slaves to Master
Synchronization of slave databases to the master's database is simple. When the slave issues the UPDATE command to the master, the master dumps the entire contents of its database to the slave. The slave may then use this information to verify and update its localt database.
5.2 Synchronization of Master to Backend Servers
When server that holds mailboxes in the namespace comes up, it should perform a LIST operation against the master mupdate server, and then issue ACTIVATE and DELETE commands as necessary to bring the master's list of what is on this particular backend servers into sync with what actually exists on the backend server. Note that this document does not specify how to deal with conflicts (where one mailbox resides on multiple backend servers).
Appendix A: References
- [IMAP] M. Crispin, "Internet Message Access Protocol", RFC 2060, December 1996.
- [ACAP] Newman, Myers, "ACAP -- Application Configuration Access Protocol", RFC 2244, November 1997.
- [SASL] Myers, "Simple Authentication and Security Layer (SASL), RFC 2222, Octover 1997.