The intent of this proposal is two-fold
1. To clarify what kind of messaging ordering guarantees an implementation must or could provide
2. To state a set of normative requirements that need to be met in order to provide these guarantees in an interoperable way.
The input spec has a section on message ordering, but it is unclear whether it's intended to be normative or just informative, so we need to clarify that. While many applications will benefit from some kind of end-to-end message ordering guarantee, there are some which don't rely on the system preserving message order, either because they send messages so infrequently that it is unlikely that they would get out of order, or because the order genuinely doesn't matter, or because they are ordering messages themselves - e.g. by a timestamp. There are potential performance advantages to server implementations if they aren't required to enforce message ordering.
It's also unclear exactly what we mean by message ordering. I therefore propose that we
- introduce the concept of an "Ordered Topic", which preserves the order of any messages published by a given publisher at a given QoS level
- require that every server must provide support for ordered topics (this should be the default)
- permit a server to offer less stringent forms of ordering on one or more topics, should it wish to
- require that all clients obey a set of rules that make sure that they can operate with either ordered or unordered topics without having to be aware of whether the topic is ordered or not
- provide a non-normative description of what we mean by ordering for QoS 1.
To do this I propose we replace the current section 4.5 (Message Ordering) with the following text...
A client MUST follow these rules when implementing the protocol flows defined elsewhere in this chapter:
When it resends any PUBLISH packets, it MUST resend them in the order in which the original PUBLISH packets were sent (this applies to QoS 1 and QoS 2 messages)
It must send PUBACK packets in the order in which the corresponding PUBLISH packets were received (QoS 1 messages)
It must send PUBREC packets in the order in which the corresponding PUBLISH packets were received (QoS 2 messages)
It must send PUBREL packets in the order in which the corresponding PUBREC packets were received (QoS 2 messages)
A server MUST by default treat each Topic as an "Ordered Topic". It MAY provide an administrative or other mechanism to allow one or more Topics to be treated as an "Unordered Topic".
When a server processes a message that has been published to an Ordered Topic, it MUST follow the rules listed above when delivering messages to each of its subscribers. In addition it MUST send PUBLISH packets to consumers (for the same Topic and QoS) in the order that they were received from any given client.
Non-normative comment
The rules listed above ensure that when a stream of messages is published and subscribed to with QoS = 1, the final copy of each message received by the subscribers will be in the order that they were originally published in, but the possibility of message duplication could result in a resend of an earlier message being received after one of its successor messages. For example a publisher might send messages in the order 1,2,3,4 and the subscriber might receive them in the order 1,2,3,2,3,4.
If both client and server make sure that no more than one message is "in-flight" at any one time (by not sending a message until its predecessor has been acknowledged), then no QoS 1 message will be received after any later one - for example a subscriber might receive them in the order 1,2,3,3,4 but not 1,2,3,2,3,4. Setting an in-flight window of 1 also means that order will be preserved even if the publisher sends a sequence of messages with different QoS levels on the same topic.
Improvement
Major