OASIS Message Queuing Telemetry Transport (MQTT) TCIcon indicating the project type

OASIS Message Queuing Telemetry Transport (MQTT) TC

Uploaded image for project: 'OASIS Message Queuing Telemetry Transport (MQTT) TC'
  1. OASIS Message Queuing Telemetry Transport (MQTT) TC
  2. MQTT-27

Handling of the Dup Flag (section 2.1.2.1)

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.1
    • Fix Version/s: 3.1.1
    • Component/s: core
    • Labels:
      None
    • Proposal:
      Hide

      Edit lines 340-343 (WD12)

      This flag MUST be set to 1 by the client or server when it attempts to re-deliver a PUBLISH Control Packet.
      The Dup flag MUST be 0 for :

      • all QoS 0 messages
      • PUBREL, SUBSCRIBE and UNSUBSCRIBE Control Packets.

      If Dup 0 then the flow is the first occasion that the client or server has attempted to send the MQTT PUBLISH Packet. If Dup 1 then the this indicates that the flow might be re-delivery of an earlier packet. The DUP flag MUST not be propagated when the PUBLISH message is sent to subscribers by the server.

      Change non-normative comment in 2.2.2.1 (in WD12 from line 337).

      Current words:

      If 1, the recipient might have previously received the Control Packet it should not treat this flag as an indication that it has previously received the MQTT Control Packet and should not use the DUP flag alone to guarantee to its application that a duplicate message is being redelivered to its application interface.

      New words:

      The recipient of a Control Packet that contains the DUP flag set to 1 cannot assume that it has seen an earlier copy of this packet. It is important to note that the DUP flag refers to the Control Packet itself and not to to the application message that it contains. When using QoS 1, it is possible for a client to receive a PUBLISH packet with DUP set to 0 that contains a repetition of an application message that it received earlier, but with a different MessageID.

      Show
      Edit lines 340-343 (WD12) This flag MUST be set to 1 by the client or server when it attempts to re-deliver a PUBLISH Control Packet. The Dup flag MUST be 0 for : all QoS 0 messages PUBREL, SUBSCRIBE and UNSUBSCRIBE Control Packets. If Dup 0 then the flow is the first occasion that the client or server has attempted to send the MQTT PUBLISH Packet. If Dup 1 then the this indicates that the flow might be re-delivery of an earlier packet. The DUP flag MUST not be propagated when the PUBLISH message is sent to subscribers by the server. Change non-normative comment in 2.2.2.1 (in WD12 from line 337). Current words: If 1, the recipient might have previously received the Control Packet it should not treat this flag as an indication that it has previously received the MQTT Control Packet and should not use the DUP flag alone to guarantee to its application that a duplicate message is being redelivered to its application interface. New words: The recipient of a Control Packet that contains the DUP flag set to 1 cannot assume that it has seen an earlier copy of this packet. It is important to note that the DUP flag refers to the Control Packet itself and not to to the application message that it contains. When using QoS 1, it is possible for a client to receive a PUBLISH packet with DUP set to 0 that contains a repetition of an application message that it received earlier, but with a different MessageID.
    • Resolution:
      Hide

      Resolved in WD13

      Show
      Resolved in WD13

      Description

      WD11, lines 255-297 contain a non-normative comment that says

      "If 1, the recipient might have previously received the message, it shuld not treat this flag as an indication that it has previously received the MQTT Command message and should not use the DUP flag alone to guarantee to its application that a duplicate message is being redelivered to its application interface."

      There is a typo "shuld", and the use of two different terms "message and MQTT Command message" which I think mean the same thing. However we haven't really defined the role of recipient or "its application" and in addition it made me notice that we don't say anything normatively about how the server should handle an incoming Control Packet that contains the DUP flag.

      I think what the current non-normative words are trying to say is that when a recipient receives a Control Packet with the DUP flag set to 1, it should not assume that it has previously seen that Control Packet, just because the DUP flag is set. It might not have received the original copy at all, or it might have received it but had failed before it had managed to do much processing of the message.

      Do we need further normative words?

      1. The specification should not dictate how a receiving client library handles its relationship with its application, so no more words are needed here. A client library might choose to dedup its incoming PUBLISH messages, or might simply hand them off to the application with the DUP flag set.

      2. Similarly we should not say exactly how the server handles incoming PUBLISH messages which have the DUP flag set to 1. However if it does decide to deliver the message to one or more subscribers, how does it set the DUP flag in those messages? Does it set it to 0 for the first attempt, or does it set it to 1 on all attempts? I think that's something which should be normatively specified.

      3. Are there any special semantics relating to a SUBSCRIBE with DUP = 1?

        Attachments

          Activity

          Ascending order - Click to sort in descending order
          Hide
          Permalink
          raphcohn Raphael Cohen (Inactive) added a comment -

          This is quite a grey area - whatever we say should be tightly defined, but try to avoid asking clients or brokers to do too much.

          My current personal interpretation was that DUP=1 was effectively defining a re-delivery attempt. That's quite a different thing from annotating a duplicate message from client to receiver - the lifetimes are quite different. But it can also be argued the opposite way. For instance, it could be argued that that interpretation does not sit easily with other areas in the spec - such as the use of QoS being propagated.

          If DUP is re-delivery, then its handling should be at the discretion of a receiver. Only a receiver will know if it has seen the message before or not, or received part of it, etc. However, if we went with the second sentence of (2), then it gets quite interesting. There can then be DUPs of DUPs, effectively.

          The DUP flag could also be seen to be overlapping in meaning with QoS for some scenarios - particularly (3), where one could be seen to be making a statement as a receive about the desirability of at-least-once messaging.

          It might be worth approaching this as thinking about how it might map to JMS or OpenMAMA, say, simply to get a feel for how client code might handle it.

          My summary: I'd prefer that DUP is simply reflecting a delivery-attempt, and is scoped to the duration of the interaction of the connection's 2 parties - ie it is persistent if the connection is disconnected and reconnected (perhaps with a nod towards its interaction with QoS) but not from client -> broker / bridge -> client. That delivery attempt should be counted as for those two parties, ie delivery attempts to other parties don't count (eg when there are several subscribers to a broker).

          Open to other views...

          Show
          raphcohn Raphael Cohen (Inactive) added a comment - This is quite a grey area - whatever we say should be tightly defined, but try to avoid asking clients or brokers to do too much. My current personal interpretation was that DUP=1 was effectively defining a re-delivery attempt. That's quite a different thing from annotating a duplicate message from client to receiver - the lifetimes are quite different. But it can also be argued the opposite way. For instance, it could be argued that that interpretation does not sit easily with other areas in the spec - such as the use of QoS being propagated. If DUP is re-delivery, then its handling should be at the discretion of a receiver. Only a receiver will know if it has seen the message before or not, or received part of it, etc. However, if we went with the second sentence of (2), then it gets quite interesting. There can then be DUPs of DUPs, effectively. The DUP flag could also be seen to be overlapping in meaning with QoS for some scenarios - particularly (3), where one could be seen to be making a statement as a receive about the desirability of at-least-once messaging. It might be worth approaching this as thinking about how it might map to JMS or OpenMAMA, say, simply to get a feel for how client code might handle it. My summary: I'd prefer that DUP is simply reflecting a delivery-attempt, and is scoped to the duration of the interaction of the connection's 2 parties - ie it is persistent if the connection is disconnected and reconnected (perhaps with a nod towards its interaction with QoS) but not from client -> broker / bridge -> client. That delivery attempt should be counted as for those two parties, ie delivery attempts to other parties don't count (eg when there are several subscribers to a broker). Open to other views...
          Hide
          Permalink
          peterniblett Peter Niblett (Inactive) added a comment - - edited

          When the server is receiving a QoS 2 message, it seems clear that it should not propagate the DUP flag. The transmission of the message from publishing client to server is independent of its transmission from the server to the subscribing clients. It's of no interest to those clients whether the message got to the server on the first attempt, or whether the publishing client had to resend it. This is in keeping with the current words in the WD08 spec which say that the DUP flag to indicates a resend of the MQTT PUBLISH control packet, not a resend of the application message that it contains.

          Things are less clear-cut when the server receives a QoS 1 message. The server is at liberty to forward it on to its subscribed clients immediately before sending the PUBACK to the client. If it fails to complete this task, then the publishing client won't see the PUBACK and so at some stage it will resend the message with the DUP flag, and this will cause the server to attempt a resend to its subscribed clients. The worst case is one where the network fails while transmitting the PUBACK, or the publishing client fails before it has processed the PUBACK. In this case all the subscribed clients will get a second copy of the message. With the DUP flag defined as it is at present (indicating a resend of the PUBLISH Control Packet) all these subscribers would have DUP = 0 in their control packets, even though it's actually the second time they have seen this application message.

          Our options are

          1. Keep the definition of DUP as it is - an indicator that the Publish packet is the second (or subsequent) attempt at sending the message by your immediate counterparty. Update the non-normative note to add a statement that the DUP flag does not refer to the application message, and that it's possible to get a second copy of the application message with DUP -= 0.

          2. Change the semantics of DUP to say that it is an indicator that the application message contained in the PUBLISH packet might be a DUP.. This would require the server to set the DUP flag if there's any possibility that the application message could be a duplicate, in particular it should set the DUP bit when forwarding any message that it receives that has the DUP bit set,

          The consensus on the "scrub call" was to go for option 1, so I will add a proposal to that effect

          Show
          peterniblett Peter Niblett (Inactive) added a comment - - edited When the server is receiving a QoS 2 message, it seems clear that it should not propagate the DUP flag. The transmission of the message from publishing client to server is independent of its transmission from the server to the subscribing clients. It's of no interest to those clients whether the message got to the server on the first attempt, or whether the publishing client had to resend it. This is in keeping with the current words in the WD08 spec which say that the DUP flag to indicates a resend of the MQTT PUBLISH control packet, not a resend of the application message that it contains. Things are less clear-cut when the server receives a QoS 1 message. The server is at liberty to forward it on to its subscribed clients immediately before sending the PUBACK to the client. If it fails to complete this task, then the publishing client won't see the PUBACK and so at some stage it will resend the message with the DUP flag, and this will cause the server to attempt a resend to its subscribed clients. The worst case is one where the network fails while transmitting the PUBACK, or the publishing client fails before it has processed the PUBACK. In this case all the subscribed clients will get a second copy of the message. With the DUP flag defined as it is at present (indicating a resend of the PUBLISH Control Packet) all these subscribers would have DUP = 0 in their control packets, even though it's actually the second time they have seen this application message. Our options are 1. Keep the definition of DUP as it is - an indicator that the Publish packet is the second (or subsequent) attempt at sending the message by your immediate counterparty. Update the non-normative note to add a statement that the DUP flag does not refer to the application message, and that it's possible to get a second copy of the application message with DUP -= 0. 2. Change the semantics of DUP to say that it is an indicator that the application message contained in the PUBLISH packet might be a DUP.. This would require the server to set the DUP flag if there's any possibility that the application message could be a duplicate, in particular it should set the DUP bit when forwarding any message that it receives that has the DUP bit set, The consensus on the "scrub call" was to go for option 1, so I will add a proposal to that effect
          Hide
          Permalink
          coppen Richard Coppen (Inactive) added a comment -

          Take to scrub call 03.10.2013

          Show
          coppen Richard Coppen (Inactive) added a comment - Take to scrub call 03.10.2013
          Hide
          Permalink
          coppen Richard Coppen (Inactive) added a comment -

          TC Approve 03.10.2013

          Show
          coppen Richard Coppen (Inactive) added a comment - TC Approve 03.10.2013
          Hide
          Permalink
          coppen Richard Coppen (Inactive) added a comment -

          fixed in WD13

          Show
          coppen Richard Coppen (Inactive) added a comment - fixed in WD13

            People

            • Assignee:
              andrew_banks Andrew Banks (Inactive)
              Reporter:
              peterniblett Peter Niblett (Inactive)
            • Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: