Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.1.1
    • Fix Version/s: 3.1.1
    • Component/s: core
    • Labels:
      None
    • Proposal:
      Hide

      A PUBLICATION packet with a retain flag set to 1 and a payload containing zero bytes will be processed as normal by the server and sent to clients with a subscription matching the topic name. Additionally any existing retained publication for the topic name will be removed and any future subscribers for the topic will not receive a retained publication. As normal means that the retained bit is not set in the message received by existing subscribers.

      Show
      A PUBLICATION packet with a retain flag set to 1 and a payload containing zero bytes will be processed as normal by the server and sent to clients with a subscription matching the topic name. Additionally any existing retained publication for the topic name will be removed and any future subscribers for the topic will not receive a retained publication. As normal means that the retained bit is not set in the message received by existing subscribers.
    • Resolution:
      Hide

      Fixed in WD10 line 372

      Show
      Fixed in WD10 line 372

      Description

      There specification suggests that a retained message published with a zero byte payload can be used to cause a retained message to be deleted by the server, although it
      leaves this behavior as optional. It is not clear whether the zero length message should be sent to subscribers.

        Attachments

          Activity

          Hide
          dobermai Dominik Obermaier (Inactive) added a comment -

          I think this behaviour should not be optional but instead it should be mandatory. This is the only chance for a client to delete a retained message. When this behaviour would be removed from the specification, it is impossible to implement a scenario where a "last good message" could be deleted by clients.

          Afaik most brokers implement the behaviour of deleting a retained publish when sending a zero byte payload message with the retained flag set and there are probably several projects out there which rely on this feature.

          Show
          dobermai Dominik Obermaier (Inactive) added a comment - I think this behaviour should not be optional but instead it should be mandatory. This is the only chance for a client to delete a retained message. When this behaviour would be removed from the specification, it is impossible to implement a scenario where a "last good message" could be deleted by clients. Afaik most brokers implement the behaviour of deleting a retained publish when sending a zero byte payload message with the retained flag set and there are probably several projects out there which rely on this feature.
          Hide
          knolleary Nick O'Leary (Inactive) added a comment -

          Just to add some historical perspective on this.

          Until 3.1, the spec said nothing about deleting retained messages. Implementations, such as Message Broker and MicroBroker had to provide an administrative API to do this out-of-band. This is a problem as it meant applications that relied on deleting retained messages were not portable between broker implementations - they had to use broker specific apis.

          RSMB added this capability around the time 3.1 was being put together and it was decided to add the description of the behaviour to the spec so that new implementations would not have to come up with their own new scheme for handling it.

          The reason most (non-IBM) brokers implement this behaviour is that they have only know the 3.1 version of the spec where the behaviour is described.

          In terms of whether the zero-length payload should be sent to the subscribers; RSMB (and I believe mosquitto) do send the zero-length payload to the subscribers.

          In terms of this specific JIRA item, I'm not clear what it's asking. The proposal suggests removing this capability altogether (which I don't support), but the description asks if the payload should be sent to the subscribers (a clarification I do support).

          Show
          knolleary Nick O'Leary (Inactive) added a comment - Just to add some historical perspective on this. Until 3.1, the spec said nothing about deleting retained messages. Implementations, such as Message Broker and MicroBroker had to provide an administrative API to do this out-of-band. This is a problem as it meant applications that relied on deleting retained messages were not portable between broker implementations - they had to use broker specific apis. RSMB added this capability around the time 3.1 was being put together and it was decided to add the description of the behaviour to the spec so that new implementations would not have to come up with their own new scheme for handling it. The reason most (non-IBM) brokers implement this behaviour is that they have only know the 3.1 version of the spec where the behaviour is described. In terms of whether the zero-length payload should be sent to the subscribers; RSMB (and I believe mosquitto) do send the zero-length payload to the subscribers. In terms of this specific JIRA item, I'm not clear what it's asking. The proposal suggests removing this capability altogether (which I don't support), but the description asks if the payload should be sent to the subscribers (a clarification I do support).
          Hide
          raphcohn Raphael Cohen (Inactive) added a comment -

          I'd second that the zero length payload is sent on to clients, and that is clarified in the spec. This makes it trivial for a client, as well as a broker, to know that a retained subscription has been deleted. It also means that brokers being retrofitted with MQTT can degrade more gracefully if they can't support retained subscriptions (by simply passing through messages) or retained subscription deletion.

          Show
          raphcohn Raphael Cohen (Inactive) added a comment - I'd second that the zero length payload is sent on to clients, and that is clarified in the spec. This makes it trivial for a client, as well as a broker, to know that a retained subscription has been deleted. It also means that brokers being retrofitted with MQTT can degrade more gracefully if they can't support retained subscriptions (by simply passing through messages) or retained subscription deletion.
          Hide
          andrew_banks Andrew Banks (Inactive) added a comment -

          Here are the problems I see in allowing a zero byte retained message to mean deleting an existing retained message.

          1) An existing application that uses a zero byte payload message would have to be modified to use MQTT.
          2) We would implicitly give some administrator type privileges (to delete retained publications) to all clients that are permitted to publish on the topic.
          3) I am wary of defining more than is necessary about the server processing. In this case, where the server is in fact a connected set of servers it is
          difficult to guarantee that a retained publication has been deleted because there are many race conditions possible. Such systems can promise very little about
          the order in which actions are taken.

          Show
          andrew_banks Andrew Banks (Inactive) added a comment - Here are the problems I see in allowing a zero byte retained message to mean deleting an existing retained message. 1) An existing application that uses a zero byte payload message would have to be modified to use MQTT. 2) We would implicitly give some administrator type privileges (to delete retained publications) to all clients that are permitted to publish on the topic. 3) I am wary of defining more than is necessary about the server processing. In this case, where the server is in fact a connected set of servers it is difficult to guarantee that a retained publication has been deleted because there are many race conditions possible. Such systems can promise very little about the order in which actions are taken.
          Hide
          raphcohn Raphael Cohen (Inactive) added a comment -

          Trying to address (2):-

          This is not necessarily an administrator permission, but it is an important consideration. If a broker wants to implement permissions here, then it's highly likely that the same permission to retain a message (with all the potential for blowing up because there are too many) is the same that would be used for removing it... Perhaps the way forward here is to use 'SHOULD'. In an ideal world, with no existing spec and implementations, I'd actually argue for a special command flow so things were unambiguous, but that's not possible

          Are we also going to clarify / enforce that a deleting message MUST also have the RETAIN flag set? It would seem prudent. That would allow differentiation of empty (do not delete) and empty (delete). Are we also going to suggest that it is not an error to try to delete a subscription that does not exist? (very important for recovery in a 'did I or didn't I situation'), ie it's idempotent.

          If we have retained message creation via the protocol, I think it's essential to have deletion via the protocol. There's no way any modern broker admin is going to want to have occasionally query and remove retained messages, especially when its the clients of the broker that have the domain knowledge, not the broker.

          Discussing (1):-

          If we take the approach above - making use of the RETAIN flag be required for deletion to differentiate - then I think we can mitigate this problem.

          Observations on (3):-

          I agree we should say very little about broker behaviour, but this is one place where have to say something, as it substantially affects client expectations.

          Messages are sent and delivered in order. If one is replicating to a server cluster, or bridging to another broker, then the command flow should be serialised and replayed - this is how most forms of replication work, for instance. Any broker worth its salt will, at the very least, know about message receipt order. There is always the possibility that the order of a retain create and a retain delete is permuted because a client is using two connections to send one or the other separated by a short amount of time. Such a race condition is always the client's responsibility - it is no different to any other abuse of message queue total ordering. of course, if it is accidental, then the client can mitigate by using the same client identifier and rely on a broker that implements connection stealing (last connection on that id wins - many brokers do this to allow client failover when one can't be 100% the other client is genuinely dead).

          Show
          raphcohn Raphael Cohen (Inactive) added a comment - Trying to address (2):- This is not necessarily an administrator permission, but it is an important consideration. If a broker wants to implement permissions here, then it's highly likely that the same permission to retain a message (with all the potential for blowing up because there are too many) is the same that would be used for removing it... Perhaps the way forward here is to use 'SHOULD'. In an ideal world, with no existing spec and implementations, I'd actually argue for a special command flow so things were unambiguous, but that's not possible Are we also going to clarify / enforce that a deleting message MUST also have the RETAIN flag set? It would seem prudent. That would allow differentiation of empty (do not delete) and empty (delete). Are we also going to suggest that it is not an error to try to delete a subscription that does not exist? (very important for recovery in a 'did I or didn't I situation'), ie it's idempotent. If we have retained message creation via the protocol, I think it's essential to have deletion via the protocol. There's no way any modern broker admin is going to want to have occasionally query and remove retained messages, especially when its the clients of the broker that have the domain knowledge, not the broker. Discussing (1):- If we take the approach above - making use of the RETAIN flag be required for deletion to differentiate - then I think we can mitigate this problem. Observations on (3):- I agree we should say very little about broker behaviour, but this is one place where have to say something, as it substantially affects client expectations. Messages are sent and delivered in order. If one is replicating to a server cluster, or bridging to another broker, then the command flow should be serialised and replayed - this is how most forms of replication work, for instance. Any broker worth its salt will, at the very least, know about message receipt order. There is always the possibility that the order of a retain create and a retain delete is permuted because a client is using two connections to send one or the other separated by a short amount of time. Such a race condition is always the client's responsibility - it is no different to any other abuse of message queue total ordering. of course, if it is accidental, then the client can mitigate by using the same client identifier and rely on a broker that implements connection stealing (last connection on that id wins - many brokers do this to allow client failover when one can't be 100% the other client is genuinely dead).
          Hide
          dobermai Dominik Obermaier (Inactive) added a comment -

          My thoughts on your points:

          1) This would only be the case if it is a message is sent with a zero byte payload and the retained message flag set. So I do not see how a client could accidentally delete the retained message on the server because the client has to explicitly set the retained flag.

          2) I personally am not in complete agreement that deleting a retained message is solely an administrative task. I see many use cases for clients deleting a retained message. (let's say a client publishes a retained message to "/warning", so every new subscriber is notified that there is a warning. When there is no warning anymore the client could delete the warning message so newly connected clients do not see the warning message. Perhaps a bit constructed but this is perfectly possible with the current 3.1 specification.)

          However, I second the idea to allow administrators to delete retained messages. I think this should be vendor specific.

          3. I think it is a problem of the server vendor to handle possible race conditions in such cases. In fact there are some parts of the MQTT specification which are much harder to implement (like QoS 2 handling) in cases where many servers are connected (like in a MQTT server cluster).

          Show
          dobermai Dominik Obermaier (Inactive) added a comment - My thoughts on your points: 1) This would only be the case if it is a message is sent with a zero byte payload and the retained message flag set. So I do not see how a client could accidentally delete the retained message on the server because the client has to explicitly set the retained flag. 2) I personally am not in complete agreement that deleting a retained message is solely an administrative task. I see many use cases for clients deleting a retained message. (let's say a client publishes a retained message to "/warning", so every new subscriber is notified that there is a warning. When there is no warning anymore the client could delete the warning message so newly connected clients do not see the warning message. Perhaps a bit constructed but this is perfectly possible with the current 3.1 specification.) However, I second the idea to allow administrators to delete retained messages. I think this should be vendor specific. 3. I think it is a problem of the server vendor to handle possible race conditions in such cases. In fact there are some parts of the MQTT specification which are much harder to implement (like QoS 2 handling) in cases where many servers are connected (like in a MQTT server cluster).
          Hide
          coppen Richard Coppen (Inactive) added a comment -

          discussed at scrub call on 08.08.2013. Proposal updated for TC vote on next call

          Show
          coppen Richard Coppen (Inactive) added a comment - discussed at scrub call on 08.08.2013. Proposal updated for TC vote on next call
          Hide
          coppen Richard Coppen (Inactive) added a comment -

          TC approve: TC call 15.08.2013
          In WD12 line 376

          Show
          coppen Richard Coppen (Inactive) added a comment - TC approve: TC call 15.08.2013 In WD12 line 376

            People

            • Assignee:
              Unassigned
              Reporter:
              andrew_banks Andrew Banks (Inactive)
            • Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: