I think this behaviour should not be optional but instead it should be mandatory. This is the only chance for a client to delete a retained message. When this behaviour would be removed from the specification, it is impossible to implement a scenario where a "last good message" could be deleted by clients.

Afaik most brokers implement the behaviour of deleting a retained publish when sending a zero byte payload message with the retained flag set and there are probably several projects out there which rely on this feature.

Just to add some historical perspective on this.

Until 3.1, the spec said nothing about deleting retained messages. Implementations, such as Message Broker and MicroBroker had to provide an administrative API to do this out-of-band. This is a problem as it meant applications that relied on deleting retained messages were not portable between broker implementations - they had to use broker specific apis.

RSMB added this capability around the time 3.1 was being put together and it was decided to add the description of the behaviour to the spec so that new implementations would not have to come up with their own new scheme for handling it.

The reason most (non-IBM) brokers implement this behaviour is that they have only know the 3.1 version of the spec where the behaviour is described.

In terms of whether the zero-length payload should be sent to the subscribers; RSMB (and I believe mosquitto) do send the zero-length payload to the subscribers.

In terms of this specific JIRA item, I'm not clear what it's asking. The proposal suggests removing this capability altogether (which I don't support), but the description asks if the payload should be sent to the subscribers (a clarification I do support).

I'd second that the zero length payload is sent on to clients, and that is clarified in the spec. This makes it trivial for a client, as well as a broker, to know that a retained subscription has been deleted. It also means that brokers being retrofitted with MQTT can degrade more gracefully if they can't support retained subscriptions (by simply passing through messages) or retained subscription deletion.

Here are the problems I see in allowing a zero byte retained message to mean deleting an existing retained message.

1) An existing application that uses a zero byte payload message would have to be modified to use MQTT.
2) We would implicitly give some administrator type privileges (to delete retained publications) to all clients that are permitted to publish on the topic.
3) I am wary of defining more than is necessary about the server processing. In this case, where the server is in fact a connected set of servers it is
difficult to guarantee that a retained publication has been deleted because there are many race conditions possible. Such systems can promise very little about
the order in which actions are taken.

Trying to address (2):-

This is not necessarily an administrator permission, but it is an important consideration. If a broker wants to implement permissions here, then it's highly likely that the same permission to retain a message (with all the potential for blowing up because there are too many) is the same that would be used for removing it... Perhaps the way forward here is to use 'SHOULD'. In an ideal world, with no existing spec and implementations, I'd actually argue for a special command flow so things were unambiguous, but that's not possible

Are we also going to clarify / enforce that a deleting message MUST also have the RETAIN flag set? It would seem prudent. That would allow differentiation of empty (do not delete) and empty (delete). Are we also going to suggest that it is not an error to try to delete a subscription that does not exist? (very important for recovery in a 'did I or didn't I situation'), ie it's idempotent.

If we have retained message creation via the protocol, I think it's essential to have deletion via the protocol. There's no way any modern broker admin is going to want to have occasionally query and remove retained messages, especially when its the clients of the broker that have the domain knowledge, not the broker.

Discussing (1):-

If we take the approach above - making use of the RETAIN flag be required for deletion to differentiate - then I think we can mitigate this problem.

Observations on (3):-

I agree we should say very little about broker behaviour, but this is one place where have to say something, as it substantially affects client expectations.

Messages are sent and delivered in order. If one is replicating to a server cluster, or bridging to another broker, then the command flow should be serialised and replayed - this is how most forms of replication work, for instance. Any broker worth its salt will, at the very least, know about message receipt order. There is always the possibility that the order of a retain create and a retain delete is permuted because a client is using two connections to send one or the other separated by a short amount of time. Such a race condition is always the client's responsibility - it is no different to any other abuse of message queue total ordering. of course, if it is accidental, then the client can mitigate by using the same client identifier and rely on a broker that implements connection stealing (last connection on that id wins - many brokers do this to allow client failover when one can't be 100% the other client is genuinely dead).

My thoughts on your points:

1) This would only be the case if it is a message is sent with a zero byte payload and the retained message flag set. So I do not see how a client could accidentally delete the retained message on the server because the client has to explicitly set the retained flag.

2) I personally am not in complete agreement that deleting a retained message is solely an administrative task. I see many use cases for clients deleting a retained message. (let's say a client publishes a retained message to "/warning", so every new subscriber is notified that there is a warning. When there is no warning anymore the client could delete the warning message so newly connected clients do not see the warning message. Perhaps a bit constructed but this is perfectly possible with the current 3.1 specification.)

However, I second the idea to allow administrators to delete retained messages. I think this should be vendor specific.

3. I think it is a problem of the server vendor to handle possible race conditions in such cases. In fact there are some parts of the MQTT specification which are much harder to implement (like QoS 2 handling) in cases where many servers are connected (like in a MQTT server cluster).

discussed at scrub call on 08.08.2013. Proposal updated for TC vote on next call

TC approve: TC call 15.08.2013
In WD12 line 376