Major The current state of the MQTT 5 specification states in Line 1007:
Clients are allowed to send further MQTT Control Packets immediately after sending a CONNECT packet; Clients need not wait (sic!) for a CONNACK packet to arrive from the Server. If the Server rejects the CONNECT, it MUST NOT process any data sent by the Client after the CONNECT packet except AUTH packets.
This behavior may be used for Denial of Service attacks. Consider the following scenario:
1. A malicious client connects to the Broker and sends an invalid CONNECT packet
2. The server has a complex authentication mechanism and it may take up to several seconds (or even longer!) until the validity of the CONNECT can be determined
3. The malicious client bombards the server with PUBLISH packets
4. The broker must queue all these messages until the CONNECT returns. The broker implementation can't stop to read from the socket (and thus apply TCP backpressure to the client) since an AUTH packet could be sent by the client.
MQTT 3.1.1 also allowed unauthenticated clients to send data, the broker implementation could stop to read from the socket until the CONNACK packet is produced.
To solve the issue, a proposal could be:
The Server MAY discard any packets sent by the Client prior to a successful authentication.
A non-normative comment could be added, that a broker may decide to use a bounded queue for "holding back" the data. If the queue is full, the PUBLISH messages (or other packets) could be dropped and an error code "0x97 - Quota exceeded" could be included in the ACK (e.g. PUBACK, ...).
A slightly better approach to fix this problem would be to disallow the sending of any packets before all AUTH packets are exchanged.