The specification is sparse in its description of how a server should behave when it finds it has a message that cannot be delivered because it is too large.
One piece of guidance is in section 3.2.2.8 Maximum Packet Size
Non-Normative comment
The Server may place the discarded packet on a ‘dead letter queue’ or preform other diagnostic action. Such actions are outside the scope of this specification.
The situation is clearer for the client because it is able to not send the message in the first place. Other errors, these are either retryable or are detected after the sending of the publication has started.