Uploaded image for project: 'OASIS Message Queuing Telemetry Transport (MQTT) TC'
  1. OASIS Message Queuing Telemetry Transport (MQTT) TC
  2. MQTT-256

Message Format indication and message metadata in general.

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 5
    • Fix Version/s: wd01, 5
    • Component/s: futures
    • Labels:
      None
    • Proposal:
      Hide

      The meta data is encoded into the Variable Header to carry Values which
      can be left to take a default value if they are not present. Initially it
      will be described for the PUBLISH packet but might be extended to other
      packets as well. Each metadata value has an Identifier and an associated Value.
      The Identifier also describes the data type of the Value.

      1) Create new data types in "1.5 Data representations"

      • Make the Remaining Length construct, a defined data type so that
        it can be used to encode the number of bytes containing the metadata,
        as well as for its original purpose.

      Move the description of Remaining Length from section "2 MQTT
      Control Packet format" to section "1.5 Data representations" as a datatype
      called Variable Byte Integer.

      • Add a Four Byte Integer data type type. This will be used initially
        to decribe expiry intervals. This is in addition to the existing
        Two Byte Integer value.

      2) Update the description of the PUBLISH Packet to include the
      Identifier/Value pairs which can be present in the Variable Header.

      • The Variable header contains the following fields in the
        following order:
        Topic Name, Packet Identifier, Length in bytes of Id/Value pairs,
        Identifier/Value pairs.
      • After the description of Packet Identifier, add:

      x.x.x.x Length of Id/Value pairs.
      The Length of the Identifier/Value pairs in the PUBLISH
      Packet variable header encoded as a Variable Byte Integer.
      The Length does not include the
      bytes used to encode the Length. A Length of
      zero indicates that there are no Identifier/Value pairs in the
      PUBLISH packet. The Id/Value pairs can appear in any order.

      x.x.x.x Identifier/Value pairs.
      An Id/Value pair consists of an Id which defines the usage and data type,
      followed by the value itself.
      The Id is encoded as a Variable Byte Integer.

      x.x.x.x.x Identifier: Payload Format Indicator.

      If absent the PUBLISH payload consists of unspecified bytes.
      Byte 1 (0x01) Identifier of the Payload Format Indicator. It is a protocol error
      to include the Payload Format Indicator more than once.

      Followed by the Value of the Payload Format Indicator, either of:
      Byte 0 (0x00) Indicates that the payload is unspecified bytes, this is
      equivalent to not sending a Payload Format Indicator.
      Byte 1 (0x01) Indicates that the payload is UTF-8 Encoded Character Data.
      Note that the UTF-8 Data in the payload does not include a
      length prefix, nor is it subject to the restrictions
      described in section 1.5.3.

      A server MUST send the Payload Format Indicator unaltered to all
      subscribers receiving the publication, using this version of MQTT.
      The receiver is not required to validate that the payload is of
      the format indicated, however it MAY close the network connection if it
      discovers that the payload does not match the Payload Format
      Indicator.

      The server MUST send the publication to an MQTT client using the
      MQTT V3.1.1 protocol with a matching subscription, in this case no
      Payload Format Indicator is included.

      On receipt of any other Identifier or Value, the receiver MUST close the
      network connection.

      • Update 3.3.2.3 Variable header non normative example to show zero
        Id/Value pairs.

      Non normative comment
      In this version of the specification only one byte Id's are used.

      Note:

      1) Jira MQTT-288 will consider making a unified and organised set of Metadata Id's
      for use across all packet types in a separate Jira once the complete set of
      values is known.

      2) Jira MQTT-282 will consider moving existing values into the optional
      metadata in .

      3) Further Message format types such as JSON and Base64 may also
      be added in future.

      4) Jira MQTT-236 will provide a better mechanism for indicating protocol
      violations in metadata.

      Show
      The meta data is encoded into the Variable Header to carry Values which can be left to take a default value if they are not present. Initially it will be described for the PUBLISH packet but might be extended to other packets as well. Each metadata value has an Identifier and an associated Value. The Identifier also describes the data type of the Value. 1) Create new data types in "1.5 Data representations" Make the Remaining Length construct, a defined data type so that it can be used to encode the number of bytes containing the metadata, as well as for its original purpose. Move the description of Remaining Length from section "2 MQTT Control Packet format" to section "1.5 Data representations" as a datatype called Variable Byte Integer. Add a Four Byte Integer data type type. This will be used initially to decribe expiry intervals. This is in addition to the existing Two Byte Integer value. 2) Update the description of the PUBLISH Packet to include the Identifier/Value pairs which can be present in the Variable Header. The Variable header contains the following fields in the following order: Topic Name, Packet Identifier, Length in bytes of Id/Value pairs, Identifier/Value pairs. After the description of Packet Identifier, add: x.x.x.x Length of Id/Value pairs. The Length of the Identifier/Value pairs in the PUBLISH Packet variable header encoded as a Variable Byte Integer. The Length does not include the bytes used to encode the Length. A Length of zero indicates that there are no Identifier/Value pairs in the PUBLISH packet. The Id/Value pairs can appear in any order. x.x.x.x Identifier/Value pairs. An Id/Value pair consists of an Id which defines the usage and data type, followed by the value itself. The Id is encoded as a Variable Byte Integer. x.x.x.x.x Identifier: Payload Format Indicator. If absent the PUBLISH payload consists of unspecified bytes. Byte 1 (0x01) Identifier of the Payload Format Indicator. It is a protocol error to include the Payload Format Indicator more than once. Followed by the Value of the Payload Format Indicator, either of: Byte 0 (0x00) Indicates that the payload is unspecified bytes, this is equivalent to not sending a Payload Format Indicator. Byte 1 (0x01) Indicates that the payload is UTF-8 Encoded Character Data. Note that the UTF-8 Data in the payload does not include a length prefix, nor is it subject to the restrictions described in section 1.5.3. A server MUST send the Payload Format Indicator unaltered to all subscribers receiving the publication, using this version of MQTT. The receiver is not required to validate that the payload is of the format indicated, however it MAY close the network connection if it discovers that the payload does not match the Payload Format Indicator. The server MUST send the publication to an MQTT client using the MQTT V3.1.1 protocol with a matching subscription, in this case no Payload Format Indicator is included. On receipt of any other Identifier or Value, the receiver MUST close the network connection. Update 3.3.2.3 Variable header non normative example to show zero Id/Value pairs. Non normative comment In this version of the specification only one byte Id's are used. Note: 1) Jira MQTT-288 will consider making a unified and organised set of Metadata Id's for use across all packet types in a separate Jira once the complete set of values is known. 2) Jira MQTT-282 will consider moving existing values into the optional metadata in . 3) Further Message format types such as JSON and Base64 may also be added in future. 4) Jira MQTT-236 will provide a better mechanism for indicating protocol violations in metadata.
    • Resolution:
      Hide

      Included in WD01

      Show
      Included in WD01

      Description

      MQTT 3.1.1 does not provide an architected way of indicating the format of the payload that is passed in a PUBLISH packet. We need to avoid the risk of spec or data bloat, and I'm not advocating adding a complicated header structure that passes schemas, messageType indicators, MIME type strings or the like, but at the moment there is no standard way for a receiver even to be able to tell whether the payload is binary or string.
      There are three ways that this can be dealt with today:

      1. The Receiver already knows. It's implementing some bigger standard or program design which has dictated the encoding format (e.g. someone decides that all messages will be JSON)

      2. There is an indicator embedded somewhere in the Topic name (The Topic name is the only header we have in MQTT 3.1.1). There isn't a standard convention for this, so topic space designers have to choose their own way of doing it.

      3. The Receiver assumes one format and has a go parsing it. If that doesn't look right it tries another

        Attachments

          Activity

            People

            • Assignee:
              andrew_banks Andrew Banks (Inactive)
              Reporter:
              PeterNiblett Peter Niblett
            • Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: