I propose to introduce an optional field id for custom defined data. Unlike other field ids that are followed by a value (of a specific data type), for custom defined data we can use 2 strings representing a 'key name' and a 'value'. The optional field id followed by the 2 strings can be repeated. Thus, multiple custom fields (with different key names) can be specified in a single packet.

I propose the following new identified data type for this purpose.

type name: user defined name-value pair = 0xNN (TBD)

ident : byte = 0xNN

.

The length field is a variable length integer which is the number of bytes contained in the two counted UTF-8 strings that follow it (including the string length fields). The value of the length field must be at least 4 (signifying 2 empty UTF-8 strings) and can be no more than 2 * 65535 + 4 = 131074, which means it can be no more than 3 bytes when encoded. The sender MUST not send a user defined name-value pair with a length of less than 4, or a length of greater than 131074.

I do not see any particular value in having an overall length in addition to the name and value lengths. This just makes it harder for someone writing this out. My preference would be to have a single string value with a = character as the separator between name and value, but I guess any of these would work.

One concern I have is the meaning of having multiple of the same name. It there an implied order? Does the Server need to not only preserve the id/value pair but the order relative to other id/value pairs? A common usage of this it to present a set of these properties as a Map and thus implement unique key names where setting the value replaces an existing one.

Jon suggested a use for these properties as bread crumbs, but even in that usage it seems better to use separate names which create an explicit order than to just have multiple properties of the same name.

Would it make sense that the value be binary data and not a UTF-8 string. The only real difference is whether we do UTF-8 and null checking on the bytes. Making the types of the name and value would be a reason to have two separate fields..

1, The value of having an overall length field is that a system that wants to skip (ignore) the value can simply read the length, and advance the stream to the next item of interest. I believe there have been several discussions on the desirability of being able to do that, so I added the length.

2. The proposal is that these are processed by the application, so the interpretation of the items, including how to handle multiple instances of the same name should not be specified in the standard. The server should pass them as they are, in the order they were received. There is no impled order on any of the other metadata tags in any other JIRA (though must prohibit multiple instances of the same tag). However, I think it would not serve a useful purpose for the server to re-order tags in a message it is forwarding.

3. The advantage of using UTF-8 is that the strings are 'counted strings' and we already have an existing type and code to process these. To prevent an explosion in the number of types, value encoding, etc. I used UTF-8 strings. They're pretty flexible. We could do a series of name=value with some separator. But if someone needs to use
the '=' or the separator character, we have to add an escape hack I'd like to avoid.

(In reply to previous comment):

1. Having an overall length means it takes 1 skips rather than two - I'm fairly ambivalent about that but have a mild preference for no overall length and lengths for each string (though a single string with a '=' separator and that separator being disallowed in names doesn't feel like a big restriction?)

2. I don't think there should be normative text about how duplicates are handled - but (especially in the case of publish) clients are going to represent these somehow and non-normative guidance about how to deal with duplicates in such a representative seems useful. In particular clients are likely to use a map/associative array/dictionary to represent these fields and we don't want security issues where validating applications check one version and applications are given a different one of the duplicates. If we could find a suitably non-constraining form of words, do people feel it would be harmless that if representing the key value pairs as a map, the last one in the message wins in the case of duplicates?

TC approved on 6/Oct/2016 Resolved, assigned to edit.

Summary:

type name: user defined name-value pair = 0xNN (TBD)

ident : byte = 0xNN

length: Variable Byte Integer
name : UTF-8 encoded string
value : UTF-8 encoded string

The user defined name-value pair is an optional value that used in CONNECT command packets to carry 'application layer' information which is NOT interpreted by MQTT. This is an optional data item, and may appear zero or more times in a CONNECT command packet.

A Server receiving this tag MUST check that two string fields contain valid UTF-8 strings. The format, meaning, and uniqueness of the name is not otherwise processed or enforced by MQTT. The user defined name-value pair may appear zero or more times in a CONNECT or PUBLISH command packet. Servers MUST preserve the order of these tags when processing them.

I thank the TC for the benefit of their comments and prompt consideration.

Added to WD07. Closing.