Understanding the MQTT Protocol Packet Structure

In this tutorial we will take a more detailed look at the MQTT protocol, and how MQTT messages or packets are formatted.

We will be looking at:

• The MQTT message format.
• The MQTT message header
• Message fields and coding
• Control Message coding example

Introduction

MQTT is a binary based protocol were the control elements are binary bytes and not text strings.

MQTT uses a command and command acknowledgement format.

That means each command has an associated acknowledgement.

Topic names, Client ID, User names and Passwords are encoded as  UTF-8 strings.

The Payload excluding MQTT protocol information like Client ID etc is binary data and the content and format is application specific.

The MQTT packet or message format consists of a 2 byte fixed header (always present) + Variable-header (not always present)+ payload (not always present).

 

Possible Packet formats are:

• Fixed Header (Control field + Length) – Example CONNACK
• Fixed Header (Control field + Length) + Variable Header -Example PUBACK
• Fixed Header (Control field + Length)  + Variable Header + payload -Example CONNECT

The fixed header field consists of the control field and the variable length packet length field.

The minimum size of the packet length field is 1 byte which is for messages with a total length less than 127 bytes.(not including control and length fields).

The maximum packet size is 256MB. Small packets less than 127 bytes have a 1 byte packet length field.

Packets larger than 127 and less than 16383 will use 2 bytes. etc.

Note: 7 bits are used with the 8th bit being a continuation bit.

The minimum packet size is just 2 bytes with a single byte control field and a single byte packet length field. E.g the disconnect message is only 2 bytes.

The Control Field

The 8 bit control field is the first byte of the 2 byte fixed header. It is divided into two 4 bit fields,and contains all of the protocol commands and responses.

The first 4 Most significant bits are the command or message type field and the other 4 bits are used as control flags.

The table below is taken from the MQTT 3.1.1 specification and shows a sample of MQTT commands, and their associated codes.

Because they are the most significant part of an 8 bit byte field I have also shown their byte values in decimal as they would appear in the data packet.

Control Flags

Although there are 16 possible flags very few are actually used.

The publish message makes the most use of these flags as shown in the table below:

The duplicate flag is used when re-publishing a message with QOS or 1 or 2

QOS Flags are used when publishing and indicate the QOS level -0,1,2

Retain Message flag is also used on publishing.

Remaining Length

This is of variable length between 1 and 4 bytes. Each byte uses 7 bits for the length with the MSB used as a continuation flag.

The remaining length is the number of bytes following the length field, includes variable length header and payload as illustrated below:

The following illustrates the length field for a packet size of 64 and 321 bytes

Remaining Packet length 64 bytes of requires only 1 byte:

Packet length of 321 bytes requires a 2 byte remaining length field:

Note error in above should be 0xC1 0x01 .

The following table taken from the specification shows packet sizes and packet length field.

Variable Length Header

As mentioned previously the variable length header field is not always present in an MQTT message.

Certain MQTT message types or commands require the use of this field to carry additional control information.

The variable length header field is similar, but not the same for all message types.

MQTT Connect and Disconnect Message Example

As an illustration we will now look at the packet details for a connect message.

Below is a real client connection and disconnect example showing the actual byte values for the sent and received data.

The CONNECT control code =0x10

The CONNACK control code =0x20

MQTT packet =control + length + protocol name + Protocol Level +Connect Flags + keep alive +Payload

Notes:

• Notice the connection (0x10) and connection acknowledge (0x20) control codes.
• Notice the total length of hex17 or 23 bytes not including the control and length fields. The length field is only 1 byte.
• You should also be able the see the client ID (python_test) in the sent packet.
• When looking at the actual packet bytes Python prints the hex values unless it can match a ASCII character.In the example above the keep alive field is x00x3C but is displayed as x00<. Where Ascii < =0x3C

 

Notes:

• The client ID field is sent as the first part of the payload, and not as part of the header.
• The Client ID is proceeded by a length field
• The Connect flags indicate that a clean session is being requested.
• Connection flags are part of the Variable Length header and used to indicate the presence or absence of username,password, and will message fields in the payload. It also contains the clean session flag and Will QOS.

Control Packet Table Summary

Control Packet	Variable Header	Payload
CONNECT	Required	Required
CONNACK	None	None
PUBLISH	Required	Optional
PUBACK	Required	None
PUBREC	Required	None
PUBREL	Required	None
PUBCOMP	Required	None
SUBSCRIBE	Required	Required
SUBACK	Required	Required
UNSUBSCRIBE	Required	Required
UNSUBACK	Required	Required
PINGREQ	None	None
PINGRESP	None	None
DISCONNECT	None	None

 

Wireshark Network Analysis

In response to a reader question regarding TCP protocol I created this screen shot taken from wireshark.

It shows an MQTT client connecting and publishing (QOS 1). You can clearly see the ACK packets which have a total packet length of 58 bytes.

We know that ACK packets are 2 bytes.

Therefore the TCP packet without MQTT is around 56 bytes.

What is also interesting to note, and something I hadn’t thought of until I did the packet capture, is that each MQTT command or response will get a TCP ACK and maybe also an MQTT ACK.

If you look at the screenshot you can the MQTT connection get a TCP ACK response and an MQTT Connect ACK response.

The MQTT Connect ACK response gets a TCP ACK response.

References:

MQTT V3.1.1 Specification pdf

Modified Client

To produce the screen shots showing the bytes I modified the original client and placed it in the same folder as the normal client.

To use it I changed the import from

import paho.mqtt.client as mqtt

to

import paho.mqtt.mclient as mqtt

Here is the download

Video -MQTT Protocol Packet Structure

Resources and related Tutorials

• MQTT for Beginners
• Using The Paho Python MQTT client