Understanding the MQTT Protocol Packet Structure

mqtt-protocolIn this tutorial we will take a more detailed look at the MQTT protocol, and how MQTT messages or packets are formatted.

We will be looking at:

  • The MQTT message format.
  • The MQTT message header
  • Message fields and coding
  • Control Message coding example





Introduction

MQTT is a binary based protocol were the control elements are binary bytes and not text strings.

MQTT uses a command and command acknowledgement format.

That means each command has an associated acknowledgement.

MQTT-Protocol-Commands

Topic names, Client ID, User names and Passwords are encoded as  UTF-8 strings.

The Payload excluding MQTT protocol information like Client ID etc is binary data and the content and format is application specific.

The MQTT packet or message format consists of a 2 byte fixed header (always present) + Variable-header (not always present)+ payload (not always present).

MQTT-Standard-Packet

The fixed header field consists of the control field and the variable length  packet length field.

The minimum size of the packet length field is 1 byte which is for messages with a total length less than 127 bytes.(not including control and length fields).

The maximum packet size is 256MB. Small packets less than 127 bytes have a 1 byte packet length field.

Packets larger than 127 and less than 16383 will use 2 bytes. etc.

Note: 7 bits are used with the 8th bit being a continuation bit.

The minimum packet size is just 2 bytes with a single byte control field and a single byte packet length field. E.g the disconnect message is only 2 bytes.

mqtt-minimum-packet

The Control Field

The 8 bit control field is the first byte of the 2 byte fixed header. It is divided into two 4 bit fields,and contains all of the protocol commands and responses.

The first 4 Most significant bits are the command or message type field and the other 4 bits are used as control flags.

mqtt-control-field-structure

The table below is taken from the MQTT 3.1.1 specification and shows a sample of MQTT commands, and their associated codes.

Because they are the most significant part of an 8 bit byte field I have also shown their byte values in decimal as they would appear in the data packet.

Sample-MQTT-Control-Message

Control Flags

Although there are 16 possible flags very few are actually used.

The publish message makes the most use of these flags as shown in the table below:

MQTT-Message-Flags

Remaining Length

This is of variable length between 1 and 4 bytes. Each byte uses 7 bits for the length with the MSB used as a continuation flag.

The remaining length is the number of bytes following the length field, includes variable length header and payload as illustrated below:

MQTT-Packet-Remaining-Length

The following illustrates the length field for a packet size of 64 and 321 bytes

Remaining Packet length 64 bytes of requires only 1 byte:

mqtt-length-field-1

Packet length of 321 bytes requires a 2 byte remaining length field:

Encoding-Remaining-Length-Field-2

The following table taken from the specification shows packet sizes and packet length field.

Remaining-Length-Field-Values

MQTT Connect and Disconnect Message Example

As an illustration we will now look at the packet details for a connect message.

The table below show the control message types and their hex codes (taken from client program listing)

mqtt-message-type-codes

Below is a real client connection and disconnect example showing the actual byte values for the sent and received data.

MQTT-Message-Example-1

Notes:

  • Notice the connection (0x10) and connection acknowledge (0x20) control codes.
  • Notice the total length of hex17 or 23 bytes not including the control and length fields. The length field is only 1 byte.
  • You should also be able the see the client ID (python_test) in the sent packet.

Variable Length Header

As mentioned previously the variable length header field is not always present in an MQTT message.

Certain MQTT message types or commands require the use of this field to carry additional control information.

The variable length header field is similar, but not the same for all message types.

As an example we will look at the variable length header in a connection packet.

MQTT packet =control + length + protocol name + Protocol Level +Connect Flags + keep alive +Payload

It is interesting that the client ID field is sent as the first part of the payload, and not as part of the header.

It has has a 2 byte length field as a prefix.
Payload=client ID = client ID length + client ID

MQTT-Connect-Message-Structure
Click to enlarge

Notes:

  • When looking at the actual packet bytes Python prints the hex values unless it can match a ASCII character.In the example above the keep alive field is x00x3C but is displayed as x00<. Where Ascii < =0x3C
  • The Client ID is proceeded by a length field
  • The Connect flags indicate that a clean session is being requested.

Wireshark Network Analysis

In response to a reader question regarding TCP protocol I created this screen shot taken from wireshark.

wireshark-mqtt-packet

It shows an MQTT client connecting and publishing (QOS 1). You can clearly see the ACK packets which have a total packet length of 58 bytes.

We know that ACK packets are 2 bytes.

Therefore the TCP packet without MQTT is around 56 bytes.

What is also interesting to note, and something I hadn’t thought of until I did the packet capture, is that each MQTT command or response will get a TCP ACK and maybe also an MQTT ACK.

If you look at the screenshot you can the MQTT connection get a TCP ACK response and an MQTT Connect ACK response.

The MQTT Connect ACK response gets a TCP ACK response.

References:

MQTT V3.1.1 Specification pdf

Resources and related Tutorials

Facebooktwittergoogle_plusredditpinterestlinkedinmail

19 comments

  1. Hi Steve,

    Am using MQTT to develop a chat system, I need to send some additional information during publish and I dont want to include it in Payload( for some reason). Is it right way to go with Variable header field for the same??

  2. I enjoyed reading through, it is very crisp and detailed article.

    Could you also please add a section how the data gets transitioned through MQTT over Google Protocol Buffer and significance of various QoS ?

  3. Thank you for your explanation, .. I can now connect and publish message to the brooker. But i still fail to publish with retain message. My first byte is 0x31 which is 0x30 for publish and 0001 for retain flag. The message succesfully sent to brooker but not retained.

  4. Hello,
    I am using “mosquitto” implementation for MQTT installed on OpenWrt and Ubuntu.
    I am sending the content of a file using mosquitto_pub and receiving it via mosquitto_sub. I did packet analysis of this transmission and found that,
    1. If the file size 1434 Bytes then the content of the file gets split into TCP packet and MQTT packet.

    Though the received content is intact, I am curious to know that why is this happening.

    1. That is the way TCP/IP works. An MQTT packet needs to be inserted into a TCP/IP packet.
      THe maximum size of the IP Packet is limited by the transmission media. See wiki MTU https://en.wikipedia.org/wiki/Maximum_transmission_unit
      This means that a large MQTT packet will need to be split over several TCP/IP packets.
      AT the other end of the link the packet is reassembled by the MQTT client and appears as a single MQTT packet. Take a look at this tutorial it may help
      http://www.steves-internet-guide.com/basic-networking-course/
      section on frames

  5. Thanks, reading this explanation makes me understand, why some people think MQTT protocol is more like practise assignment for “Network programming 101” course than universally adopted protocol for IoT. Since it is exactly like that. Sigh.

  6. Thanks for the reply and the code links, but I guess what I’m asking is more a philosophical question rather than practical (as I’m writing my own code on an arduino for memory reasons and the “difficult bit” is all done).

    It just occurred to me that since all I’m doing is posting a temperature and as I got my code up and running I discovered that works using the last will and testament method on two broker services I have tried; is is there any reason I shouldn’t do it that way as by the very virtue of having a last will and testament the protocol is designed for the broker to detect the dropped TCP connection.

    Just a thought….!

    For good practice I am implementing the Publish and disconnect as well.

    1. I think the idea is good and a pity that the spec didn’t have it i.e
      connect with small message and disconnect.
      However as it is implemented I would prefer the connect+ pub+ disconnect.
      There are also practical implications of getting a dropped connections.
      rgds
      steve

  7. Just started using MQTT for a thermometer project. At first sight It’s not clear to me why I should use the Publish packets. Isn’t it enough to simply put the reading in the last will and testament?

    so: CONNECT (receive CONNACK) and then disconnect the TCP

    The alternative would seem to be:
    CONNECT (receive CONNACK)
    PUBLISH (receive PUBACK)
    DISCONNECT and then disconnect the TCP
    which seems wasteful for a lightweight protocol ?

    1. The last will and testament is only sent if the client din’t disconnect properly i.e. a network failure. In your situation you might want to look at the publish helper module which just lets you publidh and it takes care of the connection.
      Take a look here
      https://pypi.python.org/pypi/paho-mqtt
      It’s at the bottom of the page.
      The publish module is part of the mqtt client so you don’t need to install anything else.

  8. Indeed. Many thanks for a great illustration and explanation.
    I was looking to calculate the Overall Packet Size from the Network side. My understanding is that the MQTT rides over the IP Packet, so How much shall I add to calculate the Minimum Size of Nerwork packaet carrying MQTT Port Traffic?

Leave a Reply

Your email address will not be published. Required fields are marked *