Understanding the MQTT Protocol Packet Structure

mqtt-protocolIn this tutorial we will take a more detailed look at the MQTT protocol, and how MQTT messages or packets are formatted.

We will be looking at:

  • The MQTT message format.
  • The MQTT message header
  • Message fields and coding
  • Control Message coding example

Introduction

MQTT is a binary based protocol were the control elements are binary bytes and not text strings.

MQTT uses a command and command acknowledgement format.

That means each command has an associated acknowledgement.

MQTT-Protocol-Commands

Topic names, Client ID, User names and Passwords are encoded as  UTF-8 strings.

The Payload excluding MQTT protocol information like Client ID etc is binary data and the content and format is application specific.

The MQTT packet or message format consists of a 2 byte fixed header (always present) + Variable-header (not always present)+ payload (not always present).MQTT-Standard-Packet

 

Possible Packet formats are:

  • Fixed Header (Control field + Length) – Example CONNACK
  • Fixed Header (Control field + Length) + Variable Header -Example PUBACK
  • Fixed Header (Control field + Length)  + Variable Header + payload -Example CONNECT

The fixed header field consists of the control field and the variable length packet length field.

The minimum size of the packet length field is 1 byte which is for messages with a total length less than 127 bytes.(not including control and length fields).

The maximum packet size is 256MB. Small packets less than 127 bytes have a 1 byte packet length field.

Packets larger than 127 and less than 16383 will use 2 bytes. etc.

Note: 7 bits are used with the 8th bit being a continuation bit.

The minimum packet size is just 2 bytes with a single byte control field and a single byte packet length field. E.g the disconnect message is only 2 bytes.

mqtt-minimum-packet

The Control Field

The 8 bit control field is the first byte of the 2 byte fixed header. It is divided into two 4 bit fields,and contains all of the protocol commands and responses.

The first 4 Most significant bits are the command or message type field and the other 4 bits are used as control flags.

mqtt-control-field-structure

The table below is taken from the MQTT 3.1.1 specification and shows a sample of MQTT commands, and their associated codes.

Because they are the most significant part of an 8 bit byte field I have also shown their byte values in decimal as they would appear in the data packet.

Sample-MQTT-Control-Message

Control Flags

Although there are 16 possible flags very few are actually used.

The publish message makes the most use of these flags as shown in the table below:

MQTT-Message-Flags

The duplicate flag is used when re-publishing a message with QOS or 1 or 2

QOS Flags are used when publishing and indicate the QOS level -0,1,2

Retain Message flag is also used on publishing.

Remaining Length

This is of variable length between 1 and 4 bytes. Each byte uses 7 bits for the length with the MSB used as a continuation flag.

The remaining length is the number of bytes following the length field, includes variable length header and payload as illustrated below:

MQTT-Packet-Remaining-Length

The following illustrates the length field for a packet size of 64 and 321 bytes

Remaining Packet length 64 bytes of requires only 1 byte:

mqtt-length-field-1

Packet length of 321 bytes requires a 2 byte remaining length field:

Encoding-Remaining-Length-Field-2

Note error in above should be 0xC1 0x01 .

The following table taken from the specification shows packet sizes and packet length field.

Remaining-Length-Field-Values

Variable Length Header

As mentioned previously the variable length header field is not always present in an MQTT message.

Certain MQTT message types or commands require the use of this field to carry additional control information.

The variable length header field is similar, but not the same for all message types.

MQTT Connect and Disconnect Message Example

As an illustration we will now look at the packet details for a connect message.

Below is a real client connection and disconnect example showing the actual byte values for the sent and received data.

The CONNECT control code =0x10

The CONNACK control code =0x20

MQTT packet =control + length + protocol name + Protocol Level +Connect Flags + keep alive +Payload

MQTT-Message-Example-1

Notes:

  • Notice the connection (0x10) and connection acknowledge (0x20) control codes.
  • Notice the total length of hex17 or 23 bytes not including the control and length fields. The length field is only 1 byte.
  • You should also be able the see the client ID (python_test) in the sent packet.
  • When looking at the actual packet bytes Python prints the hex values unless it can match a ASCII character.In the example above the keep alive field is x00x3C but is displayed as x00<. Where Ascii < =0x3C

 

mqtt-packet-structure-connect

Notes:

  • The client ID field is sent as the first part of the payload, and not as part of the header.
  • The Client ID is proceeded by a length field
  • The Connect flags indicate that a clean session is being requested.
  • Connection flags are part of the Variable Length header and used to indicate the presence or absence of username,password, and will message fields in the payload. It also contains the clean session flag and Will QOS.

Control Packet Table Summary

Control Packet Variable Header Payload
CONNECT Required Required
CONNACK None None
PUBLISH Required Optional
PUBACK Required None
PUBREC Required None
PUBREL Required None
PUBCOMP Required None
SUBSCRIBE Required Required
SUBACK Required Required
UNSUBSCRIBE Required Required
UNSUBACK Required Required
PINGREQ None None
PINGRESP None None
DISCONNECT None None

 

Wireshark Network Analysis

In response to a reader question regarding TCP protocol I created this screen shot taken from wireshark.

wireshark-mqtt-packet

It shows an MQTT client connecting and publishing (QOS 1). You can clearly see the ACK packets which have a total packet length of 58 bytes.

We know that ACK packets are 2 bytes.

Therefore the TCP packet without MQTT is around 56 bytes.

What is also interesting to note, and something I hadn’t thought of until I did the packet capture, is that each MQTT command or response will get a TCP ACK and maybe also an MQTT ACK.

If you look at the screenshot you can the MQTT connection get a TCP ACK response and an MQTT Connect ACK response.

The MQTT Connect ACK response gets a TCP ACK response.

References:

MQTT V3.1.1 Specification pdf

Modified Client

To produce the screen shots showing the bytes I modified the original client and placed it in the same folder as the normal client.

modified python client

To use it I changed the import from

import paho.mqtt.client as mqtt

to

import paho.mqtt.mclient as mqtt

Here is the download

Video -MQTT Protocol Packet Structure

Resources and related Tutorials

Please rate? And use Comments to let me know more

45 comments

  1. I have a few questions about the MQTT Connect Message Structure example. In the bytes 3 through 9 is the protocol and version. The last byte is a 4 (not ascii ‘4’) does that refer to MQTT 3.1.1 ? If so, is there a reference for possible values there? Also, byte 4 has a value of 4 bytes. Wouldn’t this be 5 to include the version?

  2. Example with 321, should that not be 0xC1 0x01 (LSB, MSB) instead of 0xC1 0x02?
    0xC1 comes from 0x80 | 0x41 (continuation bit + 65 hex), LSB.
    0x01 comes from 0x00 | 0x01 (no continuation bit + 256), MSB.
    256 + 65 = 321

    1. I believe you’re incorrect. In Wireshark, packets with message length of 321 are represented with 0xC102. From what I understood, 321 uses 2 bytes, but has to be converted to binary using only 14 bits (since 2 are continuation bits).

  3. Hey steve,
    I’m trying to establish MQTT connection over two arduinos (mkr series).
    The connection is successful but i’m not able to store the published message into a variable that i can use to turn on a buzzer.
    while (mqttClient.available()) {
    Serial.println((char)mqttClient.read());
    qualButton = mqttClient.read();
    if(){
    Serial.println(qualButton);
    tone(buzzerPin, 1000);
    }
    else {
    Serial.println(qualButton);
    noTone(buzzerPin);
    }
    }

  4. On the second figure, it says “E.g.. a connect message doesn’t have a payload”, then in the next section, it says an example that has a “fixed header + variable length header + payload” is a connect message. I think figure 2 is meant to say e.g. a CONNACK is an example that doesn’t have a payload? unless I’m missing something

    1. Hi
      It was meant to say connect ack doesn’t have a payload. Tks fo pointing it out I will amend the figure
      rgds
      steve

  5. Hi Steve. I’ve been following your tutorials, and have been trying to use Wireshark to determine the size of the data sent (including headers and such), but the packets don’t show to be from the MQTT protocol nor from TCP port 1883. I am using the HiveMQ broker if that helps

      1. Yes, I am using one laptop to do everything: As subscriber, publisher, running the broker and capturing with wireshark

  6. The controller that I am using to publish messages to the broker has a limitation to the number of bytes I can send at one time. If my data exceeds that limitation(but is within the mqtt broker limitations) how do I set my message up to let the broker know the data does not end at this particular message ie packet 1 of # ?

      1. I can send it in smaller blocks but it removes some of the meaning of the data. The data I am sending is better if it stays together. So there isn’t a way to send the data in smaller chunks but to put it back together on the broker side?

          1. My limitations are around 4k bytes in one message but my data can go up to 10k (maybe more). The end-user wants to see the data altogether and not in chunks. This is the data that I am sending to the broker.

  7. Hi, I am Ehsan who is a beginner in MQTT.
    I am trying to set reserved message types for doing specific tasks in MQTT but I cannot find the message types in source code of MQTT.

    I would appreciate it if you guide me.

    Thanks.

    1. They are in the client.py file under lib/site-packages/paho/ (windows)
      Here is what they look like
      # Message types
      CONNECT = 0x10
      CONNACK = 0x20
      PUBLISH = 0x30
      PUBACK = 0x40
      PUBREC = 0x50
      PUBREL = 0x60
      PUBCOMP = 0x70
      SUBSCRIBE = 0x80
      SUBACK = 0x90
      UNSUBSCRIBE = 0xA0
      UNSUBACK = 0xB0
      PINGREQ = 0xC0
      PINGRESP = 0xD0
      DISCONNECT = 0xE0

  8. Hi Steve,
    Just to clarify for the remaining length, each byte uses 7 bits of length and MSB as the continuation bit.

    I noticed in Encoding Remain Length Field Example 1 has 9 bits (1 continuation bit, 8 length bits) instead of 8 bits (1 continuation bit, 7 length bits).
    In Encoding Remain Length Field Example 2, the MSB has 9 bits too.

    Can you please clarify and clear my doubts.
    Thank you.

  9. Hello, can you send the codes of example 1. How do I capture the message that the message reaches the recipient? Is there sample code?

    1. Hi
      The modified client script is on the web page. I assume you are looking for a script that displays the header information in detail.
      I did think of creating one but not really sure of the demand.
      Can you tell Why do you need such a script.
      Rgds
      steve

  10. Hi Steve,

    We are using IBM MQ 8 and the MQXR service and have seen instances where messages reside in the MQTT transmit queue and are flagged as in-doubt out for a client. From reading what the docs say, it sounds like the server has sent the message to the client but the client has not acknowledged the receipt. Am I correct here? The client then seems to get hung up trying to read messages and requires us to ‘clear’ the stuck messages. The client seems to get a nullPointerException. Is there a good tutorial I can point them to so they can connect and clear or acknowledge the previously delivered messages?

    1. Sorry but I’m not familiar with the client or server. Is the the transmit queue on the client?I would check that the messages do get delivered.Does it happen on all QOS settings.
      I would try a google search on
      BM MQ 8 nullPointerException
      if you haven’t done so already
      rgds
      steve

      1. I was able to capture a tcpdump. Looks like what is happening is this:

        MQTT client connects (already has a durable subscription in place)
        MQTT client sends ‘Connect Command’
        MQTT Broker sends a message that is waiting in the MQTT transmit queue and marks message as ‘in doubt out’
        MQTT client sends the subscription request
        MQTT Broker ACKs it
        MQTT Broker sends a message to MQTT client
        MQTT client ACKs it

        So it looks like the MQTT client is just dropping messages until it has sent the subscribe packet. If the Broker has messages waiting, is it correct to send them once the client connects or should it be waiting until the subscribe comes in? Granted this is a durable subscription so messages are already waiting on it. My guess is the client must be ready to accept messages once the ‘Connect Command’ has been sent.

  11. Hi Steve,

    Can you share the code for the output shown in “MQTT Message Example – 1” fig? I am trying to write python code to work on the packet data but I do not know how to do it. Any examples or guidance you can provide on that would be helpful.

    thank you.

  12. Hi Steve,

    Am using MQTT to develop a chat system, I need to send some additional information during publish and I dont want to include it in Payload( for some reason). Is it right way to go with Variable header field for the same??

  13. I enjoyed reading through, it is very crisp and detailed article.

    Could you also please add a section how the data gets transitioned through MQTT over Google Protocol Buffer and significance of various QoS ?

  14. Thank you for your explanation, .. I can now connect and publish message to the brooker. But i still fail to publish with retain message. My first byte is 0x31 which is 0x30 for publish and 0001 for retain flag. The message succesfully sent to brooker but not retained.

  15. Hello,
    I am using “mosquitto” implementation for MQTT installed on OpenWrt and Ubuntu.
    I am sending the content of a file using mosquitto_pub and receiving it via mosquitto_sub. I did packet analysis of this transmission and found that,
    1. If the file size 1434 Bytes then the content of the file gets split into TCP packet and MQTT packet.

    Though the received content is intact, I am curious to know that why is this happening.

    1. That is the way TCP/IP works. An MQTT packet needs to be inserted into a TCP/IP packet.
      THe maximum size of the IP Packet is limited by the transmission media. See wiki MTU https://en.wikipedia.org/wiki/Maximum_transmission_unit
      This means that a large MQTT packet will need to be split over several TCP/IP packets.
      AT the other end of the link the packet is reassembled by the MQTT client and appears as a single MQTT packet. Take a look at this tutorial it may help
      https://stevessmarthomeguide.com/basic-networking-course/
      section on frames

Leave a Reply

Your email address will not be published. Required fields are marked *