MQTT Retained Messages Explained

mqtt-retained-messages-iconNormally if a publisher publishes a message to a topic, and no one is subscribed to that topic the message is simply discarded by the broker.

However the publisher can tell the broker to keep the last message on that topic by setting the retained message flag.

This can be very useful, as for example, if you have sensor publishing its status only when changed e.g. Door sensor. What happens if a new subscriber subscribes to this status?



Without retained messages the subscriber would have to wait for the status to change before it received a message.

However with the retained message the subscriber would see the current state of the sensor.

What is important to understand is that only one message is retained per topic.

The next message published on that topic replaces the last retained message for that topic.

MQTT Retained Messages Example Overview

In this tutorial We will use a Python script to publish and subscribe to messages.

We will then examine how retained messages work with various flag and QOS settings.

The important thing to note is that we will publish the messages before the client subscribes.

You should also note that I use clean sessions to avoid confusion. If you don’t use clean sessions then you might see messages that have been stored but not retained!

The basic process is this:

  1. Publish message on a topic with retained message flag not set, and set
  2. Subscribe to message on topic
  3. Monitor messages received and analyse results

We will also look at publishing multiple messages and how to remove or delete a retained message.

Referring to the screen shot below:

Example 1 – Retain message flag not set and the new subscriber doesn’t get the message i.e no message received.

Example 2-Retain message flag set and the new subscriber gets the last message as indicated by the message received message.

Example 3– Retain message flag set and we publish several messages OFF,OFF2,OFF3. However the new subscriber only gets the last message OFF3.

retain-message-1

QOS Settings and Retained Messages and Deleting Retained Messages

The screen shot below shows 4 connection examples to demonstrate QOS effects, and then how we delete retained messages.

Example 1: We check if the QOS settings have any effect by setting the QOS to 0 . We see that the retained message is still received and the QOS of the published message has no effect..

Example 2: Now we try to delete the retained message by setting the retained message flag to False. However we notice that he message is still retained.

Example 3: Now we set the message to blank but keep the retained message flag as False. Again the message is still retained.

Example 4:: Lastly we set the message to blank and the retain flag to true which clears the retained message.

retain-message-2

The table below is a summary showing how QOS, the clean session flag, and retained message flag affects what messages are received by a new subscriber to a topic.

mqtt-qos-retain-clean-session-table
Note: Some aspects of how the retain flag is handled is broker implementation dependent.

Python Code

The code to publish with retained flag set is:

client.publish(“bulb1”,”test message “,qos=0,retain=True)

where:

topic=bulb
nessage =test mesage
QOS=0
Reatained message Flag=True

Detecting a Retained Message

When you subscribe to a topic that has a retained message that message will be delivered to your client.

The client can detect that is a retained messages by examining the message.retain flag in the on message callback as shown in the code snippet below:

def on_message(client, userdata, message):
    print("message received  ",str(message.payload.decode("utf-8")),\
          "topic",message.topic,"retained ",message.retain)
    if message.retain==1:
        print("This is a retained message")

Scripts

I have created a Python script that will clear the retained messages on a group of topics, and also report on the topics that have retained messages. See MQTT Retained Messages Tool-Python Script

Publishing Video

Here is a video that I created  that covers publishing messages and also shows the use of the retain flag. Grateful of any feedback.

Common Questions and Answers

Q- How do I remove or delete a retained message?

A- Publish a blank message with the retain flag set to true which clears the retained message.

Q- Does the QOS of a message have any effect on retained messages?

A- No it doesn’t.

Q- How do I know If a message has been retained?

A- You only know when you subscribe to the topic and examine the retained flag.

Q- If I subscribe to a topic and receive the retained message does the broker/server then delete that message?

A- No. It is retained and will be set to new clients when they subscribe.

Q- How do I delete or clear all retained messages on Mosquitto?

A- If you’re not using the persistent database then the easiest way is to stop and restart mosquitto. Otherwise you will need to subscribe to all topics to check and clear the retained messages. You can use my Python script to do this. However this is not recommended on a busy broker.

When to Use Retained Messages

Generally you will publish a message with the retained flag set when the message contains persistent data.

For example a sensor could publish information about itself like firmware version number,IP address, Current state.

This information is unlikely to change and so you only need to publish it once using the retain flag and any new clients can retrieve that information.

See the Homie convention for example sensor topics and attributes.

Summary

The retained message feature is useful feature for keeping the last state of an object, and is especially useful when the state doesn’t change frequently.

Quality of service settings don’t impact retained messages.



MQTT by Example Series

Related tutorials and Resources:

Please rate? And use Comments to let me know more
[Total: 13    Average: 4.3/5]

21 comments

  1. “Q- Does the QOS of a message have any effect on retained messages?
    A- No it doesn’t.”

    Well it does.
    QoS 1 retained messages are handled as QoS 1 publishes and therefore a puback is sent.
    QoS 2 retained messages are also handled as QoS 2 publishes and therefore pubrec, pubrel and pubcomp will be sent.

    This happens for incoming and outgoing “retained messages” on client and server side.

      1. Ah you mean it has no effect on the retained state. I thought you mean that retained messages are not handled by the standard qos 0,1,2 flows.
        So nevermind.

  2. Hi Steve,
    Most of what I know about MQTT and am implementing on my current project comes from you and your MQTT series so firstly, thank you.
    I have a scenario that is probably normal but I wondered if there was anyway of dealing with it. Let me explain.
    I have a sensor which publishes it’s status as a retained message, “Connected” in this case and a last will message of “Disconnected”. If the last message received by the broker is “Connected” and then the broker is stopped, then the sensor stopped and the broker started again, the last retained “connected” message from the sensor is incorrect.
    How is that dealt with in the MQTT world normally?

    1. I assume that the broker is dropping the last retained message which is normal.If it is mosquitto then you can enable persistence which stores status data and should keep the retained message But I can’t remember testing it.
      Rgds
      Steve

  3. Does anyone know how to query the mosquitto MQTT broker for one specific topic for what ever is retained there, then disconnect?
    I have had a fairly extensive look through mosquito_sub man file and couldn’t see anything, unless I missed it.

  4. Hi,

    I am facing an issue where approx. 40,000 topics are retained and value of topics are updated every 15 minutes. topics are retained and its working as desired but in few cases retained messages are not updated and maintains older value although new messages are published on that topic.

    Can you please suggest what could be the reason for this.

          1. Hi,

            I am using v3.7.7, Erlang 21.0.1

            I am facing this issue in Production environment. We are publishing messages with QoS1. I have single connection to publish messages on all 40000 topics with clean session = false, retain flag = true,

            Although I am getting each and every message published in other application subscribing all 40000 topics with QoS2 to manage the queue with clean session = false.

            But When Third Party application is tries subscribing to these topics with QoS=0, Clean Session = False, last published messages is not received ( I understand last published messages in not retained by broker).

          2. If the last messages was published with the retain flag set then it should be sent to the subscribing client.
            With so many topics you may have uncounted a system limitation.
            Are you testing this by publishing and subscribing to a single topic?
            rgds
            steve

          3. With Individual Messages retained works fine in the same broker but at this scale, I am not sure what could be the limitation.

          4. If that is the case I would look at memory issues and message sizes and the server config files. I am not familiar with the Erlang broker so I can’t help there. Is the server in production or still testing?
            Rgds
            Steve

          5. Hi,

            The Server is into Production. I could identify setting to configure retained store sick based or RAM Based from https://www.rabbitmq.com/mqtt.html

            rabbitmq_mqtt.retained_message_store

            But still searching on how to increase the size of retain store in case of disc based storage.

            Thank you !

  5. How can I recognize that a Message was send to the Broker with retain set to true? If I subscribe to the Broker I will get all the retained Messages (retain == true), that works. But if the Client is already subscribed he gets the Message with retain == false. The Client just gets the ‘response’ publish Message from the Server and not the info if the Message is retained. I just want to recognize if a Message is send as a retained Message or not regardless which Client has send it.

    1. Frank
      That doesn’t seem possible. When the client is online the message it receives doesn’t have the retain flag set even though it was sent with the flag set.
      Only when it connects will it see the flag on the message which basically tells the client that it is an old message.

  6. How can I retain all messages for a topic until they are read? Problem scenario: If the consumer drops, and only the last message is retained -> data loss. I would like the broker to keep all messages until they are read by the consumer. I’m so confused why this doesn’t seem to be possible?

Leave a Reply

Your email address will not be published. Required fields are marked *