MQTT Retained Messages Explained

mqtt-retained-messages-iconNormally if a publisher publishes a message to a topic, and no one is subscribed to that topic the message is simply discarded by the broker.

However the publisher can tell the broker to keep the last message on that topic by setting the retained message flag.

This can be very useful, as for example, if you have sensor publishing its status only when changed e.g. Door sensor. What happens if a new subscriber subscribes to this status?

Without retained messages the subscriber would have to wait for the status to change before it received a message.

However with the retained message the subscriber would see the current state of the sensor.

What is important to understand is that only one message is retained per topic.

The next message published on that topic replaces the last retained message for that topic.

MQTT Retained Messages Example Overview

In this tutorial We will use a Python script to publish and subscribe to messages.

We will then examine how retained messages work with various flag and QOS settings.

The important thing to note is that we will publish the messages before the client subscribes.

You should also note that I use clean sessions to avoid confusion.

If you don’t use clean sessions then you might see messages that have been stored but not retained!

The basic process is this:

  1. Publish message on a topic with retained message flag not set, and set
  2. Subscribe to message on topic
  3. Monitor messages received and analyse results

We will also look at publishing multiple messages and how to remove or delete a retained message.

Referring to the screen shot below:

Example 1 – Retain message flag not set and the new subscriber doesn’t get the message i.e no message received.

Example 2-Retain message flag set and the new subscriber gets the last message as indicated by the message received message.

Example 3– Retain message flag set and we publish several messages OFF,OFF2,OFF3. However the new subscriber only gets the last message OFF3.

retain-message-1

QOS Settings and Retained Messages and Deleting Retained Messages

The screen shot below shows 4 connection examples to demonstrate QOS effects, and then how we delete retained messages.

Example 1: We check if the QOS settings have any effect by setting the QOS to 0 . We see that the retained message is still received and the QOS of the published message has no effect..

Example 2: Now we try to delete the retained message by setting the retained message flag to False. However we notice that he message is still retained.

Example 3: Now we set the message to blank but keep the retained message flag as False. Again the message is still retained.

Example 4:: Lastly we set the message to blank and the retain flag to true which clears the retained message.

retain-message-2

The table below is a summary showing how QOS, the clean session flag, and retained message flag affects what messages are received by a new subscriber to a topic.

mqtt-qos-retain-clean-session-table
Note: Some aspects of how the retain flag is handled is broker implementation dependent.

Python Code

The code to publish with retained flag set is:

client.publish(“bulb1”,”test message “,qos=0,retain=True)

where:

topic=bulb
nessage =test mesage
QOS=0
Reatained message Flag=True

Detecting a Retained Message

When you subscribe to a topic that has a retained message that message will be delivered to your client.

The client can detect that is a retained messages by examining the message.retain flag in the on message callback as shown in the code snippet below:

def on_message(client, userdata, message):
    print("message received  ",str(message.payload.decode("utf-8")),\
          "topic",message.topic,"retained ",message.retain)
    if message.retain==1:
        print("This is a retained message")

Scripts

I have created a Python script that will clear the retained messages on a group of topics, and also report on the topics that have retained messages.

download

Publishing Video

Here is a video that I created  that covers publishing messages and also shows the use of the retain flag.

Grateful of any feedback.

Common Questions and Answers

Q- How do I remove or delete a retained message?

A- Publish a blank message with the retain flag set to true which clears the retained message.

Q- Does the QOS of a message have any effect on retained messages?

A- No it doesn’t.

Q- How do I know If a message has been retained?

A- You only know when you subscribe to the topic and examine the retained flag.

Q- If I subscribe to a topic and receive the retained message does the broker/server then delete that message?

A- No. It is retained and will be set to new clients when they subscribe.

Q- How do I delete or clear all retained messages on Mosquitto?

A- If you’re not using the persistent database then the easiest way is to stop and restart mosquitto. Otherwise you will need to subscribe to all topics to check and clear the retained messages. You can use my Python script to do this. However this is not recommended on a busy broker.

When to Use Retained Messages

Generally you will publish a message with the retained flag set when the message contains persistent data.

For example a sensor could publish information about itself like firmware version number,IP address, Current state.

This information is unlikely to change and so you only need to publish it once using the retain flag and any new clients can retrieve that information.

Using Retained Messages as A Data Store

I have seen applications were the retained messages have been used as a data store.

If you take a look at my tutorial on home assistant and MQTT you will see this in action.

Although it is possible you should be aware of the consequences i.e what happens if someone removes the retained messages as part of a clean up?

Retained Message Format Proposal

Because retained messages are being used as a semi persistent/persistent data store then it would be useful in my option to publish extra information with the actual message payload.

As minimum you should publish a timestamp and a deletion allowed flag.So we would have

{“payload”:payload,”timestamp”:timestamp,”delete_allowed”:True/False}

Deletion utilities could check these and delete very stale messages but not ones that shouldn’t be deleted.

Summary

The retained message feature is useful feature for keeping the last state of an object, and is especially useful when the state doesn’t change frequently.

Quality of service settings don’t impact retained messages.



MQTT by Example Series

Related tutorials and Resources:

Please rate? And use Comments to let me know more

44 comments

  1. Hi, I am trying to publish the text “o” as “retained” via EasyMQTT mobile phone app. With my µC (code below) I can receive the text only in case the µC is powered on while sending the message (means: 2 clients are connected to the broker with the same topic). In my understanding the sent “o” is stored on the MQTT server until I fetch it with my µC. But as the sensor (µC) is powered on every 15 minutes I cannot receive the “o”. Any help is appreciated!

    my code (Energia)
    void messageArrived(MQTT::MessageData& md)
    {
    MQTT::Message &message = md.message;
    sprintf(printbuf, “Message %d arrived: qos %d, retained %d, dup %d, packetid %d\n”,
    ++arrivedcount, message.qos, message.retained, message.dup, message.id);
    // Serial.print(printbuf);
    sprintf(printbuf, “Payload %s\n”, (char*)message.payload);
    // Serial.print(printbuf);
    if (printbuf[8] == ‘o’) {
    for (k=0; k<3; k++) { // this is to indicate heater is on
    delay (200);
    digitalWrite(RED_LED, HIGH);
    delay (200);
    digitalWrite(RED_LED, LOW);
    }
    }

    1. The retained message should be stored on the server and you should get it when you connect. Have you tried publishing with a another client, Using another broker and using a different client to see where exactly the problem is.i.e
      is the he receiver,the publisher or the broker
      Rgds
      Steve

  2. Just want to drop a note to say “thank you”, not only for creating the article, but also for answering everyone’s questions in the comments for at least FIVE years! I like your style, sir!

    1. There isn’t a time limit but some brokers may set one but I don’t believe there is a setting currently for mosquitto.
      Rgds
      Steve

  3. For OTA firmware update on iot devices in field, we send a retained message to the MQTT broker with payload having the new firmware file and file download location. The idea is that if any device is off, then it will receive the OTA update message when it switches on.
    However, this retain message for OTA update is also being send to iot devices when they switch on for the second time (and already have updated firmware).
    Can you suggest how to ensure that the retained message is sent to a particular iot device only one.
    Thanks in advance.

    1. I don’t see a way unless the client sends a blank message to that topic after it has done the upgrade. Also if it did that then each client would require a separate topic for the OTA which I don’t think you have.
      Do you have control over the client/device software.
      Rgds
      Steve

  4. Hi I am sorry to say, the demo code is a bit out of date. Fx. the suback_flag is non existent on the client.
    Line 172 in retained-messages.py
    while not client.suback_flag: # wait for subscribe to be acknowledged

  5. Hi, it happened to me that the server (mosquitto) has restarted due to a lack of voltage, but unfortunately it has also lost the Retained messages, is there a possibility to keep them even after the server is restarted?

  6. I want to be able to use retained messages as a means of saving state for a MQTT client.
    I e when Idle, I occasionally send a retained message containing my current state to the broker
    If the publishing client dies ( say due to a power issue or memory leak), I’d like the client to reboot and then re-connect and receive the last retained message and re-set its state from that.
    What are the combination of session and retention I need to set ?
    I believe I need a clean session to prevent the broker from assuming I already have it but maybe a dirty session enables the broker to realise there are a stack of messages I can subscribe to and read.
    Does being the publisher also prevent me being the client ?

    1. The client needs to connect with a clean session and then subscribe to the topic receive the retained message and set it’s state.
      Does that make sense?
      rgds
      steve

  7. Very interesting course. Thanks for creating it. One question if you don’t mind. I’m a bit confused about the retain flag vs. the will retain flag. From what I have read, they are distinct flags. The retain flag is used to retain regular published messages on the broker whereas will retain is used to retain the last will and testament message. The mosquitto_pub client treats them separately, and the specification seems to document them separately as well. I might have this wrong so I was hoping that you could confirm. Thanks.

    1. That is correct. The last will message is simply a message but it is sent by the broker and not the client. So it has the same message properties as a normal message.
      rgds
      sreve

  8. Hi Steve,
    I do not understand what does the meaning of ‘Published message always received’ the yellow line : “Yes all messages” in your table.
    I tried to see this behavior with simulators without any success, this is what I did:
    Publisher (clean session is false) –> published 2 messages on topic ‘fountain’ Qos 2, and retain is set “test message1”
    “test message2”
    Subscriber (clean session is false) –> subscribed to topic ‘fountain’ with Qos 2
    The result:
    The Subscriber got “test message2”

    However, according to the table the Subscriber supposed to get ‘all the messages’ – what did I missing?

    Thank you,
    Maayan

    1. Dear Steve,
      Thankyou for this content. Please find some doubts . Kindly reply
      1 ) When 5 clients subscribe to a topic and listens on the topic, if data is continuously published on the topic with reatined flag set true. So will all clients get the retained msg or first client using the topic will get it? 1 msg per topic or all client get that last message?

      2) one client is subscribed to 5 topics then will it get the messages published on all topics ? Can we identify the message and topics seperately ? Can it get all the reatined messages?

      3 ) is there any mechanism to notify the subscriber about any changes in the published data? Or do we need to call the publish function repeatedly if any change in the data to publish?

      Best regards,
      Mohith

      1. Hi
        1)The retained message flag is really only useful for old messages. If 5 clients are subscribed to a topic and a publisher publishes to that topic with retained flag set they get the message just as they would without using the retain flag.
        However if a 6th subscriber subscribes after the publisher has finished sending it will get the last message.
        2) Yes it will and the topic is part of the received payload so you can separate them.
        3)Subscribers are notified on any new messages published provided they are connected. That is how MQTT works.
        Rgds
        Steve

        1. Okay thank you. Another doubt too . Kindly reply .

          1) can we create topics dynamically ? And add / delete new topics to publish / subscribe data as we wish based conditions? If yes how to do that?

          2 ) My use case is : i want 5 devices to connected to the cloud server using MQTT. Each device will subscribe to either 5 topics or common topic as per some situations. Then cloud has to publish data to all these topics. It may publish again if any change occur in the data which is published. Like wise I need to continue until device stops listening ( subscribing) Then can i use MQtt for this purpose?

          3) I can also unsubscribe right?

          Best regards
          Mohith

          1. Topics are created when someone publishes on that topic. If no one subscribes to a topic then in doesn’t stop someone publishing. MQTT is like broadcast radio or TV.
            You cannot tell if anyone is listening unless you setup your own messaging scheme. Just like with TV and radio.
            Rgds
            Steve

  9. “Q- Does the QOS of a message have any effect on retained messages?
    A- No it doesn’t.”

    Well it does.
    QoS 1 retained messages are handled as QoS 1 publishes and therefore a puback is sent.
    QoS 2 retained messages are also handled as QoS 2 publishes and therefore pubrec, pubrel and pubcomp will be sent.

    This happens for incoming and outgoing “retained messages” on client and server side.

    1. Not sure what you mean.
      A retained message is retained regardless of the QOS level that it was published with.
      Rgds
      Steve

      1. Ah you mean it has no effect on the retained state. I thought you mean that retained messages are not handled by the standard qos 0,1,2 flows.
        So nevermind.

  10. Hi Steve,
    Most of what I know about MQTT and am implementing on my current project comes from you and your MQTT series so firstly, thank you.
    I have a scenario that is probably normal but I wondered if there was anyway of dealing with it. Let me explain.
    I have a sensor which publishes it’s status as a retained message, “Connected” in this case and a last will message of “Disconnected”. If the last message received by the broker is “Connected” and then the broker is stopped, then the sensor stopped and the broker started again, the last retained “connected” message from the sensor is incorrect.
    How is that dealt with in the MQTT world normally?

    1. I assume that the broker is dropping the last retained message which is normal.If it is mosquitto then you can enable persistence which stores status data and should keep the retained message But I can’t remember testing it.
      Rgds
      Steve

  11. Does anyone know how to query the mosquitto MQTT broker for one specific topic for what ever is retained there, then disconnect?
    I have had a fairly extensive look through mosquito_sub man file and couldn’t see anything, unless I missed it.

  12. Hi,

    I am facing an issue where approx. 40,000 topics are retained and value of topics are updated every 15 minutes. topics are retained and its working as desired but in few cases retained messages are not updated and maintains older value although new messages are published on that topic.

    Can you please suggest what could be the reason for this.

          1. Hi,

            I am using v3.7.7, Erlang 21.0.1

            I am facing this issue in Production environment. We are publishing messages with QoS1. I have single connection to publish messages on all 40000 topics with clean session = false, retain flag = true,

            Although I am getting each and every message published in other application subscribing all 40000 topics with QoS2 to manage the queue with clean session = false.

            But When Third Party application is tries subscribing to these topics with QoS=0, Clean Session = False, last published messages is not received ( I understand last published messages in not retained by broker).

          2. If the last messages was published with the retain flag set then it should be sent to the subscribing client.
            With so many topics you may have uncounted a system limitation.
            Are you testing this by publishing and subscribing to a single topic?
            rgds
            steve

          3. With Individual Messages retained works fine in the same broker but at this scale, I am not sure what could be the limitation.

          4. If that is the case I would look at memory issues and message sizes and the server config files. I am not familiar with the Erlang broker so I can’t help there. Is the server in production or still testing?
            Rgds
            Steve

          5. Hi,

            The Server is into Production. I could identify setting to configure retained store sick based or RAM Based from https://www.rabbitmq.com/mqtt.html

            rabbitmq_mqtt.retained_message_store

            But still searching on how to increase the size of retain store in case of disc based storage.

            Thank you !

  13. How can I recognize that a Message was send to the Broker with retain set to true? If I subscribe to the Broker I will get all the retained Messages (retain == true), that works. But if the Client is already subscribed he gets the Message with retain == false. The Client just gets the ‘response’ publish Message from the Server and not the info if the Message is retained. I just want to recognize if a Message is send as a retained Message or not regardless which Client has send it.

    1. Frank
      That doesn’t seem possible. When the client is online the message it receives doesn’t have the retain flag set even though it was sent with the flag set.
      Only when it connects will it see the flag on the message which basically tells the client that it is an old message.

  14. How can I retain all messages for a topic until they are read? Problem scenario: If the consumer drops, and only the last message is retained -> data loss. I would like the broker to keep all messages until they are read by the consumer. I’m so confused why this doesn’t seem to be possible?

Leave a Reply

Your email address will not be published. Required fields are marked *