MQTT is a connection oriented protocol which means that you need to establish a connection before you can send data.
However what happens after the data is sent? Do you disconnect? or stay connected?
To answer the question you must understand the data that is being sent.In particular you must consider.
- Is data sent at regular intervals and how often
- Is data sent spontaneously in response to an event.
- Is the data needed in real time
- How long does it take to send the data.
- Number of Clients
- Network Connection Type.
MQTT Connection Times
How long it takes to establish an MQTT connection depends on many factors like network distance,network load,server load.
On a local network a connection time of around 1 second is quite normal.
A connection over the Internet or WAN however can take several seconds and can vary a lot.
Is Data Sent at Regular Intervals and How Often
The more frequently the data is sent then the more likely a connection should be retained and not disconnected.
So if we take a connection time of approx 1 second and data is sent every 5 seconds then the connection should be maintained.
If however data is sent every hour then dropping the connection after sending the data would be the most logical thing to do.
Is Data Sent Spontaneously in Response to an Event
Imagine a fire alarm sensor which only sends data if there is an alarm. If we take a connection time of 1 second then is it reasonable to have the sensor connect and send data and disconnect or should it keep a permanent connection.
How long a delay is tolerable between the event being triggered and the notification being registered.
For a fire alarm 1 second would probably be OK but 5 seconds would be too long
How Long Does it Take to Send the Data
If you are sending large amounts of data e.g a file, movie,Images etc then the data transfer could last many seconds.
If data is being sent at regular intervals e,g every 5 minutes and takes several minutes to send then it would be sensible to keep the connection open.
However if data is sent every hour and lasts 5 minutes then you can safely close the connection after sending.
Number of Clients
The number of clients has a great impact on the broker load and the network load.
If the connection is held open then each client will generate keepalive traffic and have resources on the broker.
It would be possible to minimise this by increasing the keepalive interval but doing this would mean that failed connections would be missed.
Network Connection Type
Devices connected over a low bandwidth connection or a costly connection will need to limit their connection time.
For example, if you are connecting over a mobile network and paying per minute then you don’t want to hold the connection open.
Likewise if multiple clients are connected using a low bandwidth connection then not using it for keep alive traffic would be sensible.
Pros and Cons of Holding a Connection Open
Advantages |
Disadvantages |
| no set up time delay | Set up time delay |
| connection is present and can be used immediately | May not be possible to establish a connection |
| Uses resources on broker even when idle | |
| Uses network resources even when idle (keep alive) | |
| Uses client resources even when idle (keep alive). Client cannot sleep. | |
| Works with all data transmission intervals | |
| Works with event type data | |
| Works in both directions i.e commands can be sent to the sensor. |
Summary
There are many factors to consider when deciding whether or not to hold a client connection open on a permanent basis.
In general I favour closing the connection if possible.
Feedback
Do you have real world experience with this topic? If so I would be grateful for your feedback. Please use the comments form below or if you prefer use the contact page.
Related tutorials and Resources:
- MQTT Sensors and Network Traffic Observations
- MQTT Keep Alive Interval Explained With Examples
- MQTT Clean Sessions and QOS Examples
I have implemented an edge device that seemed would benefit from QOS 1. There was suspicion that my device was not properly processing received PUBLISH State Change messages. But when I implement QOS 1 to be sure that I would receive the PUBLISH messages I found that the Mosquitto Broker was sending the messages, but even if I did not respond with a PUBACK the message did not repeat. Then I find that the MQTT v3.1.1 spec states “When a Client reconnects with Clean Start set to 0 and a session is present, both the Client and Server MUST resend any unacknowledged PUBLISH packets (where QoS > 0) and PUBREL packets using their original Packet Identifiers. This is the only circumstance where a Client or Server is REQUIRED to resend messages. Clients and Servers MUST NOT resend messages at any other time.”
So … a missed QOS 1 message will only be resent at reconnect? That seems nearly useless, as it would require frequent reconnects to make sure messages are sent, and thus would be a “Disadvantage” in the list above. Am I missing something here?
A qos 1 will be resent until a puback. Have you subscribed with a qos of 1?
Rgds
Steve
I subscribed with QOS 1. But I maintain the connection (I do not disconnect) mostly because of the advantages you list above. But Brokers are not required to resend the message until a disconnect/reconnect occurs (per the spec quoted above), and Mosquitto does exactly that: will not resend until a disconnect/reconnect.
Mike
How do you cause the puback not to be sent to the broker or does it just happen?
I’m pretty sure I tested it the other way from client to broker but not broker to client.
Do you have the page number for the excerpt you sent and I will take a quick look.
Rgds
Steve
Hi Steve – Check MQTT spec revision 3.1.1 section 4.4.
My test configuration is Home Assistant (as the client sending PUBLISH state changes), Mosquitto version 2.0.15 as the Broker/Server, and my device as the receiving client. I wrote and manage the firmware in the client device (called a Network Module), so when I became concerned that I might not be seeing DUP messages I added debug to the firmware to do two things: a) inform me if a DUP was received, and b) skip sending a PUBACK for every 6th PUBLISH sent by Home Assistant. This allowed a much more controlled experiment.
Test results:
a) My Network Module firmware indicated there were no DUP messages received as long as I was connected.
b) I was running Mosquitto with the -v option and could see that all PUBLISH messages of interest were at QOS 1.
c) The Mosquitto log output shows the first 5 PUBLISHes from HA entering Mosquitto, exiting Mosquitto to the Network Module, and a PUBACK returning from the Network Module to Mosquitto (and likewise the appropriate QOS 1 and PUBACKs were transacted between Home Assistant and Mosquitto).
d) On the “6th” PUBLISH from HA > Mosquitto > Network Module I can see that by debug prevented the PUBACK response … and even after waiting several minutes there is no DUP message sent to the Network Module. Once I confirmed this I searched more deeply and found a similar scenario described on StackOverflow (https://stackoverflow.com/questions/67589558/mqtt-3-1-1-broker-qos-1-at-least-once-message-redelivery), and searched the 3.1.1 spec to find section 4.4. I have not looked at the version 5 spec, but the comments on Stack Overflow indicate that with version 5 a message resend is not only not optional, it is prohibited until a reconnect.
I find that most people that have written descriptions of QOS 1 or 2, and have included their test scenarios and results, always disconnect / reconnect and then show that the message was resent. I hadn’t noticed that nuance before discovering this spec information. FYI, since it is “optional” to resend without a reconnect in 3.1.1 it is possible there is a broker/server out there somewhere that resends without a reconnect.
Of course the search brought me back to your blogs, which I must say are very good and have provided great pointers to me in my code development and test. And now I’m trying to figure out the best way around this issue as disconnecting and reconnecting frequently does not seem very palatable particularly since you don’t know something was dropped, so you don’t know when you need to go through that process. And I’ve got some users that indicate performance is important. 🙂
Mike
Mike
Tks for the detailed description I will take a look. A a quick check have you tried setting clean connection to true .
You could try the aedes broker that comes with node red.
http://stevesnoderedguide.com/install-mosca-mqtt-broker-node-red
If you don’t use node red I can set one up for you and you can try it.
If will look through the broker settings to see if there is one that fixes it.If not I will reach out to mosquitto and see if they can help.
Rgds
Steve
Steve – there was no Reply button after your last comment in our thread … hopefully this reply doesn’t appear out of sequence (used a reply button further up).
Re: “… have you tried setting clean connection to true”. In my current implementation I start with clean connection true and maintain that connection essentially forever. I have many users running this way but I am starting to doubt this is a good idea.
Re: “aedes broker”. Unfortunately most users are on Mosquitto. So I will have to stay with that.
Re: “node red”. Yes, I use Node-RED to drive my test bed when testing long run times or working on performance limit testing. Most quick tests are with Home Assistant.
Re: “broker settings”. I’ve combed through those pretty carefully and don’t see any settings to automatically trigger QOS > 0 retries. There would have to be an enable and some specification for timeout. I suspect the Mosquitto developers will point to the MQTT spec, and rightfully so.
As I have worked through this problem I think I am coming to the conclusion that this aspect of my application needs modification. Perhaps I need a timeout or other detection to force a disconnect/reconnect (with clean = false), and/or disconnect/reconnect after every PUBLISH from my application. That would eliminate the “forever connection” and should trigger QOS > 0 resend from the broker … unfortunately it would also slow the application as you point out.
Thank you for the interaction here. It has triggered some ideas on a proper solution.
Mike
Mike
Let me know how you get on. I will try and pass it on to the mosquitto developers. I know there are settings in mosquitto that aren’t supported in the spec and so maybe they will see that as worth looking at.
I must admit I can’t think of a reason for it which is why it came as a surprise to me as the broker is essentially a client when publishing messages.
I will try and do some tests over the weekend.
Rgds
Steve
Hi Steve – I’ve decided to stick with QOS 0 as my application can’t afford frequent disconnect/reconnect to find out if any messages are missing. I’m guessing the process was defined to account for network interruptions, and not as a means of verifying that messages are somehow otherwise lost.
If Mosquitto allowed the option of retransmit after a timeout it would be useful, but my application is via other brokers, and I would be stuck again. Of course, I could always write this up in my instructions and recommend Mosquitto (with retry settings) to those that have concerns.
But for now … QOS 0 only.
Mike tks for letting me know and for pointing it out.
I am still trying to figure out the rational behind it.
Rgds
Steve
You don’t mention power? If a client is battery powered and spending most of it’s time in deep-sleep then disconnects are inevitable.
Although I’m fairly new to MQTT (and Node Red), with limited experience, I agree with your assessments, and if power isn’t a problem I just use periodic or event-triggered updates. It hadn’t even occurred to me to disconnect (with plenty of power), though my stuff is still small scale.
It’s great to have this sort of info available and it’s helped me, so thank you for your continued efforts.
Paul Tks for pointing that out it was a big omission.
rgds
steve