Python MQTT Client Connections– Working with Connections

python-mqtt-client-connectionsThe MQTT client uses a TCP/IP connection to the broker.

Once the connection is established the client can send data to the broker, and the broker can send data to the client as required.

You can consider a TCP/IP connection to be similar to a telephone connection.

Once a telephone connection is established you can talk over it until one party hangs up.



In this tutorial we will look at connecting to an MQTT broker using the paho python mqqt client.

The Connect Method

To establish a connection to an MQTT broker using the Python client you use the connect method of the client object.

The method can be called with 4 parameters. The connect method declaration is shown below with the default parameters.

connect(host, port=1883, keepalive=60, bind_address=””)

The only parameter you need to provide is the host name.

This can be the IP address or domain name of the broker.

Note: you may need to setup other settings line passwords,last will and testament etc before connecting. See Working with the Client object

The connect method is a blocking function which means that your script will stop while the connection is being established.

Was The Connection Attempt Successful?

When a client issues a connect request to a broker that request should receive an acknowledgment.

mqtt-client-connection-diagram

The broker acknowledgement will generate a callback (on_connect).

If you want to be sure that the connection attempt was successful then you will need to setup a function to handle this callback before you create the connection.

The function should receive 4 parameters, and can be called anything you want.

I have called mine on_connect().

Here is an example function definition:

def on_connect(client, userdata, flags, rc):
    if rc==0:
        print("connected OK Returned code=",rc)
    else:
        print("Bad connection Returned code=",rc)

The client is a client object.

rc (return code) is used for checking that the connection was established. (see below).

Note: It is also common to subscribe in the on_connect callback

Connection Return Codes

  • 0: Connection successful
  • 1: Connection refused – incorrect protocol version
  • 2: Connection refused – invalid client identifier
  • 3: Connection refused – server unavailable
  • 4: Connection refused – bad username or password
  • 5: Connection refused – not authorised
  • 6-255: Currently unused.

Flags and userdata aren’t normally used .The full documentation is here.

Processing The On_connect Callback

To process the callback you will need to run a loop. (see understanding the network loop) .

Therefore the script generally looks like this.

  1. Create Client object.
  2. Create callback function on_connect()
  3. Bind callback to callback function (on_connect())
  4. Connect to Broker.
  5. Start a loop.

Because the callback function is asynchronous you don’t know when it will be triggered.

What is sure however is that there is a time delay between the connection being created, and the callback being triggered.

It is important that your script doesn’t proceed until the connection has been established.

For quick demo scripts I use time.sleep() to wait, and give the connection time to be established.

However for working scripts I process the callback and use it to flag a successful or unsuccessful connection.

So instead of this:

  1. Create connection
  2. Publish message

We have this:

  1. Create connection
  2. Verify successful connection or quit
  3. Publish message and or subscribe

Here is some example code that uses time.sleep() to wait, and give the connection setup time to complete:

import paho.mqtt.client as mqtt    #import client library
def on_connect(client, userdata, flags, rc):
   if rc==0
      print("connected ok")
client = mqtt.Client(“python1”)             #create new instance 
client.on_connect=onconnect  #bind call back function
client.connect(broker_address)               #connect to broker
client.loop_start()  #Start loop 
time.sleep(4) # Wait for connection setup to complete
Other code here
client.loop_stop()    #Stop loop 

Enhancing the Callback

To get better control of the connection I use a flag in the on_connect callback.

The flag I create as part of the client object so it is available throughout the script.

client.connected_flag=False

At the start of the script I set this flag (connected_flag) to False and toggle it to True when the Connection is successful, and back to False when we get a disconnect.

def on_connect(client, userdata, flags, rc):
    if rc==0:
        client.connected_flag=True #set flag
        print("connected OK Returned code=",rc)
        #client.subscribe(topic)
    else:
        print("Bad connection Returned code= ",rc)

We can now use this flag to create a wait loop.

client.connect(broker_address)      #connect to broker
while not client.connected_flag: #wait in loop
     time.sleep(1)

Example Client Connection Script

The following script is a basic client connection script


#!python3
import paho.mqtt.client as mqtt  #import the client1
import time

def on_connect(client, userdata, flags, rc):
    if rc==0:
        client.connected_flag=True #set flag
        print("connected OK")
    else:
        print("Bad connection Returned code=",rc)

mqtt.Client.connected_flag=False#create flag in class
broker="192.168.1.184"
client = mqtt.Client("python1")             #create new instance 
client.on_connect=on_connect  #bind call back function
client.loop_start()
print("Connecting to broker ",broker)
client.connect(broker)      #connect to broker
while not client.connected_flag: #wait in loop
    print("In wait loop")
    time.sleep(1)
print("in Main Loop")
client.loop_stop()    #Stop loop 
client.disconnect() # disconnect

If I run this script this is what I see:

basic-connection-script-example-run

Failed Connection Examples

There are various conditions were the connection can fail to complete. They are:

  • Incorrect client settings e.g. bad password..
  • No network connection
  • Bad Network Connection parameters e.g. bad port number

It is important that these are detected and handled by the connection script.

We are going to look at a few of these and modify our connection code to detect them.

Note: For these examples I will use the Paho MQTT client and the Mosquitto broker.

Connection Failures that Create an Exception

Trying to connect to a broker using a bad IP address or port number will generate a socket error, and raise an exception.

So the first screen shot shows the result of using bad port number.

mqtt-failed-connection-bad-port

This causes a Winsock error in Windows

In Python we can use a Try block to catch this so instead of

client.connect(broker,port) #connect to broker

We use


try:
    client1.connect(broker,port) #connect to broker
except:
    print(“connection failed”)

When the connection attempt failed we would see:

mqtt-failed-connection

Connection Failures Detected Through Return Code

To determine if the connection was successful we need to examine the return code of the on_connect callback.

A return code of 0 is successful, whereas other values indicate a failure.

In the example below we will try to connect to a broker without providing the required authentication.

mqtt-failed-connection-example-authentication

Notice the connection fails and returns a return code of 5 which indicates authentication failure.

You should also notice that because I am using the loop_start() function the client will try to reconnect, but this is pointless as the result will be the same.

So our code should :

  1. Stop the loop
  2. Stop the script

We can stop the loop in the on_connect callback. However to stop the main script we need to set a flag that we can use to exit.

I prefer to use a flag and stop the loop as part of the main script.

Here is what the modified on_connect callback looks like:

def on_connect(client, userdata, flags, rc):
    if rc==0:
        client.connected_flag=True #set flag
        print("connected OK")
    else:
        print("Bad connection Returned code=",rc)
        client.bad_connection_flag=True

Here is the main script modifications to quit.

mqtt.Client.bad_connection_flag=False #
while not client.connected_flag and not client.bad_connection_flag: #wait in loop
    print("In wait loop")
    time.sleep(1)
if client.bad_connection_flag:
    client.loop_stop()    #Stop loop
    sys.exit()

When I run the script this is what I see:

 

bad_connection-example-auth
Using Authentication

If the broker requires username and password authentication (see Mosquitto username and password authentication ) then you need to set this before connecting.

This you do using the username_pw_set() helper function. e.g

client.username_pw_set(username="steve",password="password")
# now can connect

Connecting Using Websockets

Normally the python client will connect using MQTT but it can also connect using MQTT over websockets.

To tell the client to use websockets instead of MQTT use the command

client= paho.Client(“cname”,transport=’websockets’)

instead of simply

client= paho.Client(“cname”)

You will also need to change the port..Websockets generally uses port 9001.

See MQTT over Websockets for more details

Handling Disconnects and Reconnects

A client can disconnect gracefully, if it has no more data to send by sending a disconnect message.

The Paho client provides the disconnect method for this.

It can also get disconnected due to a bad network connection.

If the connection fails for some reason then you will need to decide whether or not you should try to reconnect.

mqtt-connection-Failure

A disconnect triggers the on_disconnect callback which you will need to examine.

This callback takes 3 parameters:

Client- Client object that disconnected

Userdata- user defined data not often used

Return Code (rc)- Indication of disconnect reason. 0 is normal all other values indicate abnormal disconnection

Here is the on_disconnect() code I use:

def on_disconnect(client, userdata, rc):
    logging.info("disconnecting reason  "  +str(rc))
    client.connected_flag=False
    client.disconnect_flag=True

You can see that I simply log it, and then set flags that can be used by the main program to detect the disconnect.

Note: You will need to be calling, or running a loop to trigger the callback.

Reconnecting

Generally you will need to reconnect as soon as possible.

If you run a network loop using loop_start() or loop_forever() then re-connections are automatically handled for you.

A new connection attempt is made automatically in the background every 3 to 6 seconds.

If you call the loop() function manually then you will need to handle the re-connection attempts yourself. See understanding the loop.

You can do this by using a connection flag that is toggled by the on_connect and on_disconnect callbacks.

Client Connection Summary

Taking into account the above our client connection code should.

  • Connect to broker
  • Examine connection status and proceed if good
  • If connection status is bad attempt retry and or quit.
  • Handle disconnects and reconnects

Common Problems

1 .Not seeing any messages or not seeing all expected messages.

Possible causes

  1. You haven’t started a network loop or called the loop() function. Or you haven’t registered or created the callback functions.
  2. You haven’t subscribed to the correct topics or subscription has failed.
  3. Access restrictions are in place.

2.- My messages don’t appear in the order I expected?

Possible causes

  1. The callback functions are async functions which can be called at any time. Use a queue to store the messages and print in one place.

Video

I’ve created a video that covers all of the above you can see it here.

Grateful if you would provide feedback as it will help with subsequent videos

Using The Paho Python MQTT Client Tutorials

Related tutorials and Resources

  • You can find the documentation for the connect method here
  • The Hive MQTT essentials series especially part 3 and part 4 for this tutorial.
Facebooktwittergoogle_plusredditpinterestlinkedinmail

26 comments

  1. Hello Steve
    Thank you for your website. I have a python script running that checks for a csv file, connects to the broker, then sends the file, then disconnects from the program. My problem is that when the next file comes in it detects the file, detected that the broker is disconnected, attempts to connect with the same connect line of code, but I never get an on_connect call back and so it does not publish.

    I think the problem is that even though I get an on_disconnect call back it is not fully disconnected and therefore it won’t make a second connection. If I stop and restart the script it works the first time and disconnects but wont reconnect and publish.

    Thanks for your help
    Mark

  2. Dear Steve,

    I am totally new to MQTT so please forgive my bad explanation of my problem:
    I tried to run your mqtt-data-logger Python script, but it tells me

    No module named ‘paho’.

    I searched in my python 3.7 directory and couldn’t find any trace of ‘paho’.

    I’ve read, that it’s a library, which may be imported or installed, but I don’t know how to acquire it.

    Sidenote:
    I want to export data from MQTT spy into excel on windows. I have a machine, which sends it’s data to the broker. From there I’d like to acquire data for example the current and gather it in an excel sheet for predictive maintenance usage.

    Do you have any tips for me – that’d be so awesome.

    Thanks in advance and best regards

  3. Hi Steve,

    My application had problem: many close_wait on gunicorn and python worker when i publish to mobile (i tried both method: client.publish or publish.single).

    What’s wrong?
    Note: my application sent many messages.

      1. Hi Steve,
        Thanks for your reply.

        1. i used client.loop_forever() for on worker file. That’s subcribe for a channel
        2. i used client.publish or try with publish.single for send message to channel.
        r = publish.single (
        topic=mqtt_topic,
        payload=message,
        qos=0,
        retain=False,
        hostname=self.server,
        port=self.port,
        client_id=””,
        keepalive=self.timeout,
        will=None,
        auth={
        ‘username’: self.user,
        ‘password’: self.password
        },
        tls=None,
        protocol=mqtt.MQTTv311,
        transport=”tcp”
        )
        when can not close the socket: CLOSE_WAIT. i try to use:
        self.client.publish(mqtt_topic, message)
        But it’s same. many close_wait on my application.

        Note: all messages were received but my system will be down soon when can not close the socket

        Like:
        python 3423 htk 8u IPv4 527485 0t0 TCP localhost:35266->localhost:1883 (ESTABLISHED)
        gunicorn 3467 htk 16u IPv4 529421 0t0 TCP localhost:54220->localhost:1883 (CLOSE_WAIT)
        gunicorn 3467 htk 20u IPv4 527133 0t0 TCP localhost:32810->localhost:1883 (CLOSE_WAIT)
        gunicorn 3467 htk 22u IPv4 527053 0t0 TCP localhost:57792->localhost:1883 (CLOSE_WAIT)
        gunicorn 3467 htk 25u IPv4 528742 0t0 TCP localhost:46667->localhost:1883 (CLOSE_WAIT)
        gunicorn 3467 htk 32u IPv4 529522 0t0 TCP localhost:58477->localhost:1883 (CLOSE_WAIT)
        gunicorn 3467 htk 38u IPv4 528894 0t0 TCP localhost:60896->localhost:1883 (CLOSE_WAIT)
        mosquitto 21027 htk 5u IPv4 358221 0t0 TCP localhost:34834->localhost:1883 (ESTABLISHED)

  4. Hi, I like the tutorial. I am struggling though with a publish quirk.

    I have a python script (running under piCore on a Raspberry) calling AWS IOT successfully all the time. However once the script has connected successfully and then I force a network error by taking down my router and the script publishes another message it just goes through as though all was well with a good return code. I do understand that QOS 0 fires and forgets but I am surprised that MQTT doesn’t check that the session is valid.

    Is that how it operates?

    THanks Len

    1. Len
      Yes the client hasn’t detected a problem and so it just tries to publish the message.
      If you use QOS 1 then it will resend the message because it didn’t get a PUBACK.
      Rgds
      Steve

  5. Hi Steve, excellent article. I’m using Paho to connect to my local Mosquitto Broker, however, in my on_disconnect callback, I’m receiving Out of Memory error all the time. It disconnects and the reconnects forever always receiving this message. Have you faced this problem?

    1. Not seen out of memory error. I have seen rapid connect and disconnect. Check that another client isn’t using the same client name. Delay starting the loop until after you try to connect and see if that solves it.
      rgds
      steve

  6. Thanks again steve..seems that connection is finally established (if not,correct me):

    runfile(‘C:/Users/Nikolaos/mqtt_connection.py’, wdir=’C:/Users/Nikolaos’)
    (‘Connecting to broker ‘, ‘192.168.1.18’)
    In main loop
    Connected OK
    Waiting for data receive:

    I will now try to subscribe to MQTT, in order to receive data that already is in move.

    I will inform you, if something goes wrong.
    Thanks again, for your immediate support.

    Regards,
    Nikos.

  7. Hello Steve and thanks for your great articles.
    I try to establish connection using above steps but i receive,as result:

    (‘Connecting to broker ‘, ‘192.168.1.18’)
    [Errno 10054]
    (‘Bad Connection Returned Code=’, 5)
    (‘Bad Connection Returned Code=’, 5)

    Reading your article, this RC(=5) means “Authentication Problem Failure”.
    Installing MQTT Broker in my PC, i used authentication privileges, creating username(s) & password(s).

    I also read that using command:
    username_pw_set(username, password=None
    ,i can set a username and (optionally) a password for broker authentication.
    Have you ever use it before? Could you give me any tip about the code line that has to be added, in order to be succeded, authentication?

    Regards,
    Nikos

      1. Thanks about the immediate reply steve!

        I have to add this code line both in subscribe and line code, right?

        1. You have to add it before the client connects. If you use the same client to subscribe and publish then you only need it once.

  8. I’ve been struggling to find a real client connection status tool, dashboard or anything which will show a list of my clients and their connection status. Clearly the broker via the low-level protocol mechanics maintain an MQTT keepalive table for each client — that’s how Last Will and Testament (LWT) is triggered internally by the broker. However how does one display such table? It seems such need would be essential but I can’t find anything close to it. Everyone refers to the payload as means to track “active” clients but a client can be silent for days yet maintain a connection to the broker and that is only detectable by the broker’s internal keepalive timer and table.

    1. The only way that I see is to publish the active status to a topic when the client connects and set the last will to send a message to that topic when the connection goes down

  9. hey nice tutorial. I would like put forward a question which I came across while working with mqtt.
    I am running the client program and connecting to the broker running in the same pc using the connect() function and starting the loop using the loop_start(). Now I want to disconnect from the broker,so am using disconnect() function, but I see that I am still connected to broker. Can you help me resolve this problem.
    Thanking you.

    1. How do you know that you are still connected? Can you see the broker console screen? After you disconnect you should then stop the loop using loop_stop. However the loop shouldn’t reconnect you as the disconnect was not an error e.g network failure.
      If you send me a copy of the script you are using I can take a look.
      Rgds
      steve

  10. Hi, Steve. Do you have any tutorial for the keep alive time? I read through the documentation on HiveMQ and understand the keep alive interval, PINGREQ and PINGRESP but i’m not sure how are we going to test whether the client(both publisher and subscriber) will be disconnected if one of them didn’t reply with the PINGRESP and PINGREQ.

Leave a Reply

Your email address will not be published. Required fields are marked *