Understanding And Using MQTT v5 Request Response

Web Applications have a direct connection between the sender and receiver.

The sender is the client (web browser) and the receiver is the server.

They implement a command response message flow were the client(browser) makes a request and the server responds and there is a direct connection between client and server.



MQTT uses a publish and subscribe pattern where there is no direct communication between the sending client and the destination client.

Messages are acknowledged but not by the final receiver.

Request-response-diagram-mqttv5

In MQTTv5 the ability to publish a response topic was added in the publish message which allows you to implement the request/response pattern between clients that is common in Web applications.

We can thus create this configuration.

mqtt-server-request-response

In this configuration an MQTT client can function as a traditional server and have other clients connect to it.

The MQTT client (server) could host a database, central logging etc. functioning as in traditional client server networks that are common using http.

Although there still is no direct connection between the client and server(mqtt client) the use of the response topic can create a virtual connection.

How It Works -Client server Illustration with MQTT

Below is the step by step process on how clients can create their own communications channel and function like a tradition client and server.

In the example client1 is the client and makes the request and client2 is the server and responds.

Additional clients can be configured as per client1. These clients all publish on the common topic and receive responses on their own “private” response topic.

Step 1 -Initialisation

Client 1 – The client

Subscribes to it’s response topic e.g. org/responses/client1.

Client 2 – The Server

Subscribes to common topic e.g. org/common. All clients will publish on the topic.

Step2

Client 1 -The Request

Publishes message to common topic e.g. org/common and adds its response topic to the response topic message property of the publish message.

Python example:

properties=Properties(PacketTypes.PUBLISH)
properties.ResponseTopic='org/responses/client1'
client.publish('org/common',properties=properties)

Correlation data can be added to the message which will be used to tie the request with the response.

Correlation data is bytes and so needs converting. The code below shows how to send a counter.

    properties.CorrelationData=bytes([count])
    print("send count=",count)

Server -The Response

Receives message on the common topic e.g. org/common and extracts the response topic from the message.

Publishes its response on the response topic along with the the correlation data from the received message if present.

Python example:

def on_message(client, userdata, message):

    msg=str(message.payload.decode("utf-8"))
    messages.append(msg)
    print('RECV Topic = ',message.topic)
    print('RECV MSG =', msg)
    response_topic = message.properties.ResponseTopic
    properties=Properties(PacketTypes.PUBLISH)
    properties.CorrelationData=message.properties.CorrelationData
    print('Responding on response topic:', properties)
    #respond
    
    client.publish(response_topic,"server response message",properties=properties)

Step 3

Client 1

Receives message on the response topic e.g. org/common/client1.

Also extracts the correlation data from the message (if present) and uses it to tie the response to the request.

Example code

def on_message(client, userdata, message):

    msg=str(message.payload.decode("utf-8"))
    messages.append(msg)
    print("properties",message.properties)
    count=int.from_bytes(message.properties.CorrelationData,"big")
    print("correlation count=",count)
    print('RECV Topic = ',message.topic)
    print('RECV MSG =', msg)
    client.message_received_flag=True

Client 2,3,4,5 etc

Subscribe to common topic e.g. org/common. Publish on common topic and receive response on their own response topic.

Other Implementation Notes:

Correlation Data

In order for the sender to know what sent message the response refers to it can also send correlation data with the published message.

The receiver e.g. the sever, sends the correlation data back unaltered with the response.

Broker Security

You can configure access control lists on the broker so that only client1 can subscribe to the topic org/responses/client1.

Summary

The new response topic feature of MQTTv5 can be used to create a traditional client server architecture that is common in http networks.

This is useful when a tradition server e,g database server needs to be accessed by multiple clients and the responses need to targeted at the requesting client rather than being published to all clients.

It can also be used to create message style applications

Resources

Python Test Scripts

With the latest version of the Paho client (1.5.1) now supporting MQTTv5 I’ve redone the original scripts using this client and they can downloaded below:

download

The Python scripts below use the gmqtt client available on github and require Python 3.5 and above.

There are three scripts. 2 client scripts and a server script.

download

Related Tutorials

 

Please rate? And use Comments to let me know more

11 comments

  1. Indeed Request-Response-pattern is also possible without MQTT5. Client and backend (what steve names “server”, but the “server” is an MQTT-client, so i prefer naming “backend”) have to know a topic-scheme:

    client:
    subscribes client//to/+
    publishes under client//from/

    backend:
    subscribes client/+/from/+
    publishes under client//to/

    The message-id (which is the correlation-data) can be a random or a counter – it just must be a nonce. The backend receives a message
    i.e. unter topic client/123/from/456 and publishes the response under topic client/123/to/456.

    A JavaScript-lib based on Eclipse-Paho-JS is bublished at
    https://github.com/informatik-aalen/mqtt-fetch.js

    using this lib i can code in Browser-JS:
    var a = await(m.send({nr: 123}));
    So a webbased system is as easy to program as using http-fetch (AJAX). Multiple parallel requests are
    possible.

  2. Hey how this MQTT request-response makes the communication different from publish-subscribe as we are also using publish-subscribe with the same.

    1. It is because it makes it possible to know where to rend a response. It is possible to do this in mqtt v3.1 but you have to implement it in the application itself whereas in v5 it is part of the mqtt protocol.
      Does that make sense?
      Rgds
      Steve

  3. Hi . thx a lot for the tutorial
    i well understand the princip of the request/reply that you explained
    i tried to tun the example attached on the tutorial . but it does seemm to work ,
    the issues is on the server side , since it cant publish the response , every publishing even if is not the response is also not working as explained

      1. Hi i am using eclipse-mosquitto , i got that image from a docker container , it implements the MQTT protocol versions 5.0, 3.1.1 and 3.1.
        my intent is to enable microservices communications between services in docker swarm ( a cluster of service in a virtual network )
        rgds
        Joel Fankam

          1. Hi
            which broker do you recommend me to use and how do i know which version is implemented on the version i will be running ?

Leave a Reply

Your email address will not be published. Required fields are marked *