MQTT Topic and Payload Design

Designing a topic naming and payload scheme will be an important part of any MQTT deployment.

In this tutorial we look at possible topic and payload design schemes for MQTT networks.

MQTT Network Devices

An MQTT network will consist of devices that can.

  1. Send unsolicited data
  2. Send requested data
  3. Receive command or control data

So if we take a simple light switch as an example device then at a minimum it needs to:

  1. Publish it’s current state i.e is it on or off.
  2. Respond to a command to change state.

To accomplish this we can use the topic name and the message payload.

The topic name can include not only the topic name, but also information describing the message payload. e.g.

we could use a topic name of:

  • sensor1 – Topic has sensor name only
  • or sensor1/status -Topic has sensor name and information about the type of data e.g status information.

So a simple naming scheme for our light switch would be:

The light switch publishes status information on topic

light-switch/status or just light-switch

and receives commands on

light-switch/set

The message payload would be a simple value i.e. ON or OFF.

However we could adopt a different approach and place extra information in the message payload.

So our light switch could publish on the topic:

light-switch

and use a JSON encoded message payload like

{“status”:”ON”} or {“status”:”OFF”}  – Data

{“set”:”ON”} or {“set”:”OFF”}        – –Commands

The important thing to note is that there is always a choice of putting information in the topic field or in the message payload.

Example 1 -Sensors in a Building

If we imagine a single building with say 10 devices e.g Temperature and humidity sensors. Light switches, door locks etc.

We could adopt a topic naming scheme where each sensor publishes on its own topic like:

house/sensor1, house/sensor2 ……….   house/sensor10

If the sensor only published a single value e.g ON,OFF,Open Closed,Temperature etc then our payload could just contain that value.

Thus our living room light switch would publish.

on topichouse/living-room-light

With message payload of – ON or OFF

We could also choose to publish on a single topic and include the sensor ID in the payload as follows:

on topichouse

With message payload Json encoded – {“living-room-light”:”ON”} or {“living-room-light”:”OFF”}

Now how do we control our light switch? The light switch would need to receive commands.

So we need to create a topic to receive commands.

We could use a topic structure of:

  • house/living-room-light/cmd

with the message payload of  – {“SET”:”ON”} or {“SET”:”OFF”}

or

  • house/living-room-light/Set

with the message payload of ON or OFF

Another alternative is use the payload for the device and also to encode the command:

topic – house and

Payload of – {“living-room-light”:{“SET”:”ON”}} or  {“living-room-light”:{“SET”:”OFF”}}

Approaches to Creating a Topic Naming Scheme

When creating the topic naming scheme the approaches are:

  1. Put as much information as possible in the topic field or
  2. Put as much information as possible in the message payload

A topic hierarchy can consists of:

  1. High level topic grouping devices.- optional
  2. Assigned Sensor name – optional
  3. function e.g. status,set,get,cmd –optional

The message payload can consist of

  • Payload data
  • Sensor ID- optional
  • Function – optional

Firstly it is important to note that there currently is no standard topic naming scheme or message format.

Here is a very useful proposal on Github which I would recommend you read.

There is also no right or wrong way to do this.

However you should be very aware that you have far more control when publishing to a topic

Let’s take an example scenario described above and examine two schemes in more detail for our 10 devices/sensors.

Using the sensor name in the topic. i.e

We use the topic scheme house/sensor01……. house/sensor10 and only send data in the payload.

If the broker receives a message published to house/sensor01 it sends it only to the devices subscribed on house/sensor01. This would be sensor01.

However using the topic naming scheme were we group all 10 sensors under the topic house. and use the payload to determining the sensors e.g.

Payload of {“sensor01″:”on”}

In this scenario all sensors/devices would need to subscribe to the house topic.

A message to sensor01 would be sent to all clients subscribed to the house topic which means all 10 sensors. So now we have 10 messages being sent.

Each sensor would need to examine the message to see if is for itself.

Therefore my approach is to

  • Use a topic name for an individual device or small group of devices/sensors.
  • Use a separate topic name for data and commands.
  • Data in the message payload should be device specific were possible.
  • Data in the message payload that relates to multiple attributes of the device is JSON encoded. See packaging data below.

Packaging Topic data

In the case of a complex machine the choice will need to be made whether to publish on multiple topics or use a single topic with packaged JSON data.

Consider this topic tree which I found on a public broker

Test/AppSpin/read false
Test/Position3/read false
Test/TankEmpty/read true
Test/AppOpen/read false
Test/Position7/read false

It could easily be replaced by publishing on a single topic (test) and using a dictionary for the data as follows

{AppSpin_read:False,Position3/read:False,TankEmpty_read:True,AppOpen_read:False, Position7_read:False}

The data is easily read on the other side and it involves 1 publish and not multiple publishes.

I did some tests on network traffic based on packaged data see MQTT Sensors and Network Traffic Observations.

You should note that current IOT/MQTT dashboards take data normally as JSON encoded data, and it is likely to become standard.

See video Publishing and Receiving JSON data with Python.

Message Encryption

An advantage of using the payload and a single topic is that the payload data can be easily encrypted which would protect the topic information as it wouldn’t be public. See Encrypting message payloads

The Homie Convention

This is a Github proposal that attempts to standardise how MQTT devices make themselves discoverable on an MQTT network.

It contains a detailed description of the of the data that a device should publish when it connects to an MQTT broker.

This document should serve as a good guideline for you design.

Public Topics

Currently MQTT implementations are mainly private and the topic naming scheme is chosen by the implementation team.

In the future MQTT is likely to be deployed in a public environment e.g for:

  • Publishing traffic information and updates
  • Arrivals and departure information at airports,train stations etc
  • Beacons in shops
  • etc

In this case there will need to be some kind on naming convention to make it easy for people to locate the service and subscribe.

As far as I am aware there is currently no proposal for this.

Naming Scheme Example

This is the basis of the naming scheme that I use as a starting point in my projects and is based on Sparkplug and the Tasmota naming scheme.

The devices are capable of sending data and receiving commands.

Topic structure

Commands Sent on

base topic/cmnd/device_id/command

command responses on

base topic/response/device_id

or

base topic/response/device_id/result

For simple devices like light switches then I usually put the command value in the topic so to turn the light on we would send the topic

house/cmnd/lounge-light/power/on

in addition I would also send the command value e.g ON in the payload as well.

Devices will send state and info data state data would be the current value (on,off or open,close for a switch) and the info data would include the device friendly names,groups,battery, version,supported commands, etc).

So the device would publish on

base topic/status/device_id/state

and

base topic/status/device_id/info

The payload would contain the information and is either a simple text string or JSON.

JSON is easier as it will cover simple and complex data structures.

Finally I also publish connection state to the topic

base topic/connected/device_id

using the scheme covered in the tutorial- Checking Active MQTT Client Connections

In this manner a control application will know if the device is connected.

Other Naming Scheme Examples:

Owntracks is a mobile phone app that reports geographic location the topic hierarchy design is here and the message format design is here

MQTT Sparkplug is an existing Open Source payload and topic specification by Cirrus Link, and there is an initiative currently under way that will formalise this for IIOT applications.

If you have any examples of naming schemes on actual MQTT deployments I would be grateful if you would share them in the comments.

Was This Useful?

Resources

Related Tutorials:

Please rate? And use Comments to let me know more

11 comments

  1. Hello Steve,
    You have mentioned to subscribe to “house” topic to get all messages published by all sensors.
    I mean this,
    house/sensor01……. house/sensor10
    In this scenario all sensors/devices would need to subscribe to the house topic.
    A message to sensor01 would be sent to all clients subscribed to the house topic which means all 10 sensors. So now we have 10 messages being sent.

    But doesn’t seems to be working.
    Subscribe to individual sensor works fine. But not all by subscribing to “house”

  2. Re: standard topic naming scheme
    The Zigbee Cluster Library specifications defines standard names for data attributes within home automation devices.

  3. Nice explanation.
    Where would you put metadata about the light? For example, extra information about that particular light such as:
    – location “Living Room”
    – wattage “12W”
    – Description “Downlight”
    – Dimmable “Yes”
    Would each of these be a separate sub-topic?
    Or would you have a sub-topic called metadata and publish the metadata as JSON format payload?
    I am interested to know as we have a real world application with lots of variables (topics) and lots of metadata for each variable.
    Using MQTT with topics and JSON payloads seems to be just using MQTT as a message transport, and would significantly reduce the number of topics.
    I would appreciate your thoughts .

      1. Thanks Steve.
        This makes sense. Originally I thought that MQTT topics mirrored the structure of the data, each topic was equivalent to a field or variable in a structure. Simple examples gave me this impression. However I realized after reading your guide that this is too fine-grained. The one line I read that MQTT payloads are generally JSON messages crystallised my understanding. An MQTT topic is more like a message destination, and the MQTT broker is a message conduit. Or an MQTT topic is similar to a topic on an internet forum. The data in the payload can be anything.
        We have many sites and typically have ~150 locations within a site. Each location has 3-4 sensors. So our minimum topic structure would be /site/location/sensor. Any finer grained structure can be the fields in the JSON messages.
        Am I correct in assuming this is the intended way MQTT is to be used?

        1. Phil
          Not sure about intended. The nice thing I find about MQTT is it is simple and very flexible.
          Your proposed structure makes sense.

      2. Thanks. I like the idea of a ‘command’ topic. A device only subscribes, and never publishes to the command topic. In response the device may publish to another topic, say ‘status’.
        This way MQTT is being used as a bidirectional communications channel.

Leave a Reply to Tom Berry Cancel reply

Your email address will not be published. Required fields are marked *