Understanding MQTT Topics

MQTT topics are a form of addressing that allows MQTT clients to share information.

MQTT Topics are structured in a hierarchy similar to folders and files in a file system using the forward slash ( / )as a delimiter.

Using this system you can create a user friendly and self descriptive naming structures of you own choosing.

Topic names are:

  • Case sensitive
  • use UTF-8 strings.
  • Must consist of at least one character to be valid.





Except for the $SYS topic there is no default or standard topic structure.

That is there are no topics created on a broker by default, except for the $SYS topic.

All topics are created by a subscribing or publishing client, and they are not permanent.

A topic only exists if a client has subscribed to it, or a broker has a retained or last will messages stored for that topic.

The $SYS topic

This is a reserved topic and is used by most MQTT brokers to publish information about the broker.

They are read-only topics for the MQTT clients. There is no standard for this topic structure but there is a guideline here that most broker implementations seem to follow.

I have created a node-red dashboard that monitors the $SYS topic and produces a display as shown below:

mosquitto-broker-topic-monitor

Here is the flow:

download

Subscribing to Topics

A client can subscribe to individual or multiple topics.

When subscribing to multiple topics two wildcard characters can be used. They are:

  • # (hash character) – multi level wildcard
  • + (plus character) -single level wildcard

Wildcards can only be used to denote a level or multi-levels i.e /house/# and not as part of the name to denote multiple characters e.g. hou# is not valid

Topic naming Examples:

Valid Topic subscriptions

Single topic subscriptions

  • /
  • /house
  • house/room/main-light
  • house/room/side-light

Using Topic Wildcards

Subscribing to topic house/#

Covers

  • house/room1/main-light
  • house/room1/alarm
  • house/garage/main-light
  • house/main-door
  • etc

Subscribing to topic house/+/main-light

covers

  • house/room1/main-light
  • house/room2/main-light
  • house/garage/main-light

but doesn’t cover

  • house/room1/side-light
  • house/room2/side-light

Invalid Topic Subscriptions

  • house+ – Reason- no topic level
  • house# – Reason- no topic level

Publishing to Topics

A client can only publish to an individual topic. That is, using wildcards when publishing is not allowed.

E.G- To publish a message to two topics you need to publish the message twice

When are Topics Created

Topics are created dynamically when:

  • Someone subscribes to a topic
  • Someone publishes a message to a topic with the retained message set to True.

When are Topics Removed from a Broker

  • When the last client that is subscribing to that broker disconnects, and clean session is true.
  • When a client connects with clean session set to True.

Republishing Topic Data

This is likely to be done when changing or combining naming schemes.

The idea is that a client would subscribe to a topic, e.g.

hub1/sensor1 and republish the data using a new topic naming of house1/main-light.

Video A Beginners Guide to MQTT Topics

Common Questions and Answers

Q- How do I subscribe to all topics?

A-Subscribe to #

Q- How Do I subscribe to all $SYS topics?

A-Subscribe to $SYS/#

Q- Should I start my Topic hierarchy with a /.

A- It is not necessary and just adds another level to the structure.

Q- Can I get list of all topics on a broker?

A- Not unless you subscribe to all topics and scan them.

Q- Can I tell who is subscribed to a topic?

A– No

Q- How do I discover topics?

A- There is currently no mechanism for that except as described in list all topics.

How MQTT Works << Prev  ..Next .>>MQTT Publishing,Subscribing and Message Exchange

Related Tutorials and Resources:

Please rate? And use Comments to let me know more
[Total: 31   Average: 4.1/5]

37 comments

  1. Hello Steve
    I have a query.
    Currently I am working on a system with the following functionality
    1. Mulitple sensors are sending data to the HTTP service (on my server)
    2. The service processes the data and stores the data into a table.
    3. This data is further processed by other services that run continuously
    4. On my browser in node js, the front end application sends request to another service on my server which then responds back with the final processed data and other information related to each of these sensors and it is then displayed on the front end

    Can you suggest how I can do this functionality in MQTT. I am looking at your tutorials and so far just installed Mosquitto.
    Can you guide me with the next steps. How do I trigger these functionalities for processing the data received from each of these sensors. I want to even use subscription on front end so that the broker pushes data to the front-end only when there is change and not vice-versa.
    Basically I want to know
    1. How to create topics in Mosquiito
    2. when a data is received for a topic, it should be processed by my applications on the back-end (how to trigger them)
    3. Then how to tell Mosquitto that the data is modified, so it should push it to the browser (/android and IOS apps) front end if at all they are open / logged in
    If you can provide me with any links to your tutorials that you have already shown how to do these it would very helpful.
    I am complete newbie to MQTT. I am good with linux, C, C++, JAVA, Python. My current services are written in JAVA.

    Thanks in advance 🙂

    1. Hi
      Suggest you go to the course page
      http://www.steves-internet-guide.com/mqtt-basics-course/
      and work your way through it to get a better understanding.

      For mosquitto or any MQTT broker it does message transfer and that is all. It doesn’t process the message so if you want to only send change then you need to do that on the sensor.
      This tutorial may help
      http://www.steves-internet-guide.com/simple-controllable-mqtt-sensor/

      On the server side for display then the cloud based server must be capable of receiving mqtt which unless it is a virtual server it probably can’t, So you might need to design it so the sensor data is collected and processed locally and sent for display via http to the web server.
      Let me know how you get on
      Rgds
      Steve

  2. Hello Steve,
    Greetings!
    I am following your documentation, I am new born baby in this Mosquitto MQTT app installation and configuration.
    Got struck on my Cluster and Scale configuration.
    Steps I followed.
    1. Edited “/etc/mosquitto/mosquitto.conf”
    ## maximum connection to mqtt allowed
    max_connections 20000
    2. /etc/security/limits
    #
    #
    * soft nofile 20000
    * hard nofile 20000
    3. Based troubleshooting guidance I followed the links and created new file under
    /etc/init/mosquitto.conf, added the lines
    limit nofile 16384 16384 
    With all these settings, I didn’t successes still I am getting the same error
    cat /proc/3003/limits
    Limit Soft Limit Hard Limit Units
    Max cpu time unlimited unlimited seconds
    Max file size unlimited unlimited bytes
    Max data size unlimited unlimited bytes
    Max stack size 8388608 unlimited bytes
    Max core file size 0 unlimited bytes
    Max resident set unlimited unlimited bytes
    Max processes 14679 14679 processes
    Max open files 1024 4096 files
    Max locked memory 65536 65536 bytes
    Max address space unlimited unlimited bytes
    Max file locks unlimited unlimited locks
    Max pending signals 14679 14679 signals
    Max msgqueue size 819200 819200 bytes
    Max nice priority 0 0
    Max realtime priority 0 0
    Max realtime timeout unlimited unlimited us

    Thank you and appreciate, if you can guide me, what is the missed part here.

  3. assume we have 100 apartments and each apartment has 10 sensors. propose a topics scheme (how will you design your topic names) for the subscribers which are.
    1- tenants (a tenant is a person who is renting the apartment and should have access to all the sensors in his apartment)
    2- Maintenance company (the company should have access to all the sensors in all the apartments)
    How it works ?

    1. Use a topic base of apartments. The maintenance company will have access to this.
      for each apartment use:
      apartments/tenant1
      apartments/tenant2
      each tenant subscribes to his own topic e.g tenant 1 will subscribe and publish on
      apartments/tenant1

      You can restrict access using ACLs (access control Lists)

  4. Hello Steve,
    I publish data to 2 topics from 2 different clients and later I want to subscribe to these topics and save the data in 2 tables named gnss and acc in a database.
    For example:
    Client 1 will publish data to topics: my/device1/gnss and my/device1/acc
    Client 2 will publish data to topics: my/device2/gnss and my/device2/acc
    Now I subscribe to all these 4 topics by using multilevel wildcard: my/#
    But how can I save the data of all clients into 2 tables gnss and acc irrespective of the client.
    Code:
    if msg.topic == “my/device1/gnss”:
    sql = “””INSERT INTO gnss (topic, gnssdata) VALUES (%s,%s)”””
    tuple = (msg.topic, msg.payload)
    if msg.topic == “my/device1/acc”:
    sql1 = “””INSERT INTO acc (topic, accdata) VALUES (%s,%s)”””
    tuple = (msg.topic, msg.payload)
    The above code will just save the data from Client1. How to do this for all the clients if I am having various clients which are publishing data to the broker.
    I tried using: if msg.topic == “my/+/gnss”:
    sql = “””INSERT INTO gnss (topic, gnssdata) VALUES (%s,%s)”””
    tuple = (msg.topic, msg.payload)
    But this doesn’t work.

    1. Hi
      Just do a find on the topic for “acc” and also for “gnss” and store accordingly.
      Does that make sense.
      Rgds
      Steve

  5. Hi Steve, good intro, thanks

    is there access control of who can subscribe the topic and publish msg on the topic?

  6. Do all publishers share the same namespace on the broker? What happens if two different publishers happen to chose the same name for different items?

  7. Hi Steve,

    I was wondering if you have any idea why publishing a single message to multiple topics is not currently supported?

    Thanks!

  8. Hello Steve,

    when I clear all topics by connecting a client with clear session = true, do other clients get any notification about this? And also do the clients have to resubscribe after a clear (I assume yes)?

    Thanks!

    1. The clean session flags only applies to that client and not to the clients subscribed to a particular topic.
      So no.
      The client will need to subscribe to the topics when it connects again but not other clients.
      Does that make sense?
      rgds
      steve

  9. I am using the Eclipse Paho gateway between mqtt and mqtt-sn. The Paho gateway forwards to the Eclipse mqtt broker.

    When I add a subscribe client to the Eclipse broker with a subscribe to cmd/+ , everything works as expected. When I publish a command from my mqtt-sn app like cmd/A, The subscribed client recieves it with topicname cmd/A.

    But the other way around does not work. When my mqtt-sn app subscribes to data/+, it registers a topic with e.g. topic-id 1. When my mqtt broker client publishes something on data/A, my mqtt-sn app receives a message on a new topic-id 2. And I don’t have or did recieve the topic-name of this topic-id (no way to find the A).

    From the mqtt-sn-1.2 specs:
    “A GW sends a REGISTER message to a client if it wants to inform that client about the topic name and the assigned topic id that it will use later on when sending PUBLISH messages of the corresponding topic name. This happens for example when the client re-connects without having set the “CleanSession” flag or the client has subscribed to topic names that contain wildcard characters such as # or +”

    For some reason these REGISTER messages were not sent, or my app missed them.

    Anyone encountered a similar problem?

    1. Hi
      Sorry for the late reply. If you enable debugging on the broker you should wee the messages being send to your client.

      1. Thanks for your help!
        The sn-client is a nRF52840 DK. The dump is created with a client created with the NordicThread sdk version 2.0.0. Today I tried with 3.1.0. Behaviour is the same.
        I annotated with ///
        ***************************************************************************
        * MQTT-SN Transparent Gateway
        * Part of Project Paho in Eclipse
        * (http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt-sn.embedded-c.git/)
        *
        * Author : Tomoaki YAMAGUCHI
        * Version: 1.0.0
        ***************************************************************************
        20190707 110523.239 PahoGateway-01 has been started.
        ConfigFile: /etc/paho-mqtt-sn-gateway.conf
        SensorN/W: Gateway Port: 47193 Broadcast Address: ff03::1 Interface: wpan0
        Broker: 198.41.30.241 : 1883, 8883
        RootCApath: (null)
        RootCAfile: (null)
        CertKey: (null)
        PrivateKey: (null)
        20190707 110525.830 CONNECT MySensGw 10 14 00 04 4D 51 54 54 04 02 00 3C 00 08 4D 79 53 65 6E 73 47 77
        20190707 110526.241 CONNACK MySensGw 05 00

        /// sn-client registers MySensTx/9 for publishing. Not part of the problem
        20190707 110526.267 REGISTER 0001 MySensGw 0B 00 01 00 01 00

        /// sn-client registers MySensRx/+ fore receiving
        20190707 110530.000 REGISTER 0002 MySensGw 0B 00 02 00 02 00

        /// sn-client subscribes to MySensRx/+
        20190707 110530.673 SUBSCRIBE 0003 MySensGw 82 0F 00 03 00 0A 4D 79 53 65 6E 73 52 78 2F 2B 01
        20190707 110530.835 SUBACK 0003 MySensGw 13 20 00 02 00 03 00

        /// sn-client publishes MySensRx/9 and receives a Register. How does the client knows about this MySensRx/9 ?
        /// This was sent to this client with a publish from a different client on a different system
        /// I am also missing Ack
        20190707 110547.824 PUBLISH 0000 MySensGw 0A 00 03 00 01 9C 04 C0 73 01

        20190707 110547.825 PUBLISH 0000 —> MySensGw 0C 00 00 03 00 00 30

        20190707 110550.280 REGACK 0001 <— MySensGw 0B 00 03 00 01 00
        20190707 110626.264 PINGREQ MySensGw C0 00

        1. I think it may be because you are using wild cards for the subscribe. The register applies to MySensRX/+
          If you still have problems I will need to copy your setup and give it a try. Let me know.
          Rgds
          Steve

    2. I solved the problem.

      May be I was not clear on that, but I am trying to use the OpenThread implementation with target Nordic nRF52840 using the Nordic Thread and Zigbee SDK.

      Actually, I found three problems:
      – In the syslog of Paho, entries with the same timestamp are in random order.
      – My test application did not handle MQTTSN_EVENT_REGISTER_RECEIVED properly.
      – The register_handle method of Nordic in their file mqttsn_packet_receiver.c tries to get the topic_name from topic.cstring while it actually can be found in topic.lenstring.data. I don’t know if this is a Paho or Nordic bug.

      It is working now. I can subscribe on my nRF52840 MQTT-SN client with a wildcard and send messages from a MQTT client on the internet to my client in an OpenThread mesh network. And the topic name is registered on the client, so,when I am subscribed to cmd/+, I can find out by the topic_id whether the received message was published to cmd/test1 or cmd/test2.

      Thanks again.

  10. Hi Steve,

    Great site. Can I have multiple wildcards? For example, is this valid:

    house/+/furnace/+/temperature

    To get
    house/room1/furnace/alarm1/temperature
    house/room2/furnace/alarm2/temperature

    Also you said there is no wildcards within topics, is there any work around for this? Anyw

    1. Yes
      You can use wildcards at multiple levels.
      There is no workaround for wildcards in topic names as far as I know.
      rgds
      steve

  11. Hello steve and anybody, who can tell me the topic ‘house/room1’ and ‘house/room1/ are the same?

    1. hi
      They are different. It is effectively house/room1?. just as house/room1 and /house/room1 are different
      rgds
      steve

  12. Hello, this was a great intro to MQTT.
    Can Brokers/Servers have licensing server associated to each /topic? Does it have this facility, lend easily to this method of billing or does it need to architecting?
    Kind Regards
    Ivan

  13. Hello Steve,

    Thank you for this information! I also had a question : If a subscribe to a wildcard topic in Node-Red is there a way to separate the different topics coming through the subscribe node? For example if I am subscribe to sensor/#. How could I split it into sensor/temperature and sensor/humidity? Thanks!

    1. Yes
      The easiest way is to use the switch node and use topic not payload in the property field.
      If you have problems use the ask steve link and I’ll email you a screenshot and flow.
      rgds
      steve

  14. Hello Steve,

    first thanks for all the info. But I wonder, are usually topics implemented to be able to retrieve their levels?

    For example, with JS or Python Paho implementations of MQTT Client, if I were to send the ClientID as part of a topic (ping/clientID) for a ping subscription (ping/+), should I be able to retrieve the clientID easily, or would I have to split the topic as a string?

    Sorry for the specific question, but it strikes me as weird that something like this isn’t a little better implemented (why have a topic hierarchy if you are not going to have an specific object for it?).

    1. You would have to split the topic.
      With MQTT the sender and receiver aren’t aware of each other as there is no connection between the two. When a client receives a message it doesn’t have any idea of which client sent it unless the sender includes that information in the topic or message payload.

Leave a Reply to Hugo Burm Cancel reply

Your email address will not be published. Required fields are marked *