Understanding and Configuring Logging – Mosquitto Broker Configuration

Mosquitto has two different log types

  • System status logs
  • Informational and debugging logs

Broker System Status logs

These are always logged to the $SYS topic and can be accessed using an MQTT client subscribed to the $SYS/# topic.

System status logs are always enabled by default but you can disable them or change the refresh time using the setting:

sys_interval seconds

Setting this to zero will disable publishing of the hierarchy. The default interval is 10 seconds.

Different brokers will provide different $SYS log topics. This Github page has a list and description of the $SYS log topics split into optional and required.

Information and Debugging logs

These provide detailed information about clients, and client broker commands.

Using them you can see which clients are connected, subscribed etc.

Logging is configured in the Logging section of the mosquitto.conf file.

You can choose which event types to log, by default error, information, notice and warning events are logged.

For debugging I use log_type all.

This is essentially the same as starting mosquitto with the verbose switch – mosquitto -v

Here is a screen shot showing a client connect, subscribe and disconnect sequence and the console log messages that are generated.

mosquitto-verbose-logging

Note: On Windows logging is disabled when starting mosquitto as a service. This is because the broker is running as a background process and has no console attached.

Logging Destinations

You can log messages to multiple destinations.

Options are:

  • topic – sent on topic tree
  • stdout
  • file
  • syslog

For example you could send log messages to the console, to the $SYS topic, and to a log file.

The screenshot below shows my mosquitto.conf file configured to log to three destinations. (console, logfile and topic)

mosquitto-logging

You can also disable logging using the log_dest none option.

Viewing Topic logs in a Client

Normally you will view these logs on the Mosquitto broker console but you can also view them using an MQTT client if you have logged them to a topic.

If you send the log messages to topic they are sent to the $SYS/broker/log topic , and you need to subscribe the topic tree $SYS/broker/log/#.

topic-logs

Two common question are:

  • Can I see all connected clients?
  • Can I list all topics?

The broker doesn’t let you do this directly but by enabling logging to a topic and monitoring the topic with an MQTT client you can get a good idea.

However using syslog and remote logging you can view connection details. See below.

Using Syslog

On Linux systems you can also use syslog to log the messages.

Messages are sent to the syslog file in the /var/log folder.

There are 2 settings

log_dest syslog
log_facility 5 #this is what I use

If you use syslog for logging it is possible to send all your logs to a central log server or to a log analyser.

Using the log_facility  allows you to only send logs from mosquitto.

Coming– Centralised mosquitto logging and analytics using node-red.

Historical Logs

On Linux if you log using to the file /var/log/mosquitto.log then the log file will automatically be rotated as an entry is made in the logrotate.d folder by the install script.

The file is shown below:

mosquitto-logrotate-file
If you look in the /var/log/mosquitto folder you will see the mosquitto.log file and zipped archives.

The default configuration archives the log file when it exceeds 100k.

Log rotation is handled by cron and runs once per day.

You can manually rotate the mosquitto logs by using the command

sudo logrotate -v /etc/logrotate.d/mosquitto

Viewing Mosquitto.log on Running Broker

When testing I start mosquitto manually using the verbose option to see the console.

However if you have a running server started automatically you don’t have access to the console.

You can still debug by using the tail command to view the log file with the -f option which will effectively show new entries as they are logged.

The command is

sudo tail -n20 -f /var/log/mosquitto/mosquitto.log

Viewing The $SYS logs

To view the system status logs you need to subscribe to the $SYS topic using a MQTT client.

The topic path is $SYS/#

You can use the mosquiito_sub client that comes with the broker as shown below.

In the screenshot below ws4 is the broker name.

mosquitto-sys-log

You can also use the Paho MQTT Python client see subscribing to topics.

If you use node-red I have created a flow that you can download that displays them in browser.

download

Log Timestamps

By default you will see the Unix times stamp format which isn’t really very useful when debugging from the console so you may want to make it more user friendly by adding the entry

log_timestamp_format %Y-%m-%dT%H:%M:%S

The logs should then appear as :

mos-logs-timestamp

Recommended Settings

Default log-types are :

error,warning, notice
and information

and they are sufficient.

On all systems I would log to a file.

/var/log/mosquitto/mosquitto.log

The install script automatically assigns permissions to the mosquitto user.

If you start mosquitto from the command line and try logging to the folder you will probably get an error as the user will not have permissions.

You can either change the permissions on the folder or alternatively change the log folder to a folder that you already have permissions to.

You could also start mosquitto as the mosquitto user using:

sudo -u mosquitto mosquitto -c mos2.conf

where mos2.conf is your test configuration file.

see run command as another user

Common Questions and Answers

Q- Are $SYS logs saved to disk?

A- No

Q– Why Do I get the message:  error: unable to open log file /var/log/mosquitto/mosquitto.log for writing.

A- This is a permission problem and is caused when you start mosquitto manually. You can either start mosquitto as the mosquitto user using sudo (shown above) or when testing change the log file location to your home directory.

Q- How can I list All topics?

A_ Mosquitto doesn’t provide this functionality by default but you can use topic logging as discussed above.

Q- How can I list All connected clients?

A- Use topic logging.

Q- Can I use the $SYS logs for monitoring the broker status?

A- Yes but not probably the best method and some brokers don’t have them enabled. If the broker doesn’t have any other methods built in (mosquitto doesn’t) then you would probably be better using an MQTT client to send background data and monitor that. See this article- why you shouldn’t use the $SYS topic for monitoring.

See also monitoring MQTT brokers .


coffeeIf you found this guide useful then perhaps you would like to Buy Me a Coffee

Mosquitto Configuration Tutorials

Other Related Articles and Resources:

Save

Save

Save

Save

Please rate? And use Comments to let me know more

50 comments

  1. Hello.
    I’m just starting to look into using MQTT, I can’t use Python and am afraid I’m getting too old to learn it. I’m using a Tasmota 4 relay as I thought it would be a simple learning tool. I’m on Win10 and Mosquitto, not sure which version but it is the latest.

    I’m confused by some of the comments about logging with a Windows system. Can you please clarify.
    You stated,
    “Note: On Windows logging is disabled when starting mosquitto as a service. This is because the broker is running as a background process and has no console attached.”
    But you have Standard Out and Log to File, among others, in you example. Is your example from a Windows computer or Linux? Is there a way to enable logging on a Windows computer? Is it that logging in Windows can only be achieved when not running as a Windows service? If that is the case how do you run Mosquitto not as a service? If I don’t have Mosquitto running as a service I can’t figure out how to start it to send mosquitto_pub messages.

    When running as a service, I can use these lines at the command prompt, and a variation in my script, to toggle the relay channels. I’m running on a LAN that is isolated from the internet.
    “cd “C:\Program Files\mosquitto\”
    mosquitto_pub -h 192.168.2.2 -p 1883 -t cmnd/tas4relay/POWER1 -m TOGGLE -r -V 5 ”
    And this line shows the status when using MQTT_Viewer_v0.4.1.
    “mosquitto_rr -t stat/tas4relay/POWER1 -e cmnd/tas4relay/POWER1 -n -v -d”
    However, I don’t know how to retrieve the status for my program, which is written with AutoHotKey. Since MQTT Viewer accesses the information when running Mosquitto as a service, do I need logging to get the info for my script? In this case it tells me if the relay is on or off.

    If I need to use logs, I may be able to use standard out, if Mosquitto will use it on Windows, but I know I could use a file. First I need to know if Mosquitto will do logging on Windows and, if it can, then I could use a suggestion about what would be the best method.

    I appreciate all the effort you put into these tutorials, I’ve learned quite a bit already. Even without basic Python knowledge I have learned to control the relays. That’s saying a lot. Thanks

    1. Richardd tks for the coffee.
      The screenshots are linux as that is what I use for mosquitto.
      However you can run the broker as a service or from a command line . I recommend that for testing you use the command line so you can view the console.
      For linux and windows you need a basic configuration file and it is just 2 lines and show at the top of this page
      http://www.steves-internet-guide.com/mosquitto-broker/
      also add a logging line so the file should look like this
      log_type all
      log_dest stdout # this you might need to remove if you get an error
      listener 1883
      allow_anonymous true

      but them in a file called test.conf and start using
      mosquitto -c test.conf

      If you don’t want to stop the broker running as a service then adjust the conf file to use a different port.
      You can have lots of brokers running on the same host as long as they use different listeners.
      Get back to me if you still have problems and I’ll do a video.
      I will try and set it up on my machine later today and post any new thoughts
      rgds
      steve

      1. Thanks for the quick reply. I worked on this most of the night and figured out most everything except getting info back from the relay. I got Mosquitto working without running it as a service, although I have a Mosquitto console open but nothing every appears in it. Can’t type in it and it doesn’t show any messages so I just run it hidden. I think the problem I was having is it takes a minute for mosquitto.exe to connect to everything before it will accept a command.

        I’ve got the log file working but I’m not sure it’s going to do me any good. It only shows the status of the relays when the log all configuration is set. There are other devices I use that were automatically found that I didn’t realize were set to use MQTT, and now it logs a ton of data, so it’s going to be difficult for me to find a way to pull what I need.

        Even then I’m not sure the log files are where I need to get the returned information from. MQTT Viewer was receiving the info before I got logging working. So maybe there’s something I’m missing about how to get the data from the subscribed topics. So this is the only question I have left at the moment.

        Current it’s just if relay channels are on or off but the main reason I’m wanting to learn MQTT is to control my 3d printer. I have all my machines on an isolated LAN and Bambu Lab doesn’t like that. As punishment for not sending everything through their cloud and spyware app they block most of the functionality of the machine. It was recommended to me to use the MQTT interface but I don’t know of anyone who has actually used it so I’m trying to figure it out for myself.

        While researching last night I did find a bunch of Python scripts for using the printer’s api, so now I’m wondering if I should be learning Python instead. I would think that since the script are already written the hard part is done, but I have no idea.

        The second reason I wanted to learn MQTT is the few Tasmota devices, like this relay, I have in my house. I thought the lag time would be improved using MQTT instead of the web interface but there is no difference or MQTT may be slower. Some devices respond immediate for every command, while some can occasionally take up to a minute and a half to respond. Since there is no difference in the response I don’t see any advantage to switch MQTT instead of staying on the HTTP interface.

        Ok, enough droning on. If you can tell me the best way to get the information from a subscribed topic, and maybe point me to a video or instruct page, I’ll work on that. If I need to ask more questions I’ll get organized so I can show you what I’ve tried.

        Thanks again.

        1. Richard
          When running mosquitto from the command line the console is read only. You cannot enter commands into it.
          If you have lots of other devices on the network then run mosquitto on a different port (1884). If you aren’t see the 3d printer send commands on the console then you should check the 3d printer as it either isn’t sending them or for some reason can’t see the broker.
          you can also use MQTT explorer and subscribe to the # topic but it doesn’t give you as much detail as the console.
          regarding the tasmota devices then speed wise there is not really any difference between http and MQTT but MQTT is simpler as you don’t need to know the IP address of the device.
          If you are happy to run python scripts I can send you some test scripts that should help you get familiar with MQTT.
          If you have a link to the 3d printer manual I will take a quick look at the MQTT configuration .
          Rgds
          Steve

          1. I use mosquitto_pub to send commands, (AutoHotKey script)
            “RunWait, mosquitto_pub -r -d -h 192.168.2.2 -p 1883 -t cmnd/tas4relay/POWER1 -m TOGGLE, , Hide UseErrorLevel”
            and mosquitto_rr to subscribe to the topics.
            “RunWait, mosquitto_rr -t stat/tas4relay/POWER1 -e cmnd/tas4relay/POWER1 -n -v -d `r”
            I am under the impression that mosquitto_rr is better for when you want to quickly get info from a subscribed topic, like from a program that only runs for short periods. For instance, start a program to see if a light is on or off. Is mosquitto_rr correct or should I be using mosquitto_sub?

            I don’t get any output to the mosquitto.exe window and not sure why or if it matters.
            “Run, mosquitto.exe -v -c mosquitto.conf, , Hide”.
            However, MQTT explorer was getting the info even before I got logging to work, and it used plain English saying something like “Relay 1 On”. I see in the log where I believe the info is being logged in code, but I wonder how MQTT explorer was receiving the data… which may be due to my next question.

            I was unsuccessful when trying to use log_type topic but, thought it wasn’t available on Windows. The line in the config file directly under logging says “If using syslog logging (not on Windows)”. Maybe I misunderstood that part? Do I need to subscribe “$SYS/#” using Windows? I’ll work some more on that tonight.

            I haven’t gotten to the 3d printer yet as I thought learning using the relay would be easier. I do believe the printer was one of the other devices mosquitto was seeing and logging. I’m sure I’ll have plenty of questions once I get there but I’m not in a hurry. However the API and MQTT information can be found here.
            https://mchrisgm.github.io/bambulabs_api/api/mqtt_client.html

            Thanks again

          2. Hi
            You need to use mosquitto_sub
            The command
            mosquitto.exe -v -c mosquitto.conf
            will not work unless you have edited the file as described previously as the default mosquitto.conf file is all commented out and so is effectively blank.
            Do not edit the file but create a new one for testing.
            I think the log_type topic will work will try it tomorrow.
            The subscribe to $SYS/broker/log/#
            Rgds
            Steve

          3. Thanks Steve
            Currently I’m going through all the logs types, destinations, $Sys, ect. that I can find to get a idea of what each does. I haven’t found where to get the on/off status yet but I’ll stumble onto it eventually.

            FYI, I don’t see log_type topic as an option, and when I try to use it mosquitto crashes at startup. There is a log_dest topic I have active and am learning what it does.

        2. Ok, disregard my last post. Just use the correct program and it magically works. And now that it’s working I better understand the relationships of the three programs. pub.exe, the broker tells the client to do something. sub.exe the broker tells the client to send any messages generated. rr.exe, the broker tells the client to update and send another message. Makes more sense when using things they way they were mean to be used.

          It’s now sending back the last command received instead of the actual power status (On or Off), but I think I’m to the point I can figure more of it out on my own.

          However mosquitto.exe still crashes when I have log_type topic set. I wonder if that’s what I need to get the actual power status.

          1. So I got the Relay control working last night. All I ended up using was log_dest syslog and log_type information.

            When the script runs it opens four sub hidden windows, one for each channel of the relay, and then I can send a publish command for a channel and the sub window for that channel writes either on or off. The commands can either be On, Off, or Toggle, or a status command, but they all return the same on or off response. This does work but would there be a better way than opening four hidden windows?

            I finally figured out how to send the output of the hidden sub windows to a file that my script can read. Since it’s reading and writing to a file it’s still a little clunky, but will do until I learn more.

            One reason I am just writing to a file is there seems to be a problem with adding log_dest stdout. When I un-comment it in the config file Mostquitto won’t start. With in three or four seconds of trying to start it up it crashes. log_dest stderr works fine so I’m confused… again. I did install Mosquitto as a Windows 10 sevice but I disabled it in the Services window, so I don’t think that’s where the problem is. In my research I did see others ways to get the info to a standard out but haven’t et looked very far into them yet.

            I’m considering running Mostquitto on a Pi, thinking it may be a little easier that way.

            If you have any thoughts as to what the stdout problem could be please let me know.
            Thanks again for all our help.

          2. Richard
            will try it this weekend and let you know. Pi is a good option I use it a lot.
            Rgds
            Steve

          3. Richard
            used this conf file
            log_type all
            connection_messages true
            log_dest file C:\mos\mos.log
            log_timestamp true
            log_timestamp_format %Y-%m-%dT%H:%M:%S
            log_dest topic
            log_dest stdout
            listener 1883
            protocol mqtt
            allow_anonymous true

            and it started and worked ok. Without the line
            log_dest stdout
            noting is displayed on the console.
            Rgds
            Steve

  2. Hi Steve,
    Thanks for your article, it helps me so much.

    I have run the mosquitto on version 2.0.15 and on Linux,
    and I have some questions.
    This is my setting about the log file(mosquitto.conf) :
    ———————————————————————————–
    log_dest file ./mosquitto.log
    log_dest topic
    log_type all
    connection_messages true
    log_timestamp true
    log_timestamp_format %Y-%m-%dT%H:%M:%S
    ———————————————————————————–
    I run the mosquitto by using the command:
    ./src/mosquitto -c mosquitto.conf -v

    1. The log message printed on the log file successfully, but it wasn’t shown on the terminal screen.
    I would like to know if there is a way to print log messages on the log file and on the terminal.

    2. I am learning the way you write in this article to make a mosquiito_sub client subscribe to the topic of the log message, but it didn’t work. Except for the setting: log_dest topic, is there anything I need to set to run this function?
    If I ran the command: ./client/mosquto_sub -t $SYS/#,
    the subscriber got nothing.
    And if I ran the command: ./client/mosquto_sub -t \$SYS/#, this subscriber got some message of number which like:
    mosquitto version 2.0.15
    22 seconds
    0
    0
    0
    0
    0
    0
    1
    7.00
    1.71
    I can’t read these numbers,
    I wondered if I did anything wrong?

    Thank you so much for reading my message, wish you have a nice day.

  3. Hello, which is the timestamp that mosquitto uses for the logging? Can we configure the timestamp that it’s logged?
    Thank you in advance!

    1. Yes
      Folloing is taken from the docs here
      https://mosquitto.org/man/mosquitto-conf-5.html

      log_timestamp_format format

      Set the format of the log timestamp. If left unset, this is the number of seconds since the Unix epoch. This option is a free text string which will be passed to the strftime function as the format specifier. To get an ISO 8601 datetime, for example:

      log_timestamp_format %Y-%m-%dT%H:%M:%S

  4. When running mosquitto in Ubuntu from the command line, I can use the -V verbose option to display the output in real time. However when running it as a service and using “journalctl -f -u” to see the output in real time, it does not work. Setting “log_dest stdout” does not work because I believe the mosquitto service unit file redirects it to a log file.
    Is there ANY way to view the mosquitto output in real time when running it as a service?
    Thanks!

  5. Hi Steve, can I use mosquitto logging to print received messages too? I have some packet loss and I’m trying to understand where it’s coming from, if from the broker, the wireless connection or the publishers. I have only used logs to monitor clients and topics so I’m not 100% sure it’s possible.

    1. The logs will show you the published and received messages if you use verbose logging. However depending on the clients connected it can be very difficult to troubleshoot.
      rgds
      Steve

  6. Hi Steve
    Thanks for the useful information.
    I am working with Windows 10, I configured logging to a file: “log_dest file c:/mqtt/logs/mosquitto.log”
    Now I have the problem that I can not open the mosquitto.log file. I get the error that I don’t have permission to do so. What can I do to be able to view the log file?

  7. Permissions on the log file are set to restrictive (ie user (mosquitto) only). I would prefer this to be accessible for read to members of the mosquitto group (me 😉 ) too. Is it possible to configure this?

    This is a Linux setup – I have it logging to /var/log/mosquitto/mosquitto.log.
    Thanks

    1. Paul
      Tried to change but didn’t succeed.Will keep trying if you do find out before me then let me know.
      Rgds
      Steve

  8. Hi steve, I am trying to get a the log file but I do not know where it is located, this is the configuration that I did:
    config file:
    ——————————————————————————————————————————–
    # Note that if the broker is running as a Windows service it will default to
    # “log_dest none” and neither stdout nor stderr logging is available.
    # Use “log_dest none” if you wish to disable logging.
    #log_dest none
    log_dest topic
    log_dest syslog
    log_dest file C:\Archivos”de”programa\mosquitto\mosquitto.log
    ————————————————————————————————————————–
    am I doing something wrong?

    1. log dest syslog doesn’t work on windows. Not sure about the quotes in the pathname other than that it seems ok.
      I would start it from the command line and log to the local folder for testing
      Rgds
      Steve

  9. Hi there,
    I have a strange problem in my home

    Sometime arrive an open/close topic command for the chicken coop door in the middle of the night.
    There is 3 days that I found it opened in the morning.

    What I ask is how can I understand “who” send this message topic.
    So is there possibility to have for example a log with the IP of client topic please ?
    In this case if I know the IP I can understand from where the message arrive.

    Thanks a lot
    Denis

    1. If you look at the logs on the broker it will tell your the name of the client that sent it. You might also see the client connect with the IP address but the connection might have been a while ago as it might have been left open.

    1. Hi
      Not as far as I know you would have to use an external program to read the logs and put them in a database.
      rgds
      steve

  10. I don’t know if I am kidding myself and not actually seeing the mosquitto log but….
    If I start a terminal window and type:
    tail -f /var/log/mosquitto/mosquitto.log | perl -pe ‘s/(\d+)/localtime($1)/e’
    I will get a scrolling window that displays the last few log entries and will continue to add new lines as they are added to the log.
    Further more, the time stamp is in human readable time.
    Can this be what I think it is?

  11. I’m having issues connecting my ESP8266 to a local Mosquitto broker. It works if I connect my ESP8266 to my external IP but does not work internal IP.

    Is there anyway to log connections to see if the broker is seeing my ESP8266 trying to connect and rejecting/accepting the connection?

    1. Use log_type all.
      But you may find it better to stop mosquitto and start it manually using
      mosquitto -v
      this will show the connection attempts on the console
      Rgds
      Steve

  12. Thanks Steve,
    The mosquito logging explanation really helped. With this and other info I was able to get mqtt messages running (with a Raspberry Pi acting as the server) to and from a PC with a client running in Processing. Also a client running in Processing on the Rpi. Now moving on to interfacing multiple Arduinos to all this and probably more Rpi’s also . Mosquitto has run flawlessly – I just needed logging to see how to make the initial connections.
    Thank you again – very helpful……

  13. Hi Steve,
    Do you happen to know if more admin-level functions are logged as well, such as changing files the broker would use for ACLs or usernames and password changes through mosquitto_passwd?

    1. Patrick
      No there aren’t as far as I know but it could change as I know that it is being upgraded for mqtt v5 and that might contain additional features.

    1. Hi
      need to do more research on that but if you add the following to the conf file you will see lots of info on websockets
      log_type websockets
      websockets_log_level 9

      I need to check the log levels but level 6 seems to give basic info
      rgds
      steve

Leave a Reply

Your email address will not be published. Required fields are marked *