Logging MQTT Sensor Data Using Python

python-mqtt-data-logger

In this project we will create a simple data logger that logs IOT sensor/device data in JSON or csv format to a text log file.

The project Consists of two main modules.

  • A logger class module mlogger.py
  • The logging script mqtt-data-logger.py

It also has helper module commands.py  which gets and parses command line arguments.

Note: There is also the Simple Python MQTT Topic Logger which logs data based on topic that may be more suitable for your project.

Logger Class

The class is implemented in a module called m_logger.py (message logger).

It can be used for logging data from any source not just MQTT.

To create an instance you need to supply three parameters:

  • The log directory- defaults to mlogs
  • Number of records to log per log- defaults to 1000
  • Number of logs. 0 for no limit.- defaults to 0
log=m_logger(log_dir="logs",log_recs=1000,number_logs=0):

The logger creates the log files in the directory using the current date and time for the file names.

The format is month-day-hour-minute e.g.

m-logger-directory June 14th 10:50 am

You can log data either in plain text format or JSON format.

To log data either in plain text then use the

  • log_data(data) method. Data must be a string

To log data as JSON encoded data call the

  • json_log(data) method.

Data is a list or dictionary..

e.g.

log.log_data(data) 
or
log.log_json(data)

#The log file will contain the data as 
#plain text or  JSON encoded data strings
#each on a newline.

The logger will return True if successful and False if not.

To prevent loss of data in the case of computer failure the logs are continuously flushed to disk .

MQTT Data Logger Code Overview

This script will log data on a collection of topics and by default only if changed. It logs

  • Message time
  • Message topic
  • message

Data is logged as a JSON string. The following lines are added to the on_message callback or a function called by the on_message callback..


    data["time"]=tnow
    data["topic"]=topic
    data["message"]=msg
    #log.log_json(data) #log in JSON format

You can log the data direct as shown above, but I’ve commented it out.

Instead I prefer to use a separate thread to log the data and so I place the data in a queue.

The message handler function calls the has_changed function to check if the message status is different from the last message.

If it is the same then the message isn’t stored as there is not point storing the same message value multiple times.

The worker takes the data from the queue and logs it to disk.

The relevant code is shown below.

def on_message(client,userdata, msg):
    topic=msg.topic
    m_decode=str(msg.payload.decode("utf-8","ignore"))
    message_handler(client,m_decode,topic)
    #print("message received")
def message_handler(client,msg,topic):
    data=dict()
    tnow=time.localtime(time.time())
    m=time.asctime(tnow)+" "+topic+" "+msg
    data["time"]=tnow
    data["topic"]=topic
    data["message"]=msg
    if has_changed(topic,msg):
        print("storing changed data",topic, "   ",msg)
        q.put(data) #put messages on queue
def has_changed(topic,msg):
    topic2=topic.lower()
    if topic2.find("control")!=-1:
        return False
    if topic in last_message:
        if last_message[topic]==msg:
            return False
    last_message[topic]=msg
    return True
def log_worker():
    """runs in own thread to log data"""
    while Log_worker_flag:
        while not q.empty():
            results = q.get()
            if results is None:
                continue
            log.log_json(results)
            #print("message saved ",results["message"])
    log.close_file()

The worker is started at the beginning of the script.

t = threading.Thread(target=log_worker) #start logger
Log_worker_flag=True
t.start() #start logging thread

The Log_worker_flag is used to stop the worker when the script terminates.

Using the Data Logger

Note: By default the logger will only log changed data. This is for sensor data that is repeated.

Also the changed data routine only examines the complete message and will not work correctly if the message contains a time stamp or message counter.

So ideally for this to work then the data should be a simple state value like on/off or a simple reading like 20.1 and not a complex JSON value.

If the message is JSON data then the change routine may not work but the data will still be logged.

For example a door sensor publishing it’s status every 10 seconds will send the same message repeatedly until someone opens or closes it.

You need to provide the script with:

  • List of topics to monitor
  • broker name and port
  • user name and password if needed.

The base log directory and number of logs have defaults.
The script is run from the command line. Type

python3 mqtt-data-logger.py -h

for a list of options.

Example Usage:

You will always need to specify the broker name or IP address and the topics to log

Note: You may not need to use the python prefix or may
need to use python3 mqtt_data_logger.py (Linux)

Specify broker and topics

python mqtt_data_logger.py -b 192.168.1.157 -t sensors/#

Log as JSON

python mqtt-data-logger.py -b 192.168.1.157 -t sensors/# -j

Log as CSV

python mqtt-data-logger.py -b 192.168.1.157 -t sensors/# -c

Specify broker and multiple topics

python mqtt-data-logger.py -b 192.168.1.157 -t sensors/# -t home/#

Log All Data:

python mqtt-data-logger.py b 192.168.1.157 -t sensors/# -s

Specify the client name used by the logger

python mqtt-data-logger.py b 192.168.1.157 -t sensors/# -n data-logger

Specify the log directory

python mqtt-data-logger.py b 192.168.1.157 -t sensors/# -l mylogs

Note 29/2/2024

Updated to include logging as a csv file see readme in the zip file

 

Comments and Feedback

Was there enough detail in this tutorial for you to follow?

Please help me improve these tutorials by leaving your comments,rating them,asking questions.

 


Download

download

If you are using the logger in your projects then let me know and I can feature them on this page.


coffeeIf you found this guide useful then perhaps you would like to Buy Me a Coffee

Related Tutorials

Please rate? And use Comments to let me know more

41 comments

  1. Hi Steve! I noticed that in your code mqtt-topic-logger.py you set data[“time_ms”]=int(tnow*1000)
    but in the log file there are only “time”, “topic” and “message”. “time_ms” is missing.
    I would interested to get messages from sensors every ms. Do you know how can I get messages every ms?

    1. even though it is labled time it is time in ms. If you are using python3.7 or above use
      unix_time_ms_1 = int(time.time_ns() / 1000000)
      for a more accurate value.
      Does it make sense?
      Rgds
      Steve

  2. Hi Steve,

    Will this script work for mosquito broker ???
    What changes are required if I want to do logging on same pi on which broker is running ???

    1. Hi
      It isn’t a broker logger in that it doesn’t log all messages passing through the broker.It is a client script that logs selected messages. I can be run on the same pi as the broker.
      Rgds
      Steve

  3. Hi I have used this.But I am not getting how it is publishing the data to my cloud.If you could throw some light on this.

  4. Hi! I’ve thi issue

    Traceback (most recent call last):
    File “mqtt-data-logger.py”, line 174, in
    client.username_pw_set(options[“username”], options[“password”])
    NameError: name ‘client’ is not defined
    closing log file

    1. Hi
      There was an error with passwords on the script but I thought I fixed it several months ago. If you don’t have a recent download then download it again from the site.
      Did you get it from the site or from github?
      Can you let me know so I can correct it.
      Rgds
      Steve

  5. Hi Steve and thanks for sharing!

    I’ve got it working on my Raspberry and PC with Win10. What concerns me is that it drains 25% of processor on both machines. Did you notice that on yours or is it only mine issue?

    Thanks,
    Mark

      1. I’m running it for educational purposes, so I publish single message once on a minute. It drains 25% even if it is idle (not processing any messages). Strange behavior :/ Well, if you didn’t encounter such issue, then I will write my own version 😉 My goal is to log messages into MySQL database.

        On Raspberry is Python 2.7 and PC is Python 3.7

        Thanks,
        Mark

        1. Mark
          In tests I sent 100s of messages a second to the logger and didn’t notice any slowness. I’m due to revisit the code shortly and will look specifically at the processor usage.
          Rgds
          Steve

  6. Hi Steve i have followed all instructions but I’m receiving this error:

    File “c:\mqtt-data-logger-master\mqtt-data-logger.py”, line 75, in
    client1.username_pw_set(options[“username”], options[“password”])
    NameError: name ‘client1’ is not defined

    Any help would be great thanks.

    1. Hi
      There’s an error in the script change client1 to client and it should work ok. I’m rewriting the script and I’ll try to get it on the downloads page in the next few weeks.

        1. I’ll try an email it you next week. Is the problem you are getting because you try to use username/password.
          Can ou let me know the command line options you use.
          rgds
          steve

  7. Hi Steve,

    The logger seems to work for a while but eventually stops reporting new values and needs to be stopped and restarted again. Any thoughts on this?

    I’m invoking it as:
    python3 ./Downloads/pythLogger/mqtt-data-logger/mqtt-data-logger.py -b RPi5 -t /ESP14/#

    Thanks,

    Brian H.

  8. I surmise I need to move mlogger.py to somewhere known to python3:

    bjh@bjhProBook1:~/Downloads/pythLogger/mqtt-data-logger$ python3 ./mqtt-data-logger.py
    Traceback (most recent call last):
    File “./mqtt-data-logger.py”, line 14, in
    import classes.mlogger as mlogger
    ModuleNotFoundError: No module named ‘classes’

    Where do I need to move it to?

    1. You need to create a folder called classes in the folder that hold the data logger scrippt and put the mlogger.py file there. Alternatively edit the code to the location of the mlogger.py file.
      You might find this command at the top of the python script useful
      sys.path.append(“c:/python34/steve/”)
      it places the folder in the search path just adjust it to what you need.

        1. This was my work around:

          from importlib.machinery import SourceFileLoader

          mlogger = SourceFileLoader(“mlogger”, “/home/pi/serial/classes/mlogger.py”).load_module()
          command = SourceFileLoader(“command”, “/home/pi/serial/classes/command.py”).load_module()

      1. Shouldn’t the script do a “$(mkdir -p …)” type command to ensure it has the directory structure it needs. Same with $(which {module}) commands to ensure it has all the right python modules??

        1. The script currently uses the local folder unless you explicitly set it. You can modify the script any way you want as it is open source and is meant as demo code. I use it often for recording broker data so I can replay it later.
          Rgds
          Steve

  9. Hi Steve,

    What do you mean by creating a logger class in module m_logger.py? Do we have to create the module ourselves? If so how to go about it.

    Thanks,
    Dhaval

    1. The logger class is what does the data logging. The code is in the file m_logger.py.
      It is created by the script so you don’t need to create it yourself.
      rgds
      steve

  10. Hi Stave,
    I tried the mosquitto sever and the python data logger script in the windows it’s running ok. But I want to remember you that the multiline comments (” ” ” ) in the code of logger scripts give a fatal error! I fixed it by using (#) comment for each line.

    Some of the command lines above also in the github readme “mqtt_data_logger.py” written by the underscores but code files names with dashes you can unify all of them.

    Thanks

  11. I’ve already tried to run under all python version:
    python mqtt_data_logger.py -b 192.168.1.110 -t 4te/sonoff-TH/tele/SENSOR/#
    python2 mqtt_data_logger.py -b 192.168.1.110 -t 4te/sonoff-TH/tele/SENSOR/#
    python3 mqtt_data_logger.py -b 192.168.1.110 -t 4te/sonoff-TH/tele/SENSOR/#

    but the issue persist

  12. Thanks for your job Steve,
    I’ve tried using your data logger, but it seems that module “queue” is missing. Thanks for your attention.
    Raffaele

    pi@raspberrypi:~/Desktop/mqtt-data-logger-master $ python mqtt_data_logger.py -b 192.168.1.110 -t 4te/sonoff-TH/tele/SENSOR/#
    Traceback (most recent call last):
    File “mqtt_data_logger.py”, line 18, in
    from queue import Queue
    ImportError: No module named queue

    1. I just downloaded i and tried it and it seems to work ok. Can you send me a screenshot of the error message
      rgds
      steve

        1. Hi
          Looks like you need to install the queue module using pip. I thought it was already installed by default

  13. Thanks Steve.

    Brand new to MQTT, but the simplicity and clarity of your website got me started in a few minutes monitoring my first Mosquitto remote broker. Looking forward to testing the logger and SQL scripts next.

    Barry

Leave a Reply

Your email address will not be published. Required fields are marked *