Simple Python MQTT Data Logger

python-mqtt-data-loggerMost MQTT brokers don’t provide any mechanism for logging historical data for later analysis.

However due to the publish/subscribe nature of MQTT is is easy to monitor, and log a data stream or streams using an MQTT client.



In fact this method may in many circumstances better than logging the data on the broker/server especially when the broker is being shared.

In this project we will create a simple data logger that logs data in JSON format to a text log file.

The project Consists of two main modules.

  • A logger class module mlogger.py
  • The logging script mqtt-data-logger.py

It also has helper module commands.py  which gets and parses command line arguments.

Note: There is also the Simple Python MQTT Topic Logger which logs data based on topic that may be more suitable for your project.

Logger Class

The class is implemented in a module called m_logger.py (message logger).

It can be used for logging data from any source not just MQTT.

To create an instance you need to supply three parameters:

  • The log directory- defaults to mlogs
  • Number of records to log per log- defaults to 1000
  • Number of logs. 0 for no limit.- defaults to 0
log=m_logger(log_dir="logs",log_recs=1000,number_logs=0):

The logger creates the log files in the directory using the current date and time for the directory names.

The format is month-day-hour-minute e.g.

m-logger-directory June 14th 10:50 am

You can log data either in plain text format or JSON format.

To log data either in plain text then use the

  • log_data(data) method. Data must be a string

To log data as JSON encoded data call the

  • json_log(data) method.

Data is a list or dictionary..

e.g.

log.log_data(data) 
or
log.log_json(data)

#The log file will contain the data as 
#plain text or  JSON encoded data strings
#each on a newline.

The logger will return True if successful and False if not.

To prevent loss of data in the case of computer failure the logs are continuously flushed to disk .

MQTT Data Logger Code Overview

This script will log data on a collections of topics and by default only if changed. It logs

  • Message time
  • Message topic
  • message

Data is logged as a json string. The following lines are added to the on_message callback or a function called by the on_message callback..


    data["time"]=tnow
    data["topic"]=topic
    data["message"]=msg
    #log.log_json(data) #log in JSON format

You can log the data direct as shown above, but I’ve commented it out.

Instead I prefer to use a separate thread to log the data and so I place the data in a queue.

The message handler function calls the has_changed function to check if the message status is different from the last message.

If it is the same then the message isn’t stored as there is not point storing the same message value multiple times.

The worker takes the data from the queue and logs it to disk.

The relevant code is shown below.

def on_message(client,userdata, msg):
    topic=msg.topic
    m_decode=str(msg.payload.decode("utf-8","ignore"))
    message_handler(client,m_decode,topic)
    #print("message received")
def message_handler(client,msg,topic):
    data=dict()
    tnow=time.localtime(time.time())
    m=time.asctime(tnow)+" "+topic+" "+msg
    data["time"]=tnow
    data["topic"]=topic
    data["message"]=msg
    if has_changed(topic,msg):
        print("storing changed data",topic, "   ",msg)
        q.put(data) #put messages on queue
def has_changed(topic,msg):
    topic2=topic.lower()
    if topic2.find("control")!=-1:
        return False
    if topic in last_message:
        if last_message[topic]==msg:
            return False
    last_message[topic]=msg
    return True
def log_worker():
    """runs in own thread to log data"""
    while Log_worker_flag:
        while not q.empty():
            results = q.get()
            if results is None:
                continue
            log.log_json(results)
            #print("message saved ",results["message"])
    log.close_file()

The worker is started at the beginning of the script.

t = threading.Thread(target=log_worker) #start logger
Log_worker_flag=True
t.start() #start logging thread

The Log_worker_flag is used to stop the worker when the script terminates.

Using the Data Logger

Note: By default the logger will only log changed data. This is for sensor data that is repeated.

For example a door sensor publishing it’s status every 10 seconds will send the same message repeatedly until someone opens or closes it.

You need to provide the script with:

  • List of topics to monitor
  • broker name and port
  • username and password if needed.

base log directory and number of logs have defaults.
The script is run from the command line. Type

python3 mqtt-data-logger.py -h

for a list of options.

Example Usage:

You will always need to specify the broker name or IP address and the topics to log

Note: You may not need to use the python prefix or may
need to use python3 mqtt_data_logger.py (Linux)

Specify broker and topics

python mqtt_data_logger.py -b 192.168.1.157 -t sensors/#

Specify broker and multiple topics

python mqtt_data_logger.py -b 192.168.1.157 -t sensors/# -t home/#

Log All Data:

python mqtt_data_logger.py b 192.168.1.157 -t sensors/# -s

Specify the client name used by the logger

python mqtt_data_logger.py b 192.168.1.157 -t sensors/# -n data-logger

Specify the log directory

python mqtt_data_logger.py b 192.168.1.157 -t sensors/# -l mylogs

Comments and Feedback

Was there enough detail in this tutorial for you to follow?

Please help me improve these tutorials by leaving your comments,rating them,asking questions.

Make a Contribution

pay-pal-buy-nowIf you find this script useful and would like to support its development then please consider making a small contribution by clicking here.


Download

If you are using the logger in your projects then let me know and I can feature them on this page.

Related Tutorials

Facebooktwittergoogle_plusredditpinterestlinkedinmail

25 comments

  1. Hi Steve i have followed all instructions but I’m receiving this error:

    File “c:\mqtt-data-logger-master\mqtt-data-logger.py”, line 75, in
    client1.username_pw_set(options[“username”], options[“password”])
    NameError: name ‘client1’ is not defined

    Any help would be great thanks.

    1. Hi
      There’s an error in the script change client1 to client and it should work ok. I’m rewriting the script and I’ll try to get it on the downloads page in the next few weeks.

        1. I’ll try an email it you next week. Is the problem you are getting because you try to use username/password.
          Can ou let me know the command line options you use.
          rgds
          steve

  2. Hi Steve,

    The logger seems to work for a while but eventually stops reporting new values and needs to be stopped and restarted again. Any thoughts on this?

    I’m invoking it as:
    python3 ./Downloads/pythLogger/mqtt-data-logger/mqtt-data-logger.py -b RPi5 -t /ESP14/#

    Thanks,

    Brian H.

  3. I surmise I need to move mlogger.py to somewhere known to python3:

    bjh@bjhProBook1:~/Downloads/pythLogger/mqtt-data-logger$ python3 ./mqtt-data-logger.py
    Traceback (most recent call last):
    File “./mqtt-data-logger.py”, line 14, in
    import classes.mlogger as mlogger
    ModuleNotFoundError: No module named ‘classes’

    Where do I need to move it to?

    1. You need to create a folder called classes in the folder that hold the data logger scrippt and put the mlogger.py file there. Alternatively edit the code to the location of the mlogger.py file.
      You might find this command at the top of the python script useful
      sys.path.append(“c:/python34/steve/”)
      it places the folder in the search path just adjust it to what you need.

  4. Hi Steve,

    What do you mean by creating a logger class in module m_logger.py? Do we have to create the module ourselves? If so how to go about it.

    Thanks,
    Dhaval

    1. The logger class is what does the data logging. The code is in the file m_logger.py.
      It is created by the script so you don’t need to create it yourself.
      rgds
      steve

  5. Hi Stave,
    I tried the mosquitto sever and the python data logger script in the windows it’s running ok. But I want to remember you that the multiline comments (” ” ” ) in the code of logger scripts give a fatal error! I fixed it by using (#) comment for each line.

    Some of the command lines above also in the github readme “mqtt_data_logger.py” written by the underscores but code files names with dashes you can unify all of them.

    Thanks

  6. I’ve already tried to run under all python version:
    python mqtt_data_logger.py -b 192.168.1.110 -t 4te/sonoff-TH/tele/SENSOR/#
    python2 mqtt_data_logger.py -b 192.168.1.110 -t 4te/sonoff-TH/tele/SENSOR/#
    python3 mqtt_data_logger.py -b 192.168.1.110 -t 4te/sonoff-TH/tele/SENSOR/#

    but the issue persist

  7. Thanks for your job Steve,
    I’ve tried using your data logger, but it seems that module “queue” is missing. Thanks for your attention.
    Raffaele

    pi@raspberrypi:~/Desktop/mqtt-data-logger-master $ python mqtt_data_logger.py -b 192.168.1.110 -t 4te/sonoff-TH/tele/SENSOR/#
    Traceback (most recent call last):
    File “mqtt_data_logger.py”, line 18, in
    from queue import Queue
    ImportError: No module named queue

    1. I just downloaded i and tried it and it seems to work ok. Can you send me a screenshot of the error message
      rgds
      steve

        1. Hi
          Looks like you need to install the queue module using pip. I thought it was already installed by default

  8. Thanks Steve.

    Brand new to MQTT, but the simplicity and clarity of your website got me started in a few minutes monitoring my first Mosquitto remote broker. Looking forward to testing the logger and SQL scripts next.

    Barry

Leave a Reply

Your email address will not be published. Required fields are marked *