Logging MQTT Sensor Data Using Python

In this project we will create a simple data logger that logs IOT sensor/device data in JSON format to a text log file.

The project Consists of two main modules.

• A logger class module mlogger.py
• The logging script mqtt-data-logger.py

It also has helper module commands.py  which gets and parses command line arguments.

Note: There is also the Simple Python MQTT Topic Logger which logs data based on topic that may be more suitable for your project.

Logger Class

The class is implemented in a module called m_logger.py (message logger).

It can be used for logging data from any source not just MQTT.

To create an instance you need to supply three parameters:

• The log directory- defaults to mlogs
• Number of records to log per log- defaults to 1000
• Number of logs. 0 for no limit.- defaults to 0

log=m_logger(log_dir="logs",log_recs=1000,number_logs=0):

The logger creates the log files in the directory using the current date and time for the directory names.

The format is month-day-hour-minute e.g.

June 14th 10:50 am

You can log data either in plain text format or JSON format.

To log data either in plain text then use the

• log_data(data) method. Data must be a string

To log data as JSON encoded data call the

• json_log(data) method.

Data is a list or dictionary..

e.g.

log.log_data(data) 
or
log.log_json(data)

#The log file will contain the data as 
#plain text or  JSON encoded data strings
#each on a newline.

The logger will return True if successful and False if not.

To prevent loss of data in the case of computer failure the logs are continuously flushed to disk .

MQTT Data Logger Code Overview

This script will log data on a collections of topics and by default only if changed. It logs

• Message time
• Message topic
• message

Data is logged as a JSON string. The following lines are added to the on_message callback or a function called by the on_message callback..

    data["time"]=tnow
    data["topic"]=topic
    data["message"]=msg
    #log.log_json(data) #log in JSON format

You can log the data direct as shown above, but I’ve commented it out.

Instead I prefer to use a separate thread to log the data and so I place the data in a queue.

The message handler function calls the has_changed function to check if the message status is different from the last message.

If it is the same then the message isn’t stored as there is not point storing the same message value multiple times.

The worker takes the data from the queue and logs it to disk.

The relevant code is shown below.

def on_message(client,userdata, msg):
    topic=msg.topic
    m_decode=str(msg.payload.decode("utf-8","ignore"))
    message_handler(client,m_decode,topic)
    #print("message received")
def message_handler(client,msg,topic):
    data=dict()
    tnow=time.localtime(time.time())
    m=time.asctime(tnow)+" "+topic+" "+msg
    data["time"]=tnow
    data["topic"]=topic
    data["message"]=msg
    if has_changed(topic,msg):
        print("storing changed data",topic, "   ",msg)
        q.put(data) #put messages on queue
def has_changed(topic,msg):
    topic2=topic.lower()
    if topic2.find("control")!=-1:
        return False
    if topic in last_message:
        if last_message[topic]==msg:
            return False
    last_message[topic]=msg
    return True
def log_worker():
    """runs in own thread to log data"""
    while Log_worker_flag:
        while not q.empty():
            results = q.get()
            if results is None:
                continue
            log.log_json(results)
            #print("message saved ",results["message"])
    log.close_file()

The worker is started at the beginning of the script.

t = threading.Thread(target=log_worker) #start logger
Log_worker_flag=True
t.start() #start logging thread

The Log_worker_flag is used to stop the worker when the script terminates.

Using the Data Logger

Note: By default the logger will only log changed data. This is for sensor data that is repeated.

Also the changed data routine only examines the complete message and will not work correctly if the message contains a time stamp or message counter.

So ideally for the to work then the data should be a simple state value like on/off or a simple reading like 20.1 and not a complex JSON value.

If the message is JSON data then the change routine may not work but the data will still be logged.

For example a door sensor publishing it’s status every 10 seconds will send the same message repeatedly until someone opens or closes it.

You need to provide the script with:

• List of topics to monitor
• broker name and port
• user name and password if needed.

base log directory and number of logs have defaults.
The script is run from the command line. Type

python3 mqtt-data-logger.py -h

for a list of options.

Example Usage:

You will always need to specify the broker name or IP address and the topics to log

Note: You may not need to use the python prefix or may
need to use python3 mqtt_data_logger.py (Linux)

Specify broker and topics

python mqtt_data_logger.py -b 192.168.1.157 -t sensors/#

Log as JSON

python mqtt-data-logger.py -b 192.168.1.157 -t sensors/# -j

Log as CSV

python mqtt-data-logger.py -b 192.168.1.157 -t sensors/# -c

Specify broker and multiple topics

python mqtt-data-logger.py -b 192.168.1.157 -t sensors/# -t home/#

Log All Data:

python mqtt-data-logger.py b 192.168.1.157 -t sensors/# -s

Specify the client name used by the logger

python mqtt-data-logger.py b 192.168.1.157 -t sensors/# -n data-logger

Specify the log directory

python mqtt-data-logger.py b 192.168.1.157 -t sensors/# -l mylogs

Note 29/2/2024

Updated to include logging as a csv file see readme in the zip file

 

Comments and Feedback

Was there enough detail in this tutorial for you to follow?

Please help me improve these tutorials by leaving your comments,rating them,asking questions.

 

---

Download

If you are using the logger in your projects then let me know and I can feature them on this page.

Related Tutorials

• Video Encoding/decoding JSON data in Python
• Logging MQTT sensor data
• My Python Working Notes
• Simple Controllable MQTT Sensor Simulator in Python
• Send a File Using MQTT -MQTT Projects
• Encrypting MQTT Payloads with Python – Example Code
• Store MQTT data in database
• Converting JSON to CSV with Python