The MQTT sensor data logger logs data on all monitored topics to the same log files.
If the data need to be split into topics then it will need to be done by the log analyser software.
However the topics logger logs sensor/device data based on topic so that each topic has it’s own log file as shown in the diagram below:
Note: Node-red version here
Topic Logger Class
This class is similar to the mlogger class but automatically creates the folder hierarchy to match the topic hierarchy.
Data is stored in log files starting with log000.txt. When the log reaches the maximum size which is set at 5MB by default then the log file is rotated.
MQTT Data Logger Code Overview
The topic logger uses the tlogger class.
This script will log data on a collections of topics and by default only if changed. It logs
- Message time
- Message topic
Data is logged as a json string. The following lines are added to the on_message callback or a function called by the on_message callback..
data["time"]=tnow data["topic"]=topic data["message"]=msg #log.log_json(data) #log in JSON format
You can log the data direct as shown above, but I’ve commented it out.
Instead I prefer to use a separate thread to log the data and so I place the data in a queue using the Queue class.
The message handler function calls the has_changed function to check if the message status is different from the last message.
If it is the same then the message isn’t stored (default mode) as there is no point storing the same message value multiple times.
This can be changed using the -s command line switch.
In addition the message handler function checks if the msg is already JSON encoded.
The worker takes the data from the queue and logs it to disk using the log_json method, which converts the data to a JSON encoded string before storing.
The relevant code is shown below.
The worker is started at the beginning of the script.
t = threading.Thread(target=log_worker) #start logger Log_worker_flag=True t.start() #start logging thread
The Log_worker_flag is used to stop the worker when the script terminates.
Logging as csv
The logger will now log data in csv format instead of JSON.
The csv file will contain a header file which the logger creates from the incoming data.
In addition to ensure the header file is in a predetermined order an optional header file can be used as a template.
Full details are in the read_me file in the download package.
Using the Topic Logger
Note: By default the logger will only log changed data.
For example a door sensor publishing it’s status every 10 seconds will send the same message repeatedly until someone opens or closes it.
You need to provide the script with:
- List of topics to monitor
- broker name and port
- username and password if needed.
The script is run from the command line. Type
python3 mqtt-data-logger.py -h #Linux or C:\python34\python.exe mqtt-data-logger.py -h #Windows
for a list of options.
You will always need to specify the broker name or IP address and the topics to log
Note: You may not need to use the python prefix or may need to use python3 mqtt_topic_logger.py (Linux)
Specify broker and topics
python mqtt_data_logger.py -b 192.168.1.157 -t sensors/#
Specify broker and multiple topics
python mqtt_topic_logger.py -b 192.168.1.157 -t sensors/# -t home/#
Log All Data:
python mqtt_topic_logger.py b 192.168.1.157 -t sensors/# -s
Log as csv:
python mqtt_topic_logger.py b 192.168.1.157 -t sensors/# -c
Specify the client name used by the logger
python mqtt_topic_logger.py b 192.168.1.157 -t sensors/# -n data-logger
Specify the log directory
python mqtt_topic_logger.py b 192.168.1.157 -t sensors/# -l mylogs
Comments and Feedback
Was there enough detail in this tutorial for you to follow?
Please help me improve these tutorials by leaving your comments,rating them,asking questions.
Make a Contribution
If you find this script useful and would like to support its development then please consider making a small contribution by clicking here.
If you are using the logger in your projects then let me know and I can feature them on this page.
I have tested this logger on a raspberry pi as the data logger with a message stream of 40,000 messages each message approximately 60bytes in length.
When logging changes I achieved a message rate of 76 messages/second.
When logging all messages I achieved a message rate of 50 messages/second.
Note: The topic logger has been containerised in Docker see here
Related Tutorials and Resources
- MQTT Data Logger
- Video Encoding/decoding JSON data in Python
- MQTT Data Logger -Node-Red
- Logging MQTT Sensor Data to SQL DataBase With Python
- Node-Red – Storing IOT Data in a SQL Database
- My Python Working Notes
- Simple Controllable MQTT Sensor Simulator in Python