In this project we will create a simple data logger that logs IOT sensor/device data in JSON format to a text log file.
The project Consists of two main modules.
- A logger class module mlogger.py
- The logging script mqtt-data-logger.py
It also has helper module commands.py which gets and parses command line arguments.
Note: There is also the Simple Python MQTT Topic Logger which logs data based on topic that may be more suitable for your project.
The class is implemented in a module called m_logger.py (message logger).
It can be used for logging data from any source not just MQTT.
To create an instance you need to supply three parameters:
- The log directory- defaults to mlogs
- Number of records to log per log- defaults to 1000
- Number of logs. 0 for no limit.- defaults to 0
The logger creates the log files in the directory using the current date and time for the directory names.
The format is month-day-hour-minute e.g.
June 14th 10:50 am
You can log data either in plain text format or JSON format.
To log data either in plain text then use the
- log_data(data) method. Data must be a string
To log data as JSON encoded data call the
- json_log(data) method.
Data is a list or dictionary..
log.log_data(data) or log.log_json(data) #The log file will contain the data as #plain text or JSON encoded data strings #each on a newline.
The logger will return True if successful and False if not.
To prevent loss of data in the case of computer failure the logs are continuously flushed to disk .
MQTT Data Logger Code Overview
This script will log data on a collections of topics and by default only if changed. It logs
- Message time
- Message topic
Data is logged as a JSON string. The following lines are added to the on_message callback or a function called by the on_message callback..
data["time"]=tnow data["topic"]=topic data["message"]=msg #log.log_json(data) #log in JSON format
You can log the data direct as shown above, but I’ve commented it out.
Instead I prefer to use a separate thread to log the data and so I place the data in a queue.
The message handler function calls the has_changed function to check if the message status is different from the last message.
If it is the same then the message isn’t stored as there is not point storing the same message value multiple times.
The worker takes the data from the queue and logs it to disk.
The relevant code is shown below.
def on_message(client,userdata, msg): topic=msg.topic m_decode=str(msg.payload.decode("utf-8","ignore")) message_handler(client,m_decode,topic) #print("message received") def message_handler(client,msg,topic): data=dict() tnow=time.localtime(time.time()) m=time.asctime(tnow)+" "+topic+" "+msg data["time"]=tnow data["topic"]=topic data["message"]=msg if has_changed(topic,msg): print("storing changed data",topic, " ",msg) q.put(data) #put messages on queue def has_changed(topic,msg): topic2=topic.lower() if topic2.find("control")!=-1: return False if topic in last_message: if last_message[topic]==msg: return False last_message[topic]=msg return True def log_worker(): """runs in own thread to log data""" while Log_worker_flag: while not q.empty(): results = q.get() if results is None: continue log.log_json(results) #print("message saved ",results["message"]) log.close_file()
The worker is started at the beginning of the script.
t = threading.Thread(target=log_worker) #start logger Log_worker_flag=True t.start() #start logging thread
The Log_worker_flag is used to stop the worker when the script terminates.
Using the Data Logger
Note: By default the logger will only log changed data. This is for sensor data that is repeated.
Also the changed data routine only examines the complete message and will not work correctly if the message contains a time stamp or message counter.
So ideally for the to work then the data should be a simple state value like on/off or a simple reading like 20.1 and not a complex JSON value.
If the message is JSON data then the change routine may not work but the data will still be logged.
For example a door sensor publishing it’s status every 10 seconds will send the same message repeatedly until someone opens or closes it.
You need to provide the script with:
- List of topics to monitor
- broker name and port
- user name and password if needed.
base log directory and number of logs have defaults.
The script is run from the command line. Type
python3 mqtt-data-logger.py -h
for a list of options.
You will always need to specify the broker name or IP address and the topics to log
Note: You may not need to use the python prefix or may
need to use python3 mqtt_data_logger.py (Linux)
Specify broker and topics
python mqtt_data_logger.py -b 192.168.1.157 -t sensors/#
Specify broker and multiple topics
python mqtt_data_logger.py -b 192.168.1.157 -t sensors/# -t home/#
Log All Data:
python mqtt_data_logger.py b 192.168.1.157 -t sensors/# -s
Specify the client name used by the logger
python mqtt_data_logger.py b 192.168.1.157 -t sensors/# -n data-logger
Specify the log directory
python mqtt_data_logger.py b 192.168.1.157 -t sensors/# -l mylogs
Comments and Feedback
Was there enough detail in this tutorial for you to follow?
Please help me improve these tutorials by leaving your comments,rating them,asking questions.
If you are using the logger in your projects then let me know and I can feature them on this page.
- Video Encoding/decoding JSON data in Python
- Logging MQTT sensor data
- My Python Working Notes
- Simple Controllable MQTT Sensor Simulator in Python
- Send a File Using MQTT -MQTT Projects
- Encrypting MQTT Payloads with Python – Example Code
- Store MQTT data in database