In this project we will create a simple data logger that logs IOT sensor/device data in JSON or csv format to a text log file.
The project Consists of two main modules.
- A logger class module mlogger.py
- The logging script mqtt-data-logger.py
It also has helper module commands.py which gets and parses command line arguments.
Note: There is also the Simple Python MQTT Topic Logger which logs data based on topic that may be more suitable for your project.
Logger Class
The class is implemented in a module called m_logger.py (message logger).
It can be used for logging data from any source not just MQTT.
To create an instance you need to supply three parameters:
- The log directory- defaults to mlogs
- Number of records to log per log- defaults to 1000
- Number of logs. 0 for no limit.- defaults to 0
log=m_logger(log_dir="logs",log_recs=1000,number_logs=0):
The logger creates the log files in the directory using the current date and time for the file names.
The format is month-day-hour-minute e.g.
June 14th 10:50 am
You can log data either in plain text format or JSON format.
To log data either in plain text then use the
- log_data(data) method. Data must be a string
To log data as JSON encoded data call the
- json_log(data) method.
Data is a list or dictionary..
e.g.
log.log_data(data) or log.log_json(data) #The log file will contain the data as #plain text or JSON encoded data strings #each on a newline.
The logger will return True if successful and False if not.
To prevent loss of data in the case of computer failure the logs are continuously flushed to disk .
MQTT Data Logger Code Overview
This script will log data on a collection of topics and by default only if changed. It logs
- Message time
- Message topic
- message
Data is logged as a JSON string. The following lines are added to the on_message callback or a function called by the on_message callback..
data["time"]=tnow
data["topic"]=topic
data["message"]=msg
#log.log_json(data) #log in JSON format
You can log the data direct as shown above, but I’ve commented it out.
Instead I prefer to use a separate thread to log the data and so I place the data in a queue.
The message handler function calls the has_changed function to check if the message status is different from the last message.
If it is the same then the message isn’t stored as there is not point storing the same message value multiple times.
The worker takes the data from the queue and logs it to disk.
The relevant code is shown below.
def on_message(client,userdata, msg): topic=msg.topic m_decode=str(msg.payload.decode("utf-8","ignore")) message_handler(client,m_decode,topic) #print("message received") def message_handler(client,msg,topic): data=dict() tnow=time.localtime(time.time()) m=time.asctime(tnow)+" "+topic+" "+msg data["time"]=tnow data["topic"]=topic data["message"]=msg if has_changed(topic,msg): print("storing changed data",topic, " ",msg) q.put(data) #put messages on queue def has_changed(topic,msg): topic2=topic.lower() if topic2.find("control")!=-1: return False if topic in last_message: if last_message[topic]==msg: return False last_message[topic]=msg return True def log_worker(): """runs in own thread to log data""" while Log_worker_flag: while not q.empty(): results = q.get() if results is None: continue log.log_json(results) #print("message saved ",results["message"]) log.close_file()
The worker is started at the beginning of the script.
t = threading.Thread(target=log_worker) #start logger Log_worker_flag=True t.start() #start logging thread
The Log_worker_flag is used to stop the worker when the script terminates.
Using the Data Logger
Note: By default the logger will only log changed data. This is for sensor data that is repeated.
Also the changed data routine only examines the complete message and will not work correctly if the message contains a time stamp or message counter.
So ideally for this to work then the data should be a simple state value like on/off or a simple reading like 20.1 and not a complex JSON value.
If the message is JSON data then the change routine may not work but the data will still be logged.
For example a door sensor publishing it’s status every 10 seconds will send the same message repeatedly until someone opens or closes it.
You need to provide the script with:
- List of topics to monitor
- broker name and port
- user name and password if needed.
The base log directory and number of logs have defaults.
The script is run from the command line. Type
python3 mqtt-data-logger.py -h
for a list of options.
Example Usage:
You will always need to specify the broker name or IP address and the topics to log
Note: You may not need to use the python prefix or may
need to use python3 mqtt_data_logger.py (Linux)
Specify broker and topics
python mqtt_data_logger.py -b 192.168.1.157 -t sensors/#
Log as JSON
python mqtt-data-logger.py -b 192.168.1.157 -t sensors/# -j
Log as CSV
python mqtt-data-logger.py -b 192.168.1.157 -t sensors/# -c
Specify broker and multiple topics
python mqtt-data-logger.py -b 192.168.1.157 -t sensors/# -t home/#
Log All Data:
python mqtt-data-logger.py b 192.168.1.157 -t sensors/# -s
Specify the client name used by the logger
python mqtt-data-logger.py b 192.168.1.157 -t sensors/# -n data-logger
Specify the log directory
python mqtt-data-logger.py b 192.168.1.157 -t sensors/# -l mylogs
Note 29/2/2024
Updated to include logging as a csv file see readme in the zip file
Comments and Feedback
Was there enough detail in this tutorial for you to follow?
Please help me improve these tutorials by leaving your comments,rating them,asking questions.
Download
If you are using the logger in your projects then let me know and I can feature them on this page.
If you found this guide useful then perhaps you would like to Buy Me a Coffee
Related Tutorials
- Video Encoding/decoding JSON data in Python
- Logging MQTT sensor data
- My Python Working Notes
- Simple Controllable MQTT Sensor Simulator in Python
- Send a File Using MQTT -MQTT Projects
- Encrypting MQTT Payloads with Python – Example Code
- Store MQTT data in database
- Converting JSON to CSV with Python
Hi Steve! I noticed that in your code mqtt-topic-logger.py you set data[“time_ms”]=int(tnow*1000)
but in the log file there are only “time”, “topic” and “message”. “time_ms” is missing.
I would interested to get messages from sensors every ms. Do you know how can I get messages every ms?
even though it is labled time it is time in ms. If you are using python3.7 or above use
unix_time_ms_1 = int(time.time_ns() / 1000000)
for a more accurate value.
Does it make sense?
Rgds
Steve
Hi Steve,
Will this script work for mosquito broker ???
What changes are required if I want to do logging on same pi on which broker is running ???
Hi
It isn’t a broker logger in that it doesn’t log all messages passing through the broker.It is a client script that logs selected messages. I can be run on the same pi as the broker.
Rgds
Steve
Hi I have used this.But I am not getting how it is publishing the data to my cloud.If you could throw some light on this.
The data logger logs data to a local file and not the cloud.
Rgds
Steve
Hi! I’ve thi issue
Traceback (most recent call last):
File “mqtt-data-logger.py”, line 174, in
client.username_pw_set(options[“username”], options[“password”])
NameError: name ‘client’ is not defined
closing log file
Hi
There was an error with passwords on the script but I thought I fixed it several months ago. If you don’t have a recent download then download it again from the site.
Did you get it from the site or from github?
Can you let me know so I can correct it.
Rgds
Steve
Hi Steve and thanks for sharing!
I’ve got it working on my Raspberry and PC with Win10. What concerns me is that it drains 25% of processor on both machines. Did you notice that on yours or is it only mine issue?
Thanks,
Mark
Mark
No I didn’t How many messages are you logging?
rgds
steve
I’m running it for educational purposes, so I publish single message once on a minute. It drains 25% even if it is idle (not processing any messages). Strange behavior :/ Well, if you didn’t encounter such issue, then I will write my own version 😉 My goal is to log messages into MySQL database.
On Raspberry is Python 2.7 and PC is Python 3.7
Thanks,
Mark
Mark
In tests I sent 100s of messages a second to the logger and didn’t notice any slowness. I’m due to revisit the code shortly and will look specifically at the processor usage.
Rgds
Steve
I have the same problem(25% draining ), did you find a solution for it?
Tks
Hi Steve i have followed all instructions but I’m receiving this error:
File “c:\mqtt-data-logger-master\mqtt-data-logger.py”, line 75, in
client1.username_pw_set(options[“username”], options[“password”])
NameError: name ‘client1’ is not defined
Any help would be great thanks.
Hi
There’s an error in the script change client1 to client and it should work ok. I’m rewriting the script and I’ll try to get it on the downloads page in the next few weeks.
Still getting the same error. No worries i’ll just wait until the updated version , thank you.
I’ll try an email it you next week. Is the problem you are getting because you try to use username/password.
Can ou let me know the command line options you use.
rgds
steve
Hi Steve,
The logger seems to work for a while but eventually stops reporting new values and needs to be stopped and restarted again. Any thoughts on this?
I’m invoking it as:
python3 ./Downloads/pythLogger/mqtt-data-logger/mqtt-data-logger.py -b RPi5 -t /ESP14/#
Thanks,
Brian H.
Did you notice any network problems when it stopped working? How long did it run for?
rgds
steve
I surmise I need to move mlogger.py to somewhere known to python3:
bjh@bjhProBook1:~/Downloads/pythLogger/mqtt-data-logger$ python3 ./mqtt-data-logger.py
Traceback (most recent call last):
File “./mqtt-data-logger.py”, line 14, in
import classes.mlogger as mlogger
ModuleNotFoundError: No module named ‘classes’
Where do I need to move it to?
You need to create a folder called classes in the folder that hold the data logger scrippt and put the mlogger.py file there. Alternatively edit the code to the location of the mlogger.py file.
You might find this command at the top of the python script useful
sys.path.append(“c:/python34/steve/”)
it places the folder in the search path just adjust it to what you need.
Thanks Steve.
This was my work around:
from importlib.machinery import SourceFileLoader
mlogger = SourceFileLoader(“mlogger”, “/home/pi/serial/classes/mlogger.py”).load_module()
command = SourceFileLoader(“command”, “/home/pi/serial/classes/command.py”).load_module()
Shouldn’t the script do a “$(mkdir -p …)” type command to ensure it has the directory structure it needs. Same with $(which {module}) commands to ensure it has all the right python modules??
The script currently uses the local folder unless you explicitly set it. You can modify the script any way you want as it is open source and is meant as demo code. I use it often for recording broker data so I can replay it later.
Rgds
Steve
Hi Steve,
What do you mean by creating a logger class in module m_logger.py? Do we have to create the module ourselves? If so how to go about it.
Thanks,
Dhaval
The logger class is what does the data logging. The code is in the file m_logger.py.
It is created by the script so you don’t need to create it yourself.
rgds
steve
Hi Stave,
I tried the mosquitto sever and the python data logger script in the windows it’s running ok. But I want to remember you that the multiline comments (” ” ” ) in the code of logger scripts give a fatal error! I fixed it by using (#) comment for each line.
Some of the command lines above also in the github readme “mqtt_data_logger.py” written by the underscores but code files names with dashes you can unify all of them.
Thanks
Tks for the heads up. Is the the version from github or from the download page?
The version of the github.
Hello Stave,
Does the script run on Windows?
Thanks
yes
Problem solved! Your script runs under python3
Thanks for your support
regards
Raffaele
I’ve already tried to run under all python version:
python mqtt_data_logger.py -b 192.168.1.110 -t 4te/sonoff-TH/tele/SENSOR/#
python2 mqtt_data_logger.py -b 192.168.1.110 -t 4te/sonoff-TH/tele/SENSOR/#
python3 mqtt_data_logger.py -b 192.168.1.110 -t 4te/sonoff-TH/tele/SENSOR/#
but the issue persist
Thanks for your job Steve,
I’ve tried using your data logger, but it seems that module “queue” is missing. Thanks for your attention.
Raffaele
pi@raspberrypi:~/Desktop/mqtt-data-logger-master $ python mqtt_data_logger.py -b 192.168.1.110 -t 4te/sonoff-TH/tele/SENSOR/#
Traceback (most recent call last):
File “mqtt_data_logger.py”, line 18, in
from queue import Queue
ImportError: No module named queue
I just downloaded i and tried it and it seems to work ok. Can you send me a screenshot of the error message
rgds
steve
Thanks for your replay Steve,
please find the screenshot the screenshot of the error message
https://www.dropbox.com/s/hthad8ggj8ffqyf/SS.png?dl=0
regards
Raffaele
Hi
Looks like you need to install the queue module using pip. I thought it was already installed by default
Hi Steve,
I’ve checked and the “queue” module is already present into the Python repos, as you can see in the following screenshot https://www.dropbox.com/s/d7pf10dpknhajuq/SS2.png?dl=0
Any suggestions to overcome the issue?
Regards
Raffaele
I think it may be trying to run under python2.7 try
python3 mqtt-data-logger.py
Thanks Steve.
Brand new to MQTT, but the simplicity and clarity of your website got me started in a few minutes monitoring my first Mosquitto remote broker. Looking forward to testing the logger and SQL scripts next.
Barry