Data from sensors will need to be stored. But how much data do you really need to store?
Many sensors like state sensors (ON/OFF), temperature etc will fluctuate very little over the course of a day so does it makes sense to store the same sensor value every x seconds?
Take for example a light sensor that publishes its status every second.
After 1 hour it has published 3600 status messages.
If we assume that the light status hasn’t changed during that period then it doesn’t make much sense to store the 3600 status messages.
The only messages that need to be stored are changed messages.
With this method if our light changed state twice in the hour then we would store 2 messages.
Over a day assuming 20 state changes we would store 20 messages rather than 86,400(24*3600) messages.
However even if the sensor does send data a regular intervals regardless of state then it doesn’t make sense to store it.
Therefore a data logger should have the ability to only log changes.
The sensor needs to publish with the retain message flag set.
A sensor shouldn’t publish a timestamp with the sensor data if it needs to be logged. If it does the message will change and the data logger would need to take this into account.
I have created several data loggers for logging IOT data and specifically MQTT data.
Simple Simple Python MQTT Data logger Logs data to text file as JSON encoded Data.
The SQL MQTT data Logger which logs data to a sqlite database.
Simple MQTT Topic logger -Logs data to a text file as JSON encoded Data
Resources and related tutorials
- Simple Controllable MQTT Sensor Simulator in Python
- MQTT Sensors and Network Traffic Observations
- MQTT Topic and Payload Design Notes
- Checking Active MQTT Client Connections