Node-Red MQTT Data and Topic Logger

This is a data and topic logger similar to the python data and topic loggers that can be used for logging MQTT data to log files either as a single file containing multiple topics or multiple files based on topic.

It is controlled using a node-red dashboard making it much easier to use than the python loggers.



node-red-topic-logger-dashboard

Starting The Flow

When starting the flow it defaults to logging all data to a single log file. (see log files names and sizes below).

The data source in the flow is MQTT but can be other data sources as the data is passed using the msg.payload.

The data source doesn’t need to provide a topic unless you are trying to log by topic.

By default only the payload is logged but you can also log the topic, and a time stamp.

By default all data is logged.

Logging by Topic

If you enable the switch to log by topic the the logger creates separate log files for each topic and follows the MQTT topic hierarchy as shown in the schematic below:

MQTT-Topic-Logger File-Folder-Structure

Log File Names and Sizes

The current maximum log file file is 5MB after which a new log file is created the names use the structure

logxxx.txt

where xxx is a number starting with 000 so the first log file is log000.txt.

If you want to change these settings you will need to edit the store data function node.

Flow Overview

The node consists of several switches for setting the various options.

These switches are initialized using an inject node and reset button.

The control logic for these switches is contained in the Set Switches function node.

The switches store their settings in a flow object which can be read in the store data function node.

Each function node starts with this line

var local=flow.get('local') || {};

The MQTT node will need to be edited to reflect your system. You will need to change the broker and the topic.

You could add additional MQTT nodes to subscribe to multiple topic hierarchies

Using the Logger

You need to set the various option i.e. log by topic etc before activating the start logger switch.

Any change to the switches after logging has been started are ignored.

Stopping the logger actually pauses the logger as if you start it again it will continue to log without resetting any counters or file names.

To start a new log then use the reset button and manually clear the existing logs or the logger will append to them.

By default it logs all data but you can change it using the slider control to only log changes data.

This works very well when logging by topic.

Performance

In the tests I did uses node-red and the mosquitto broker on a Raspberry pi I logged 6000 messages (40 bytes) at 173 messages/second without data loss.

Topic-logger-tester-results

Video

I will be creating a new video to cover he current logger. However it is based on the data logger covered in an earlier Youtube video which you can see here

Flow Download

Comments and New Feature Requests

Grateful any comments and ideas for new features

Related Tutorials and Resources

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Leave a Reply

Your email address will not be published. Required fields are marked *