Newsletter

Due to lots of false signups I have had to remove the form until I can find a better solution

Subscribe to Receive Latest Posts

I do not publish a regular newsletter but If you want to to be notified when there are new posts on the site, new scripts to try and new videos on line then sign up below.

Best Regards

Steve

May Newsletter

Hi

last-will-messages-iconThe last will and testament message is used to notify subscribers of an unexpected shut down of the publisher.

The basic process is.

  1. The publisher tells the broker to notify all subscribers to a topic, using the last will message , in the event that the connection breaks
  2. If the broker detects a connection break it sends the last will message to all subscribers of that topic.
  3.  If the client disconnects using the disconnect message then no last will message is sent.

MQTT Last Will and Testament Use and Examples

MQTTv5 added more properties to the last will you can read about them here

Birth Messages are not part of the MQTT specification but they are part of the Sparkplug standard and they have always been a setting in the node-red MQTT node.

See Understanding MQTT birth and death messages for an example of how it is used.

Republish web data using MQTT

Because http is a request/response protocol a visitor viewing a web page must issue a new request in order to refresh the data.

For most websites this isn’t a problem as the page content is static, but there are a class of sites were the information being displayed is changing, and for which http may not be the best way to deliver it.

Flight Information Example- The Problem

Data for Live flight arrivals or departures is displayed on a web page and is updated on a regular basis (every 60 seconds is common).

Due to the nature of http protocol the page refresh is usually accomplished automatically by JavaScript code running on the web page.

When the web page is updated then usually the entire contents of the page is sent to the web browser regardless of the changes.http-data-request

Ses  Republish HTML Data Over MQTT

Books

Just a reminder that I have several books on Amazon:

My Other Sites

You may or may note be aware that I have two other related sites that you may find useful. They are:

Feedback and Suggestions

I am always happy to receive content suggestions and feedback on tutorials on the site. Please use the suggestions page.

If you need help then use the Ask Steve page to get in touch.

Until next time.

Best Regards

Steve

———————-

April Newsletter

In IOT and home Automation there are two basic device types :

  • Sensors
  • Actuators

1. Sensors
– Definition: Devices that detect and measure physical or environmental changes and convert them into electrical signals.
– Purpose: Gather data from the environment or a system.
– Examples:
– Temperature Sensors (Thermistors, Thermocouples)
– Proximity Sensors (Ultrasonic, Infrared)
– Light Sensors (LDR, Photodiodes)
– Pressure Sensors (Barometers, Strain Gauges)

2. Actuators
– Definition: Devices that receive signals (usually electrical) and perform a physical action.
– Purpose: Convert electrical energy into mechanical movement or other actions.
– Examples:
– Motors (DC, Servo, Stepper)
– Relays (Switching Circuits)
-Fans

Although it is commonly thought that sensors only send data as that is there primary purpose they can also receive data in the form of commands.

For example a Temperature/Humidity sensor will send temperature and humidity data, but may also receive configuration commands.For example it can receive commands to:

  • Change the sensor Name
  • Change the MQTT broker
  • Change the Wi-Fi Network

Therefore regardless of whether you are dealing with sensors or actuator and MQTT you will need to design a command/response topic structure.

A good starting point is to look at the structure used by Tasmota and Shelly to control their devices. See

Here are two videos that demonstrate using node-red to control devices.

 

Sensor Data Type

Most sensors publish additional information in addition to the main data.

As an example a temperature/humidity sensor will publish the temperature and humidity readings.

In addition it will also publish other data like iP address,Wi-Fi access point,Battery level etc.

This additional information can also be used by the control dashboard.

For Example the battery level can be used to trigger an alarm when it gets low so that someone can replace it before it fails.

Sample data from a Zigbee temperature sensor is shown below.

temp-sensor-data

Very often this auxiliary data is published on a separate topic.

Latest tutorials on Site

HTTP vs MQTT for IOT Devices
Design A Network of 100s of MQTT Parking lot Sensors

My Other Sites

You may or may note be aware that I have two other related sites that you may find useful. They are:

Until next time.

Best Regards

Steve

March 2025

Python Version2 Client

I’ve mentioned the changes that have been made to this client that can cause problems when used with older scripts.

All of my scripts that are availble for download were written for version1 of the client and will need modifying if you upgrade to version2.

I have done a tutorial with more detailed notes to help and it is here

Python Broker Checker

I have developed a number of tools for monitoring MQTT brokers that use node-red.

The reason for using node-red is that it is easy to use, and comes with a dashboard.

However when it comes to monitoring multiple brokers it get complicated.

My solution it to do the monitoring using Python and then use node-red to log and display the results.

The script can be preloaded with a list of brokers to monitor and new brokers can be added or removed by sending a command to the script using MQTT.

Full details are here

MQTTv5

I did a poll a while back and was a bit surprised to find that some of you were using MQTTv5 already. I assume that since that poll that many more are now doing so.

So I thought I would feature MQTTv5 in the next few newsletters to get you aquainted with it if you haven’t already started using it.

Most brokers  have supported both protocols for several years now but the choice of protocol is made by the connecting client.

For Python clients the choice is made when creating the client object as shown below:

client = mqtt.Client("mqtt5_client",protocol=MQTTv5)

MQTTv5 Properties

Probably the most important addition to the MQTT protocol that was included in MQTTv5 was the properties field.

MQTTv5 Specification

The last field in the Variable Header of the CONNECT, CONNACK, PUBLISH, PUBACK, PUBREC, 477 PUBREL, PUBCOMP, SUBSCRIBE, SUBACK, UNSUBSCRIBE, UNSUBACK, DISCONNECT, and 478 AUTH packet is a set of Properties. In the CONNECT packet there is also an optional set of Properties in 479 the Will Properties field with the Payload.

What properties are available depends on the message type. This tutorial goes into detail on the properties per message type.

As mentioned in the Python tutorial, the main change in the new python client was made to accommodate the properties field present in most MQTTv5 messages.

Feedback

I would be interested to hear from anyone who is using MQTTv5 because they need to use one of the new features. Just reply to this email to let be know what feature and the application.

Tks in advance

My Other Sites

You may or may note be aware that I have two other related sites that you may find useful. They are:

Until next time.

Best Regards

Steve

January 2025

This is my first newsletter in 2025, so I would like to wish you all a  happy new year.

IPv6

My IPv6 tutorial was written some time ago and has been very popular.

I am in the process of reviewing this tutorial and expanding it with examples network configurations.

If you have any other suggestions on what you would like included then please let me know on the suggestions page.

On that note I have put together a tutorial on using IPv6 with Mosquitto which has been possible for a long time.

See How to use the Mosquitto Broker with IPv6

Mosquitto and Log_dest topic

This is a very useful option which send console log information to the $SYS topic which you can view by subscribing to the $SYS/broker/log/#

This is what it looks like

mosquitto_log_dest_topic

Mosquitto_sub and Pub for Mosquitto v5

I have updated the mosquitto_pub and sub tutorial to included some MQTTv5 options. –Using The Mosquitto_pub and Mosquitto_sub MQTT Client Tools- Examples

 MQTT News

This $3,000 Android Trojan Targeting Banks and Cryptocurrency Exchanges

What is interesting is that it uses MQTT to send back the data and https to receive instructions.

You can read more here

https://thehackernews.com/2024/12/this-3000-android-trojan-targeting.html

Smart Labels -This looks like a trend to really watch.

Open Source Voice Assistant

This is something I’ve been waiting for and it is certainly something that most smart homes need.

Instead of using Google, Alexa or Siri to process voice commands you can do it all locally with this hardware from the company behind Home Assistant. Read More here

Order link is here but it is currently on back order.

Have you Heard of Smart Quotes?

Q- What are Smart Quotes?

Smart Quotes are found in Windows Word documents and also on some web pages. They can cause trouble if you have them mixed in with JSON data and so you will need to replace them. The problem is that it is not always obvious as the error message you get is invalid JSON data.See this online tool

The reason I mention it is that I had a problem with them myself when pasting a command from one on my web pages.

Take a look at this command (with smart quotes)

Zbsend {“device”:”0xD027″,”Send”:{“AddGroup”:100}}

and This command (normal quotes

Zbsend {“device”:”0xD027″,”Send”:{“AddGroup”:100}}

The first command will result in a JSON error.

Data Extraction and Conversion

One of the most common task when dealing with MQTT data is extracting data from incoming messages and converting between different data formats i.e JSON to CSV.

I have also a book on Amazon that contains many more data extraction examples.

JSON and JavaScript Workbook

Latest videos

My Other Sites

You may or may note be aware that I have two other related sites that you may find useful. They are:

Until next time.

Best Regards

Steve

November 2024

This will be my last newsletter in 2024, so I would like to wish you all a happy Xmas and a happy new year.

Important Python Client Information

I mentioned last month about the Python MQTT client changes in version 2.1.

I went back and read the notes and found that it is actually quite easy to overcome by changing a single statement in the existing code.

When you create a new client instance using

client=Initialise_clients(cname)#create and initialise client object

then use:

client= mqtt.Client(mqtt.CallbackAPIVersion.VERSION1,cname)

Everything else stays the same.

I hope to be going into the client in more detail over hte next month and will do a few tutorials on the changes.

MQTT-SN Poll

There is a poll on the site regarding using MQTT-SN which surprised me somewhat as I wasn’t expecting so many responses as I didn’t think so many people were using it.

If you are using it then i would appreciate it if you would let me know why? and a little bit about the application etc.

You can use the suggestions page or simply reply to this newsletter.

MQTT Tools

I’m a big user of the mosquitto_pub and sub tools but have recently been using the MQTT explorer tool a lot.

It is very useful when you need to publish and subscribe as you can do it all in a single window.

I did a quick introductory tutorial a while ago which you can find here.

SSl Certificates

I’m already big fan of using your own self generated certificates whenever possible.

I recentlry received an email from fiesty duck regarding reducing the length of a certificate to 45 days down from the current 1 year.

What I also never realised is that browsers seem  to ignore certificate revocation.

The article is well worth a read if you are using or thinking of using commercial certificates.

Are you Staying Connected?

MQTT is a connection orientated protocol and so it is common for a client to connect and stay connected even though there is no data being sent.

The problem with this is that each connection takes up resources on the MQTT broker.

This is not a big problem with a few hundred clients but with thousands of client it could be.

I wrote an article a while ago on this which you can read here and would be happy to hear your thoughts and hear of experiences you have when dealing with lots of clients.

 

October 2024

Important Python Client Information

The new Python client version 2.1 has some breaking changes which are documented here and I recommend that you read.

To avoid having to upgrade older code you should not upgrade to this version.

If you have upgraded and your old code isn’t running then you can go back to the old version using:

pip install -v --force-reinstall "paho-mqtt==1.6.1"

you can use pip list to show the current version you are using:

pip-list

MQTT and Mosquitto Security Scenarios

In last months newsletter we looked at Mosquitto ACLs.

ACLS are not used alone and depending on your requirements there are other measures like authentication,encryption, firewalls etc that will need to be considered.

I have added a second section to the MQTT security tutorial to discuss these requirements.

Getting Test data using Node-Red

One of the things I do on a regular basis is to gather test data.

This might be for trouble shooting or for development where it is necessary to see the data structure.

To do this I use a simple node-red flow that stores the data in a file with a time stamp and the topic.

The flow looks like thisstore-mqtt-data

You can download the flow using the link below:

download

You will need to change the broker details and the file storage location.

You can view the saved data with a text editor and even play the data back into another flow .

This I do quite often as instead of working on live data I take a snap shot and use that stored data to develop the code.

In next months newsletter we look at how we replay the stored data into a flow.

Arduino and ESP Clients

I have done a few tutorials on Arduino before but haven’t worked with it for a few years until recently.

Because of a recent project I thought it would be a good opportunity to expand the Arduino tutorials and also to include ESP32 and ESP8266.

Also I did receive a comment regarding the pubsub client being the preferred client when it comes to tutorials on the Internet, so I decided to try out the Arduino mqtt client as well.

The basic tutorials are still not complete as I hope not only to cover Arduino with MQTT but also MQTT-SN.

You can find the current tutorials here.

Grateful for you feedback if you have time.

Python Code -Creating a Client Name

I am slowly working my way through my python tutorials and revamping code were necessary and so thought it might be a good idea to include some code in the newsletter that I use in many of my scripts.

Because MQTT clients require a unique name then it is common to generate a random name. However random names aren’t useful if you use ACLs and client ids.

So this is the way I generate my client names. The schemes are:

  • Client prefix + Serial number of device
  • Client prefix + Random Number
  • Client prefix + Time
  • Client prefix + time+random number

Using Random

r=random.randrange(1,10000)
cname="testclient-"+str(r)

Using time

r=str(time.time())[-5:-1]
cname="prefix-"+r

Interesting Articles I’ve come Across

Books

Just a reminder that I have several books on Amazon:

My Other Sites

You may or may note be aware that I have two other related sites that you may find useful. They are:

Until next time.

Best Regards

Steve

September 2024

Security -ACL Restrictions

ACL restrictions allow you to control which topics users/sensors/devices can publish and subscribe to.

For a standard test broker then it is usually completely open but for a production broker it should be almost completely closed.

Mosquitto provides two ways of configuring ACL restrictions. They are.

According to a poll I did on the site the old style restrictions are way more popular than using the dynamic security restrictions.

Testing ACLs

As far as I can make out this is generally done manually but it is one area I am looking at to develop some python testing tools as currently I do it manually.

If you have automated the process I would be grateful if you could let me know some details just by simply replying to this newsletter.

ACL restrictions are part of the Mosquitto/MQTT security mechanisms. Other mechanisms are SSL and authentication.

Next month we will look at some security scenarios.

MQTT-SN QOS3 or -1 Gateway

If you are interested in MQTT-SN then this gateway may be of interest.

My personal views as been that MQTT-SN has been ignored due to the lack of reliable broker software and I have never been a fan of the gateways.

In addition I thought that they made a big mistake with MQTT-SN by making it connection orientated using a virtual connection.

It was probably just a quick way of getting the protocol specified as quickly as possibly.

However IMO the saving grace was the ability to public without a connection using QOS of -1 (3).

See MQTT-SN To MQTT QOS -1 Gateway

Interesting Articles I’ve come Across