A MQTT bridge lets you connect two MQTT brokers together. They are generally used for sharing messages between systems.
A common usage is connect edge MQTT brokers to a central or remote MQTT network. The Mosquitto broker (server) can be configured to work as an MQTT bridge.
Generally the local edge bridge will only bridge a subset of the local MQTT traffic.
In addition you only need to configure 1 of the brokers to act as the bridge, the other will act as a normal broker.
How an MQTT Bridge Works
When you configure the mosquitto broker to act as a bridge it becomes an MQTT client to the remote broker and subscribes and publishes to topics on the other broker just like a normal MQTT client.
Configuring the Broker as a Bridge
You will need to configure
- The Address and port of the remote broker or brokers. – Address keyword
- A client name. – Connection keyword
- What topics the broker will publish, and which topics it will subscribe to. -topic keyword
- Remapping if needed.
Note: You can configure multiple remote broker addresses and the broker will switch to another broker if the current broker connection fails.
Topic Bridging
What topics are being bridged, and if they are being remapped is controlled by the topic entry. Wildcards can be used in the topic entry.
The general format is
topic topic pattern direction QOS local prefix/remote prefix.
Direction can be out,in or both.
out = publish from the broker
in = receive from remote broker
both = publish and receive
Note: If you aren’t using topic prefixes then you don’t need to add them.
This is best seen in examples using the layout in the schematic below
Example 1
Configuration file topics
topic house/# out
Result:
client 1 publishes on house/sensor1 or house/anything and the message is sent to broker 2
client 1 publishes on house1/sensor1 and the message is not sent to broker 2
client 2 publishes to any topic and nothing is sent to broker B1.
Example 2
Configuration file topics
topic house/# in
Result:
client1 publishes to any topic and nothing is sent to broker B2.
client 2 publishes on house/sensor1 or house/anything and the message is sent to broker B1
client 2 publishes on house1/sensor1 and the message is not sent to broker B1
Example 3
topic house/sensor1 both 0 “” “”
or
topic house/sensor1 both 0
The entries:
topic house/sensor1 in 0 “” “”
topic house/sensor1 out 0 “” “”
Are equivalent to the single entry:
topic house/sensor1 both 0 “” “”
and also
topic house/sensor1 both 0
Result:
client 1 publishes on house/sensor1 or house/anything and the message is sent to broker B2
client 2 publishes on house/sensor1 or house/anything and the message is sent to broker B1
client 1 publishes on house1/sensor1 and the message is not sent to broker B2
client 2 publishes on house1/sensor1 and the message is not sent to broker B1
The screen shot below shows how to bridge all topics with no remapping.
MQTT Topic Bridging and Remapping
This is accomplished using the local and remote prefixes.
When the bridge on broker1 subscribes to a topic on broker2 it uses the form
remote_prefix + topic name e.g. b2/house/sensor1
When the bridge on broker1 subscribes to a topic on broker1 it uses the form:
local_prefix + topic name e.g. b1/house/sensor1
The topic prefixes follow the QOS entry.
The screen shot below shows a remapping configuration for broker 1:
With the above configuration then:
- When Client1 publishes on any topic that message will be sent to broker 2 with the topic prefix b2/. Example incoming client message to B1 on topic house/sensor1 is published to broker 2 on topic b2/house/sensor1.
- When a client publishes a message to broker2 then messages published on b2/# are sent from broker2 to broker 1 but without the b2/ prefix. A message published to broker 2 on house/sensor1 is not sent to broker 2 as it doesn’t have the b2/ prefix
Broker 2 Configuration
Broker2 is functioning as a normal broker (not a bridge) and doesn’t normally need any extra configuration.
Bridging and Topic Remapping Examples
The following examples refer to the schematic above with two brokers Broker1 and Broker2 . Broker1 is configured as the bridge.
Normal clients will also be connected to both brokers.
Entry uses remote prefix:
topic house/sensor1 both 0 “” b2/
Equivalent to
topic house/sensor1 out 0
topic house/sensor1 in 0 “” b2/
Result:
Incoming messages from a client connected to broker 1 on the topic house/sensor1 are being bridged and will be remapped and sent to broker2 on topic b2/house/sensor1.
Incoming messages from a client connected to broker2 on topic b2/house/sensor1. will be sent to broker1 on topic b2/house/sensor1.
Broker1 removes the b2 prefix and will send them to clients subscribed to house/sensor1 on broker 1.
Entry uses local and remote prefixes:
topic house/sensor1 both 0 b1/ b2/
this is the same as:
topic house/sensor1 out 0 b1/ “”
topic house/sensor1 in 0 “” b2/
Result:
Incoming messages from a client connected to broker1 on topic b1/house/sensor1 are being bridged and will be remapped and sent to broker2 on the topic b2/house/sensor1.
Incoming messages from a client connected to broker2 on the topic b2/house/sensor1. will be sent to broker1 on topic b2/house/sensor1.
Broker 1 removes the remote prefix (b2/) and then adds the local prefix (b1/) and will send them to clients subscribed to b1/house/sensor1 on broker 1
Working Bridge Example
Using the configuration shown above on Broker1 I am going to publish a message to Broker1.
Broker1 has the bridge configured to connect to Broker2 (IP address 192.168.1.157).
The topic bridging entry is:
topic # both 0 “” b2/
Broker1 accepts messages and publishes them to Broker2 with a prefix of b2/.
Broker1 subscribes to messages from Broker2 on topic b2/#
Using a simple python client we will :
- Publish a message to Broker1 on topic house/sensor1. Using a client name of P1
- Subscribe to Broker2 on topic b2/house/sensor1 -Using client name of P2
When you start the broker on broker 1.
- It connects to broker 2 and subscribes to the topics b2/#.
- Connects to itself and subscribes to topic #
This can be seen from the console log messages:
When I run the script this is what I see.
Broker 1 receives message on topic house/sensor1 and publishes them to broker2 on b2/house/sensor1.
Broker2 receives messages on b2/house/sensor1 from broker1.
Backwards Testing
I can also publish to broker 2 on topic b2/house/sensor1 and it should be sent to broker 1 and made available as house/sensor1.
So now I will alter my script to send to broker 2 on topic b2/house/sensor1 and subscribe to house/sensor1 on broker 1.
Here is a screen shot of the results.
Notice the message is received by the client subscribed to broker 1 n the topic house/sensor1
Topic Remapping and Topic Republishing
Topic remapping takes an incoming topic and renames it.
E.g A broker would receive messages to topic sensor1 and remap them to new_sensor1.
Currently this form of remapping is not available, but prefix remapping is as shown above.
Bridging to Multiple Brokers- Round Robin
It is possible to bridge to another broker in the event of a broker failure.
To do so you need to set multiple address:port entries in the address field i.e
address 192.168.1.168:1883, 192.168.1.185:1883
will use broker 192.168.1.185 if the connection to the first broker (192.168.1.168) fails.
The round robin setting controls what happens when the original connection is restored. Here is the description taken from the online manual:
Authentication and Encryption
If the remote broker requires a username and password then you need to enter this using the remote username and remote password options.
You can also encrypt the connection between the brokers.
This is likely to be a common configuration and Is covered in more detail in the Configure Mosquitto Bridge With SSL Encryption- Examples tutorial.
Looping
Publishing and subscribing to the same topic on the broker could result in a loop.
However if you try this you find that a broker detects that another broker sent the message, and doesn’t send the message back to it even though the broker has subscribed to that topic.
This confused me at first as looking through the specs I couldn’t find any reference for broker detection.
I did find this comment from Roger Light on a forum that explains it.
This seems to be triggered by the try private statement in the configuration file. This is what the config file states:
# If try_private is set to true, the bridge will attempt to indicate to the # remote broker that it is a bridge not an ordinary client. If successful, this # means that loop detection will be more effective and that retained messages # will be propagated correctly. Not all brokers support this feature so it may # be necessary to set try_private to false if your bridge does not connect # properly. #try_private true
Bridging to Thingspeak Note
I had a request for help regarding bridge configuration on mosquitto when creating a bridge to thingspeak.
If you need to do this you need the following two entries:
notifications false
cleansession true
See this forum comment
Problems Encountered
When configuring bridges you may encounter some issues. Here are some of the errors I’ve encountered.
Error message on broker –socket error on client <unknown>disconnecting
Cause – Configuration file error. In my case it was the PSK setting.
Problem – Remapping not working as it should.
Cause -I forgot the QOS entry
Video -How to configure a bridged connection on the mosquitto mqtt broker
Note: Bridging to Beebotte
If you need to create a bridge to beebotte then you need to use the try_private command as shown below:
This mosquitto.conf file works (Note that is not my real user token)
############ port 1888 log_type all connection bridge-01 address mqtt.beebotte.com try_private false cleansession true topic house/# out 0 topic house/# in 0 remote_clientid broker1 remote_username 5I6IJSjYNFtfcvNhEZXAAZXOpWWGZt6WfgAY #remote_password ' ' #################################
Bridging to Reducing TCP/IP connections
MQTT is a connection orientated protocol which means that each MQTT client has a connection to the broker.
Broker connections are TCP/IP connections which take up memory on the broker as each connection is allocated a memory buffer.
A very effective way of relieving the connection load on a central broker is to use edge brokers configured as bridges.
If you had 1000 clients that needed to connect to a cloud broker then the broker would need to support 1000 connections if they connected directly, but if you connected 200 clients to a gateway broker then the cloud broker would only need to support 5 client connections.
Summary
MQTT bridging on Mosquitto allows you to connect two or more MQTT networks together. MQTT bridging is likely to be used to connect edge networks to a central network. Generally only a subset of the local topics will be bridged.
Bridging Alternatives
You can also use a client to do the bridging rather than the broker.
This means that you don’t need to be a broker administrator to do it and your broker doesn’t need to support it.
See Bridging using the python client
Related Articles and Resources:
- Mosquitto.conf documentation
- Connecting the last 100 metres –
- Owntracks -bridging
- Implementing SSL on the mosquitto broker.
Mosquitto Configuration Tutorials
- Installing The Mosquitto broker on Windows and Linux
- Quick Guide to The Mosquitto.conf File With Examples
- MQTT TLS Security – Mosquitto Broker Configuration
- Configuring and Testing MQTT Topic Restrictions
- Mosquitto username and Password Authentication Configuration Guide
- Configuring Logging on The Mosquitto Broker
Hi Steve, great Tutorial …
… but I have a issue…
I have a Broker in a secure VLAN (no INTERNET) 10.0.0.241:1883 and a Bridge Broker in my private LAN 192.168.178.2.
– PRIVATE LAN can connect to VLAN
– VLAN can’t connect to PRIVATE LAN
So, I have in VLAN this simple config:
“`
listener 1883
allow_anonymous true
“`
In my PRIVATE LAN I have the Broker config:
“`
# Get data from VLAN
connection VLAN_MQTT
address 10.0.0.241:1883
# ✅ Allow read all topics
topic # in 0
# ✅ Only zigbee topics/commands allowed to VLAN
topic zigbee2mqtt/# out 0
# ❌ Block all other topics from PRIVATE LAN to VLAN
topic # out 0 “”
“`
My Problem is now, if I send from the Bridge Broker (PRIVATE LAN) a Message like:
`mosquitto_pub -h 10.0.0.241 -t “foo/var” -m “TestMessage”`
The Message appear in the VLAN Broker:
`mosquitto_sub -h 127.0.0.1 -t “#” -v`
How can I block all Traffic from LAN to VLAN, except zigbee topics to control on/off command to the VLAN ?
You need to remove the
topic # out 0 entry
However in this configuration you won’t see any responses from the other broker on the VLAN regardless of the ACL entries on the bridge.
Rgds
Steve
Thanks;
https://ibb.co/kGHbm2k
Should have explained that the tinygs username is autogenerated and is numeric signed.
The top level topic is tinygs and then next level is your username -XXXXXX
Then all of your satelite stations are children of that:
tinygs
-XXXXXX
station_1
cmd {…..}
data {….}
station_2
cmd {…..}
data {….}
Hi
It looks like a dash is allowed did you add a direction and qos and does it work?
Rgds
Steve
did you see the image I sent from MQTT explorer ?
This is current config, still nothing shown:
connection tinygs-bridge
address mqtt.tinygs.com:1883
remote_username -1090xxxx
remote_password vqPwi7mxxxxxxxx
topic tinygs/-1090701563/# both 0
Thanks for this.
I have setup a bridge on home assistant for tinygs, the connection seems to work however I do not see the topic / messages from tinygs on my ha mosquitto.
I do see the stats for the connection under $SYS\broker however the topic nor the messages appear on the broker, should they ???
===
connection tinygs-bridge
address mqtt.tinygs.com:1883
topic -109070XXXX/#
remote_username -109070XXXX
remote_password vqPwxxx
===
Appreciate your advice ?
The topic should look like this
topic house/sensor1 both 0
yours is
topic -109070XXXX/#
You need a direction and QOS. Not sure about the – in the topic name I would need to check the spec.
Rgds
Steve
Hello Steve.
Thank you very much for all your work. The knowledge you share is priceless.
I have one issue with which I’ve been struggling for a while. Wondering if you could help me out with it.
I’ve set my bridge to beebotte. It looks like all messages I publish from my local broker mosquitto are being received at beebotte server. This however does not work the opposite way, so when i publish a test message from beebotte, such a message is not received by my local broker. Of course, in both cases, I publish under the topic homeassistant. It looks like one-way-only communication.
My bridge setting are as follows:
connection bridge-khjhgdfgcbvjhhh623364gfrt
address mqtt.beebotte.com:1883
try_private false
remote_clientid BEEBOTTE
log_type all
cleansession true
topic homeassistant/# in 0
topic homeassistant/# out 0
topic esphome/# in
topic esphome/# out
remote_username DELETED
remote_password DELETED
log_type all
bridge_protocol_version mqttv311
persistence false
Your feedback would be much appreciated.
Your config looks to be ok. I tried it on mine but the bridge connection kept disconnecting and reconnecting. Is your connection stable?
rgds
Steve
Hello.
Hymm, not really. I checked my mosquitto logs and I did not really find any disconnections. I do get PINGREQ to beebotte and also PINGRESP back.
Will try using a different version of mosquitto. Are you using the free account or the paid account?
rgds
Steve
Hello Steve.
I am using a free account.
Thanks an br
Alex
tks
You are getting further than I am. I can send andreceive using various client but the bridge connects and gets connack but then subscribes and gets disconnected. Is there a beebotte setting I am missing.
Rgds
Steve
Ok
Finally managed I had to use a Iam token on the bridge. It now works in both directions. On the client that I use for testing I use a channel token.
Rgds
Steve
I’ve done the same, iam token for the bridge. Nothing moved forward 🙁 still the same. Did you do anything in beebotte account, any settings ie channels etc? Maybe I am doing something wrong in the way i am testing it? But what would be easier than mqttexplorer and publishing messages from one to the other broker?
Thanks in advance for any hints.
When testing you can only send the a channel that you have defined.I subscribed to my local broker and then sent a message direct to beebotte on the channel.
When publishing and subscribing I used the channel token.
Is that what you did?
Rgds
Steve
I do it in two different ways:
1) I open two mqtt explorers. In one, I log myself in to my local – mosquitto – broker. In order to publish, I type in “topic field” -> “homeassistant” and in “json field” -> “test from local”
In the second mqttexploer i log myself in by using IAM token (same as in bridge) and I immediately can see the message I have sent from local broker
2) I go to mosquitto addon in my HA. I go to configure and similarly, i set topic to “homeassistant” and publish a “test from local”. Again, immediately on mqttexploer that is logget to beebotte i can see such message.
Whenever i publish anything in mqttexplorer that is logged in to beebotte, using same topic – “homeassistant”, I do not get the message in either HA mosquitto nor mqttexplorer logged into local broker.
Now, I also checked the “console” tab on beebotte website but there I can not see any messages from my local mosquitto broker nor I can publish anything.
Thanks
Alex
I had no luck with the console either.
Just to clarify when you publish do you use the topic homeassistant or homeassistant/sensorx
sensorx is just an example.
rgds
Steve
Hello Steve.
I just tried by publishing with topic homeassistant and homeassistant/0x00158d00070b7cd7 where 0x00158d00070b7cd7 is my real sensor under homeasssistant.
Neither way works
Can you contact me using the ask steve page and we can use email. I will need your tokens to test it.
Rgds
Steve
Alex
Did you see my last comment. If you want you can try my tokens as I don’t use beebotte for any real work. You will need to contact me on the ask steve page so I can email you.
Rgds
Steve
Hello again Steve.
I tried to send you a messgae via ask steve.
However, after i pressed “submit” i got an error. I’ve done it twice with the same outcome.
Please advise how to overcome it.
Thank you
BR
Alex
Hi
You should have got an email
Rgds
Steve
Hi Seve, thank you for your very helpful internet guide.
We currently have many MQTT-Clients where wie publisch a state “online” in each payload and a state “offline” in a LastWill message to a first level broker, similar to the chapter “MQTT Last Will and Testament Use and Examples”. If we bridge the topics of the clients from the first level broker to an upper level broker, the last will messages with the state “offline” is delivered from the upper level broker when a client disconnects, but NOT when the first level broker is stopped. I have the following question: Is the last will information transmitted over a bridge so that they are delivered in case of broker failure ?
Many greetings, Andreas
Andreas
Don’t see why not as it is just like any other message but I will test it and get back.
Rgds
Steve
Andreas
I checked it and it works as expected it could be the way you are testing.
The way I test is with a python script and I stop the client loop which eventually triggers a fail on the ping req/response which the broker sees as a bad disconnect and trigger the lwm.
Rgds
Steve
Hi !. Its the great article!.
I’m having a problem like I have one remote address and the two ports. The default 1884 will listens for all the topics messages sent in and out and the another port 2883 is only used to sent a message.
How can I achieve this using paho mqtt in javascript?
Sorry but can you explain the setup in more detail. Are you trying to bridge or something else.
Rgds
Steve
Steve, this is a great tutorial for anyone learning about bridges.
I suspect that there is a small error that needs to be fixed. Please consider looking into it.
At the first place you show the remapping , the section of “MQTT Topic Bridging and Remapping”
(The one that uses the image src of http://www.steves-internet-guide.com/wp-content/uploads/mosquitto-bridge-config-remapping-1.jpg).
There’s a mismatch between what the image shows the config to be and what the text below it describes it will do.
For the explanation text to be true, the brige config needs to be different.
topic # out 0 b1/ “”
should actually be
topic # out 0 b2/ “”
But actually, I think the topic config is right. You would want the broker to put the right prefix to say what realm the message came from. So, I think you should actually let the config be as is and you should change the first part of the text below the config to be:
When Client1 publishes on any topic that message will be sent to broker 2 with the topic prefix b1/. Example incoming client message to B1 on topic house/sensor1 is published to broker 2 on topic b1/house/sensor1.
Maybe you should also change the image to say “Bridge all topics and use prefixes appropriately on message I/O to a bridged broker” instead of “Bridge all topics and use a remote prefix b2/ and no local prefix”
Maybe you should say that it would then be functionally equivalent to
topic # both 0 b1/ b2/
Tks for pointing that out I’ve change the diagram so there is no local prefix and that fits the text.
The comment
When Client1 publishes on any topic that message will be sent to broker 2 with the topic prefix b1/. Example incoming client message to B1 on topic house/sensor1 is published to broker 2 on topic b1/house/sensor1.
isn’t quite true as the client needs to publish on b1/house/sensor1 and it is sent to b2 as house/sensor1. So a subscriber on broker B would see it on house/sensor1.
I’ve been meaning to revisit this tutorial for a while and will go through it again and refresh my understanding and hopefully make the tutorial clearer when it comes to remapping.
Please let me know if you find anything else that seems not to make sense.
Rgds
Steve
Hello Steve,
I want to send message using mqtt in linux from one computer to another which are connected on same network.
how can we perform?
Take a look at the python tutorials . You can also use the mosquitto_pub and sub tools
http://www.steves-internet-guide.com/mosquitto_pub-sub-clients/
rgds
steve
Great tutorial Steve. One question. I have the bridge setup in mosquitto (primary) bridging with my test EMQX broker with:
topic # out 0
topic # in 0
I can see all new messages being delivered from mosquitto to EMQX, however when I publish a test message to my EMQX broker it doesn’t make it back to mosquitto. Looking at the EMQX UI I see that the mosquitto connection is not subscribed to anything. Any idea what could be wrong? I assumed the line “topic # in 0” would have handled that.
The topic in #
only controls the topics over the connection. The subscribe is done automatically. If you start mosquitto from the command line you should see it subscribe.
rgds
steve
Thanks. That’s the odd part. It’s not subscribed to any topic. Not sure if you are familiar with EMQX but you can look at all connections and see what topics they are subscribed to. I see the mosquitto bridge connection but no subscription. Which I can confirm as any messages sent to any topic in EMQX don’t make it to mosquitto.
can you post the mosqquitto.conf file
connection mqttbridge
address 192.168.x.x:1883
bridge_protocol_version mqttv50
remote_clientid ha-mosquitto
remote_username xxxxxx
remote_password xxxxxxxxx
topic # out 0
topic # in 0
I’ve sent you an email
Rgds
Steve
Hello,
Thank you for this article. Is it possible for the remote_password to use a password file instead?
The remote password is the password on the remote broker which uses the standard password file. The remote broker sees the bridge as just another client.
Does that make sense?
Rgds
Steve
Hi Steve.
Thanks for the great article.
Please help me to solve this problem. It is necessary that the system has 2 equivalent brokers with the same data set, and one of them at one time can be the master to which all devices listen. But if the master fails, then the second broker must have all the relevant information and completely replace it, and all clients must switch to it.
It turns out that everything that goes to the master should immediately go to the slave. And the master with the slave, as you understand, can change for certain reasons.
Thanks
John
John
There is no master slave in the setup described in the tutorial.
One broker is the bridge and the other is just any broker with no special setup.
With the setup in the tutorial if the bridge fails then there is no switching.
Do you have a scenario in mind?
Rgds
Steve
Hi Steve,
Thanks for the explanation, very enlightening.
I am facing an issue which is as follows:
We are serving a customer who already has a central MQTT broker configured, and who receives information about the production of various remote equipment through it. These devices cannot persist the information in the event of an internet outage.
So I came up with the idea of configuring a broker as a bridge at each of these remote points, to store messages from local clients and send them to the central broker as soon as the internet is available. I ask if my concept is correct and if I can do that, because in the tests I did, the central broker receives messages while there is an internet connection, but the bridge ends up not sending messages when the connection drops and returns later. Below is my configuration:
listener 1884 0.0.0.0
allow_anonymous true
persistence true
persistence_location /var/lib/mosquitto/
persistence_file mosquitto.db
autosave_interval 1
autosave_on_changes true
max_queued_messages 0
include_dir /etc/mosquitto/conf.d
connection central
address 192.168.0.176:1883
topic per/# out
keepalive_interval 5
restart_timeout 5 10
cleansession false
remote_username central
remote_password 12345
Try changing
topic per/# out
to
topic per/# out 1
and retest. You might also want to consider a max for the queued messages.
rgds
steve
Thank you for your quick response.
So I did that and I kept losing messages so I added the following to the config file:
remote_clientid per
topic per/# both 2
And now it seems to be running fine.
Yes i will consider a max of the queued messages, i will consider this when the application is in use so I can specify an acceptable value.
Now i loose only the last message that was send to the bridge, is there any change on config to avoid this?
It should work with qos of 1. 2 is best avoided if possible. Not sure why you should loose last message are you sure it is not a client issue.
For a client if it looses the broker connection it will start discarding new messages until the connection returns regardless of QOS.
If you still can’t get it working then send me a sketch of your setup using ask-steve page to contact me and I will try and simulate it.
Rgds
Steve
Great guide.
I am in need of a concentrator application that works sort of like a bridge but with multiple brokers in such a way that a client connect to the parent broker and can read and write to the children brokers using prefixes in the parent broker, there are some proprietary solutions for this but they are resource intensive and costly, is there any chance that mosquitto will support a concentrator setup by expanding the bridge functionality?
I think you can do this with a bridge.
the children brokers are bridges and connect to the broker and subscribe to the prefix topic.
Anything published to the parent with a child prefix gets sent to the child broker.
Rgds
Steve
Hi Steve,
this might be a very dumb question as I’m still a MQTT noob.
If got it right, remapping on the same broker is not possible at this time?
I would like to map an incoming topic to a different topic on the same broker, like:
shellies/PV/PVSchuppen/# –>> Photovoltaic/Schuppen/
Any way to do it? Preferable without having the “shellies/” directory on the broker.
I need this because it seems that there is no way to define the “shellies” prefix.
Regards & sorry if I miss the unmissable…
Jürgen
The broker can’t do that but if you take a look at the python bridge project you can do it with a client
http://www.steves-internet-guide.com/python-mqtt-bridge-project/
By the way you could bridge
Photovoltaic/Schuppen/ to shellies/Photovoltaic/Schuppen
that is you can add a prefix but not remove one.
rgds
steve
Hy there
I am facing a problem where the plan is to have one mqtt main broker cluster, surrounded and connected to many (>100) mosquitto brige brokers. The setup allows to publish/subscribe messages from a clients on one of those many bridge brokers to another client on another bridge broker. The bridge brokers still work for their connected clients even if there is no connection to the main mqtt broker cluster (which is a benefit, and the transfer is faster and more reliable as well) . The problem now is that the main broker cluster forwards the shared topics to all bridge brokers, even if there is no single subscriber. The amount of data distributed increases with the number of bridge brokers, an this unneeded sending needs to much resources. It seems that a bridge broker does not forward the information of its subscribers to the main broker cluster, which in turn would know to which specific bridge brokers to send the shared topics to. Does anyone has experience or tips for such a setup?
Thank you in advance.
Hi
Even though there is no client subscribed to the bridge broker the bridge broker is a client subscribed to the central broker so is should be send messages on that topic. It doesn’t know what clients are subscribed to the bridge broker.
Rgds
Steve
Hi,
I came across with an issue using mosquitto with QoS = 1 that I want to share with you. I am using a Rpi with the mosquitto broker bridge to AWS IoT Core and a couple of local clients running in the Rpi. The bridge is configured not to clean session and the topics with QoS =1. Everything works fine when internet drops happen. I tried to disconnect the Rpi for 3 minutes and when it connects again to internet, I receive on AWS all the messages published during these 3 minutes. I tried also disconnect de Rpi for 3 minutes and reboot before connecting again (to force the broker to be restarted) and works fine.
However, I lost the messages published when the Rpi boots, mosquitto broker starts but internet is not yet available to connect the bridge. In this time (a few seconds), the messages published from local clients are never received in AWS. It is strange, because the local clients do not have this behavior, if one of them is not connected to mosquitto at start, no matter when it will be connected that it will receive any messages on the subscription topic if any message was published when it was not connected.
It is the normal behavior?
Thank you!
Not quite sure I have understood the problem.When are the messages being published that you don’t receive.
Is it while the pi is rebooting?
Rgds
Steve
I’ve been struggling with a problem for a while now and I can’t figure it out.
If you can help me or anyone else, I owe you.
My example:
I have 2 clients for whom I can’t change the topic in any way
They transmit the data to the same broker.
Customer1 sends:
Inverter / Name: Jhon
Client2 sends:
Inverter / Name: Doe
And so is the rest of the data.
I would like to put the client’s name in front of the Inverter topic so that I can differentiate and I will not succeed.
I’m still tormented by mosquitto.conf dar ….
If you have an example … that would be great.
Thanks in advance for your help
is this data going to be sent over a bridged connection?
are the current topics
inverter/jhon
inverter/Doe
or
inverter/Name:Jhon
etc
and do you want
customer1/inverter/Jhon
rgds
steve
Hi Steve,
Thanks a lot for this helpful guide. I have successfully bridged my local Mosquitto broker to a remote Flespi broker. However I have a strange issue whereby it seems to ignore the topic line in my .conf. I have verified this by putting “topic FAKETOPIC/# in” in my bridge config, however using mosquitto_sub pointed at my local broker, I can still subscribe to real, active topics from the remote
Am I misunderstanding how the “topic” line works in the bridge config? My test seems contrary to your Example 2 in the Topic Bridging section of the article, which suggests that my local client would only be able to receive messages from the FAKETOPIC/# topic
Many thanks
Yes the remote subscribe should only get messages on faketopic can you send me the conf file
rgds
steve
Thanks for the prompt reply. My config is as follow:
—————————————————————————-
# allow remote connections
allow_anonymous true
listener 1883
# bridge to remote broker
connection REMOTE
address mqtt.flespi.io:8883
remote_username
bridge_cafile .pem
bridge_tls_version tlsv1.3
bridge_insecure false
try_private false
topic FAKETOPIC/# in
—————————————————————————-
Here are some of the lines from when I turn the broker on. You can see it subscribes to FAKETOPIC/#, however the subsequent line shows a message on the ‘flespi/state/mqtt/sessions/2953846/cid’ topic.
—————————————————————————-
1644499649: mosquitto version 2.0.14 running
1644499649: Received CONNACK on connection local.DPVW-LOCAL-01.REMOTE.
1644499649: Bridge local.DPVW-LOCAL-01.REMOTEsending SUBSCRIBE (Mid: 2, Topic: FAKETOPIC/#, QoS: 0, Options: 0x00)
1644499649: Received PUBLISH from local.DPVW-LOCAL-01.REMOTE(d0, q0, r0, m0, ‘flespi/state/mqtt/sessions/2953846/cid’, … (7 bytes))
1644499649: Received PUBLISH from local.DPVW-LOCAL-01.REMOTE(d0, q0, r0, m0, ‘flespi/state/mqtt/sessions/2953846/clean’, … (5 bytes))
1644499649: Received PUBLISH from local.DPVW-LOCAL-01.REMOTE(d0, q0, r0, m0, ‘flespi/state/mqtt/sessions/2953846/client_id’, … (25 bytes))
—————————————————————————-
In a rush at the moment but try
topic FAKETOPIC/# in 0
I seem to remember that is needs a qos value
Let me know what happens and I’ll look at it later if need be
rgds
steve
Yep I’ve given that a go but I’m still seeing the same issue unfortunately. Subscribing directly to a specific topic from the Flespi broker behaves as expected, so the issue seems to be specifically in the mosquitto bridge functionality.
I’ve then tried setting up a second local broker bridged to the first one (mosquitto to mosquitto) and interestingly that seems to behave as expected, so it appears that the problem lies in bridging mosquitto to Flespi. It’s not the most elegant workaround but it at least works, I’ll post on here if I find a better solution in case anyone else runs into the issue
Cheers,
Adam
Tks for letting me know.
Rgds
Steve
Hi Steve, and congrats for the great explanation on bridging !
I Tried my mosquitto bridge (B1) with AzurHubIot (B2) and it fails, but still , connection and subscription seems OK
Bridge local.device doing local SUBSCRIBE on topic devices/device/messages/events/
Connecting bridge (step 1) bridgetesting (IoTHubPoCGilles.azure-devices.net:8883)
Connecting bridge (step 2) bridgetesting (IoTHubPoCGilles.azure-devices.net:8883)
Bridge mybridge sending CONNECT
Received CONNACK on connection local.device
Bridge local.device sending SUBSCRIBE (Mid: 6, Topic: devices/extracotest/messages/events/, QoS: 0, Options: 0x00)
Received SUBACK from local.device
Client local.device closed its connection. => Can’t figure out why
I have a standard MQTT client C1 to subscribe to listner B1, OK
I have another python MQTT client C2 to publish M2 to AZUR B2
But I never catch M2 from C1
did you tried this bridging with AZUR HUB ioT once ?
No I haven’t.
Hi Steve.
Great website you have made and very useful indeed
I have an issue with bridging whicgh I cannot seem to get to the bottom of and perhaps you may be able to shed some light as to what might be wrong
I have a local MQTT mosquitto broker(V1.5.7) in my home network (currently not password protected whilst I am testing) This runs on a Raspberry Pi
I have a remote Mosquitto broker (V2.0.11) on a Digital Ocean droplet.
I seem unable to bridge from my local mqtt broker to the remote one
In local mosquitto log I see
=== start ===============
Bridge DigitalOcean sending CONNECT
1641937030: Received CONNACK on connection local.DigitalOcean.
1641937030: Connection Refused: not authorised
1641937030: Socket error on client local.DigitalOcean, disconnecting.
=== end ===============
In remote mosquitto log I see
=== start ===============
1641937216: New connection from 2.127.226.147:34904 on port 1883.
1641937216: Client disconnected, not authorised.
=== end ===============
My local mosquitto config is as follows
=== start ===============
log_type all
listener 1883
protocol mqtt
#listener 9001
#protocol websockets
# bridge to Hive MQ cloud
connection DigitalOcean
address 68.183.33.88:1883
try_private false
cleansession false
topic mysticWeb/# out 0
topic mysticWeb/# in 0
#bridge_protocol_version mqttv311
remote_clientid DigitalOcean
remote_username MysticIoT
remote_password ‘xxxxxxxx’
=== end ================
My remote mosquitto config is as follows
=== start ===============
listener 1883
protocol mqtt
allow_anonymous false
password_file /etc/mosquitto/passwd
=== end =================
From any mqtt client app , I can happily connect to the remote mosquitto broker
From the SSH putty terminal on the RPI I can do the mosquitto-pub command and that also works quite well
In all cases I can see the mqtt messages appearing.
The bridge connection is the only that fails and I cannot see what is causing this
I have doubled checked that I am using the correct username and password
Thanks in advance Steve
I must admit the files look ok. Have you tried using anonymous access? If you give the username/password details I will try it use the ask steve page to send them to me.
I stephan
regarding you rline :
remote_password ‘xxxxxxxx’
I think you should NOT quote (I had the same with my AZUR HUB IOT) => mosquitto_sub needs quote because on shell command, but not in the conf file
So it was the password mistake that was stopping you connecting? Is that correct? If so I’m sorry for the error and will correct it.
Rgds
Steve
How do i retrieve the bridge connection once it has been disconnected without using restart.
It should reconnect automatically. If not check your config file.
Rgds
Steve
Trying to get my apartment server to pull in Owntracks data from the broker I have running at home. The address is correct, I can connect to it with MQTT Explorer, and I see the server resolving the domain, but Mosquitto gives me this error: 0: Warning: Error resolving bridge address: Try again.
Any reason why this would be?
connection bridge-01
address home.example.com:8883
try_private true
cleansession true
topic owntracks/# in 0
remote_clientid broker1
remote_username owntracks
remote_password ‘xxx’
it looks like this is your problem home.example.com:8883.
Use the IP address instead.
Rgds
Steve
Hi! How are you?
Greetings from Brazil. I always follow your posts (I’m learning a lot).
I managed to make my bridge with your tutorial between two RPI, but I have the following problem:
RPI2 bridges to RPI1.
RPI1 is reading messages from RPI2 but cannot write, change switch on RPI (tasmota+esphome).
messages are sent for pi1 to pi2 but not from 2 to 1 Is that correct.
What does you config file look like
rgds
Steve
Hi steve, do you know if this option of making a bridge between serveral Mosquitto instances can distribute the load and increase the currently conections? I have read some posts and comments in some web pages talking about this posibility but I want to know from and expert.
Thanks for you time and have a nice day.
Yes you can have multiple instances and bridge between them. Take a look at load balancing
https://www.nginx.com/blog/nginx-plus-iot-load-balancing-mqtt/
I’m afraid that my expertise doesn’t go that far as I don’t have a need for an high load broker.
Rgds
Steve
good morning sir,
can i connect multiple brokers to make as a cluster
There are clusters but I have never created one and I’m not sure what broker they use.
Rgds
Steve
Hello,
I did a MQTT bridge using QOS 2 to mirror the message between a local and a remote broker. My idea is to not loose local messages even with internet failures. But, with 10 minutes of disconnection, the bridge sends 20 to 40 old messages and then jumps to the current one.
Do you have any idea why this happens? I was expecting that the bridge will send all the “lost” messages because the bridge is configures with QOS 2.
Hi
There is a limit to how many messages it will keep which you can change in the config file. The idea is to cater for a network glitch and not a prolonged outage.
If you want to keep all messages then I would use a client to capture them and forward them. Using a client you can store them to disk and then forward them on.
rgds
steve
Whaouh, thank you for the details !
Have you already tried to setup a bridge between a local mosquitto broker and AWS IoT Core ? 🙂
Sorry but No I haven’t.
Rgds
Steve
Hi
Have you ever setup a bridge between a locally installed Mosquito instance and the cloud HiveMQ free service?
I just cant get it to connect – some cert related error I just cannot figure out.
Thanks
Have you tried with using ssl?
Yes, just cant get it to connect – using a cert sourced from: openssl s_client -connect :8883 -showcerts /dev/null | sed -n ‘/BEGIN/,/END/p’ > server.pem
Am at point of give up and use beebotte, which works perfectly…..
It may be that it doesn’t support SSL. Are you just using this as a test or do you plan using it for real. If you want to use it for real I would go for beebotte or cloudmqtt as they are commercial services. The hive test broker might be taken down at any time.
I will do some tests on it anyway and let you know.
Rgds
Steve
Hi,
thank you very much for this helpful tutorial.
I have a problem. In my case, if the main broker go down and after several seconds goes up, the client broker does not reconnect.
Example: I have B1 and B2. B2 is connected to B1. If I stop B1 for 30/40 seconds and then I restart it, B2 doesn’t reconnect. I have this messages:
B1:
New connection from IP:44300 on port 1883.
Client closed its connection.
B2
Connecting bridge br-me-to-broker0 (IP:1883)
Connecting bridge br-me-to-broker0 (IP:1883)
Connecting bridge br-me-to-broker0 (IP:1883)
….
Could you help me?
thank you very much!
I think that B1 is remembering the connection and thinks it has a duplicate client. If that is the case you should see b2 showing connect and reconnect.
Try to configure the broker connection to use a random client name.
Rgds
Steve
Hello Steve,
Thank you for reply.
I have deleted “remote_clientid” from mosquitto.conf. This is my Bridge config:
# BRIDGES
connection br-me-to-broker0
address IP:1884
topic # both 0
cleansession false
notifications false
#remote_clientid broker0
start_type automatic
When I start B1 and then B2, I have:
B1:
1620195224: New connection from IPB2:54362 on port 1884.
1620195224: New bridge connected from IPB2:54362 as broker0 (p2, c0, k60).
B2
1620195224: Connecting bridge br-me-to-broker0 (IPB1:1884)
1620195224: mosquitto version 2.0.10 running
if I stop and restart B1:
B1:
1620195561: New connection from IPB2:50568 on port 1884.
1620195621: Client closed its connection.
1620195652: New connection from IPB2:50586 on port 1884.
1620195712: Client closed its connection.
B2:
1620196016: Connecting bridge br-me-to-broker0 (IPB1:1884)
1620196094: Connecting bridge br-me-to-broker0 (IPB1:1884)
1620196172: Connecting bridge br-me-to-broker0 (IPB1:1884)
In your opinion there is a bridge reconnection problem in mosquitto or I have a problem in my configuration?
Thank you very much!
Rgds
Dado
Sorry Steve,
I forgot that if I restart B2 manually, B2 reconnects to B1 correctly.
Thank you
Dado
It is I think a client id problem as the broker still thinks the client is connected. Try setting clean sessions to true in the config . I will take another look at possible bridge settings.
Rgds
Steve
I tried to setting cleansession to true in the B2 config, but I had same result.
Is there anything that I should add on B1?
Thank you very much!
What version of mosquitto are you running on both brokers.Try adding the try private option to the config file on b2. If that doesn’t work try changing the protocol version.
bridge_protocol_version version
Set the version of the MQTT protocol to use with for this bridge. Can be one of mqttv50, mqttv311 or mqttv31. Defaults to mqttv311.
I’m using mosquitto 2.0.10 for both brokers.
I tried to add try_private on true and false but I hadn’t success.
I tried to change bridge_protocol_version but doesn’t work. All the change I made on B2.
B2 config:
allow_anonymous true
# BRIDGES
connection br-me-to-broker0
address B2IP:1884
#try_private true
topic # both 0
cleansession true
notifications false
#remote_clientid broker0
start_type automatic
bridge_protocol_version mqttv31
Thanks for your patience!
Try setting persistence to false. I thought I saw something new in the conf file for duplicate cleint_ids but can’t locate it at the moment.
Rgds
Steve
Hello Steve,
First, Thank you for this article, it was very helpful. I have one question:
I am using Linux-CentOS 7, I built the mosquitto-2.0.10.tar.gz (by make and make install).
The mosquitto executable created in /usr/local/sbin. (I run the mosquitto with the command line).
When I run broker on one machine and broker configures as bridge on other machine, there is successful connection. However, I cannot run broker and broker configures as bridge on the same machine. When I am trying to do so I get “Address already in use”.
I succeed to run broker and broker configures as bridge on the same machine on Windows, so I know this is possible.
Can you help me? If I am not mistake this is a configuration issue.
Thank you,
Maayan
You need to use a different ports use 1883 on broker1 and 1884 on broker 2
I correct my self –
I did not succeed to run broker and broker configures as bridge on the same machine on Windows, so I do not know if this is possible.
Is that possible?
It should be. I will check as I haven’t done it myself. Send me a reminder in a few days if I haven’t replied
rgds
steve
Hi Steve,
is it possible to bridge or map an incoming topic to another one?
Following situation:
I am using Home Assistant with Mosquito Broker and a doorstation sends mqtt messages to the broker.
Home Assistant has an mqtt-autodiscovery but only listens on special topic-syntax.
I want to receive the message vom the doorstation and want to do some mapping to the special autodiscovery syntax.
After that all relevant topics from doorstation should autocreate things in Home Assistant.
Is this possible with Mosquito? Or can I map only topics to other brokers?
No it isn’t possible except on a bridge. You need a another client to do that for you by subscribing to one topic and publishing on another.
I made a start on it using python and node-red but never finished it but it is possible to do.
Rgds
steve
Hi Steve,
In my scenario of having multiple clients and 2 brokers (1 main and 1 backup).
In order to achieve this scenario of having a backup broker and answering questions like what happens when a the main broker fails,
should the clients re-subscribe to the topics in the second, etc.,
does bridging come into the picture at all or if it doesn’t can you point me in the right direction?
Regards,
Ray
A bridge is no help to you here as that becomes a single point of failure.
You can connect the client to two brokers and only publish on one.As soon as you detect a failure then you publish on the backup.
You will need to subscribe on both brokers.
Rgds
Steve
Thanks a lot Steve. Would you also point me to a reference site for more specific details?
I tried finding one, but failed to do so.
Thanks,
Ray
Details on bridges or dual connections? Are you using python?
The goal is to increase the availability (so that if one broker fails, I want all the clients to automatically use the back up broker.
I’m using JavaScript (MQTT.js library).
1. I was also wondering in the first place if there was a way to transfer all the state (details including subscriptions, topics, etc.) of one broker to another broker
2. If the main broker with an ip:port is down, can the backup broker take the main broker’s place (ip:port) ? So that the clients don’t have to re-subscribe etc.
No they would need to resubscribe.
Rgds
Steve
Hi Steve,
Thank you for the very informative material provided on this website. I have the following use case: I’ve got a home equipped with a heating system, which can be monitored and managed remotely (set on/off, set some parameters) through an online interface provided by the manufacturer. I’d like to integrate the data collected/published by the heating system into a more comprehensive and customized dashboard (Grafana based for instance). Assuming the heater do communicate via MQTT with a manufacturer’s cloud broker AND I can capture and “decode” this MQTT traffic (which I plan to investigate next time I go on site), what would be the best approach to collect the data published by the heater without compromising its MQTT interactions with the manufacturer’s broker? Based on your article, I’m thinking of setting up a local broker as a bridge. As I couldn’t change anything on the remote broker side, I would like to have my local bridge behave like a pure pass-though, so that it is perceived as the heater’s IOT from the remote broker. Does this make sense? I apologize for any shortcomings, my interest in and knowledge of MQTT and IOTs is very new. Thanks
The way MQTT works is ideal for sending data to multiple sources. By adding your client to receive data doesn’t affect the data going to the manufacturer.Think of it like TV. If you switch on your TV and tune to a channel it has not affect on the other people already watching their TV on the same channel.
Use a client not broker to collect the data.
Rgds
Steve
Hi Steve,
Really grateful for your articles on the usage of MQTT. I’ve hopefully a simple question I’m hoping you can answer to check my understanding before I spend fruitless hours trying to get something to work.
I’ve a situation where I have a number of sensors distributed into a set of discreet locations. These are primarily responsible for publishing data (don’t really need to subscribe). Their connection to the broker, which is used to collect and store the messages in a database, however cannot be guaranteed as the connection isn’t totally reliable. It’s important to not loose any of the published messages ie when connection is re-established I’d like to put into the database all these messages that failed to send.
Would I be correct in thinking that I can set up a situation as follows using the functionality of bridging.
Sensors 1,2&3 in location A publish from their client to a broker A with QoS set to 1 or 2 to enable queuing.
Broker B is bridged with broker A with client C subscribing to the topics off of broker B.
If connection between broker A and B is lost, the messages will queue (default 100 in mosquito but can be changed) on broker A. Once connection is re-established the x number of queued messages would then be delivered to broker B, where client C could then read them out and store them into the database.
Hope that makes sense and thanks for any clarity/advice you can give
Hi
Depending on the sensor and sensor code I would look at queuing messages locally on the client.
If this can’t be changed then I would send them to a local broker and bridge that broker to a central one. the bridge would queue the messages.
You are correct that you need qos of 1 or2 for it to work.
Rgds
Steve
Hi Steve!
Great article! it helped me to set up a “cluster” with 3 front and one back broker who is synchronizing data between the other three; they all are behind a HAproxy pool and it works like a charm! 😉 thanks!!
Now I have a few doubts
– if I shutdown the back broker (a reboot, for example), the synchronization stops; when the service gets up again the synchronization starts as well but the messages are not resend from the front cluster. So those messages are not synchronized. Is there any way to get them?
– As far as you know, is there any way to set a mosquitto cluster with high availability? I have seen other MQTT Brokers like EMQX, Verne or Hive that can do that (I have to say that I hadn’t tried any of them, just saw that feature on their webs), but the further I got with Mosquitto is this this design.
Looking forward to get your answer
Happy new Year!
Esther
Hi
You are ahead of me there as I’ve never set up a cluster. However the messages will only be held for sending if the qos of the bridge is 1 or 2 and the messages arriving on the front end brokers is 1 or 2.
You can set the front end brokers to store x messages the default is 100 I believe but you can check the manual
Hope that helps.
The QoS is already 2 in all of them, so I will check the storage on the front end brokers
Thank you very much!
Hello,
I want to make a bridge on broker1 to broker2 so when I publish something with client on b2, every client thats subscribed to this topic on b1 gets the message.
My configuration on b1 is:
#connection
connection u0_bridge
address it.pharaon.iotool.io
topic # out 0
topic # in 0
When I run mosquitto on b1 or b2 with “mosquitto -v”, I can connect to either of them with any client and publish and subscribe to topics normally, but the bridge won’t connect. If i run b2 with same command and run b1 with “mosquitto -c /etc/mosquitto/mosquitto.conf” so that configuration is loaded, the bridge connects to b2 and I can still normally connect with clients on b2, but I can’t connect clients to b1. If I try, I get error message:
“Client connection from 178.58.166.83 failed: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknwon protocol.”
“New connection from 178.58.166.83 on port 1883.”
“OpenSSL Error[0]: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO: unknown protocol.”
“Socket error on client , disconnecting.”
And I can’t connect to it. What am I doing wrong?
Regards
It looks like you need to run a listener on port 1883. Not sure of the SSL are you also trying to use encryption. If so disable it and get the bridge to work without it and add it later.
You were right, there were some certificate files that blocked me from connecting to the broker, I disabled them. Now I can connect to both brokers, but I can’t publish to b1 (the one with a bridge). There are no topics opened on this broker (not even $SYS, which was otherwise already opened on all the other clients) and if I try to publish a topic, I get an error message “Denied PUBLISH from *my userID*”.
Hi
It looks like you have an ACL that is blocking take a look at the config file for that broker
I solved the problem with publishing, it was the ACL file that didn’t allow me to publish. Now the bridge works, if I send the message on one broker I get it to the client on the other broker.
Now I have another problem. If I start b2 with “mousquitto -v”, that works as described above, but if I want to load conifugration with “mosquitto -c …”, I can’t connect to b2 either with b1 or clients. I get error “Socket error on client , disconnecting.”
That sounds like you are trying to connect with a client of the same name
Can you please share any working example of shared topics in bridge configuration.
So that would be to have two brokers bridged to one single bridge on shared topic so the bridge could load balance.
We are trying to bridge our cluster of two mosquitto brokers (broker1 and broker2) with one bridge (broker3), and we want them to be subscribed to shared topic so the messages could be load balanced.
Is there a solution for this through mosquitto ?
Regards,
John
Not done that I will give it some thought. I assume broker 1 and 2 are subscribing to a shred topic on broker 3? is that correct?
Hello,
Correct.
We have tried several options and only one that works is when we set
$share/group/service/a in 0
When we try any combination with local/remote prefix we are unable to get any message flow.
Hi
That makes sense as the $share is a special $share and you almost certainly can’t prefix it.
Okey thank you , so conclusion is that we can’t have local/remote prefixes with bridge shared subscription ?
That is what I believe.
Hi John,
Were you able to figure out a solution to this? If not, what other approach can we take to achieve shared subscription on bridge configuration.
Dear Steve,
I have configured two brokers in two virtual machine instance in the Google cloud. I want to set up a bridge between these two brokers. But the setup is not working . Can you please help me on reasons for not working.
With cloud instances I would always check firewalls as often they are set up by default.
rgds
steve
Hi,
Is a bridge configuration valid if the topic is missing? Why?
Thank you.
Hi
Can you post your bridge conf file so I can see what you mean
Hi Steve,
I have changed my cloud broker from hiveMq to mosquitto and by doing so my problem has been resolved but now my local broker (bridge) does not receive any messages from the cloud Broker and vice versa.
Furthermore, when the client connected to the local broker publishes then on cmd i can see the message “publishing to cloud MQTT broker” but i did not receive that message on the client that is connected to cloud broker and has a subscription for that topic.
Hi
Post the entire configuration file and I will try it
Hi Steve,
I have followed the configurations as you instructed in this tutorial but i could not receive messages from subscribed topics on a public broker?
My Config file setting:
connection bridge-to-HIVEMQ
address broker.hivemq.com:1883
topic automation/station both 0 “” “”
topic sensors/octocopter01/altitude both 0 “” “”
I also run the command that you mentioned and got the following error:
“1603189003: Error: Only one usage of each socket address (protocol/network address/port) is normally permitted.”
Hi
Sorry what command do you run?
Does the bridge start up ok?
Do you see messages being sent to hivemq?
Rgds
Steve
Hi,
I have tried to connect two brokers with a bridge. so i gave ,in the config file of broker 1 ,
connection bridge-02
address 192.168.23.24:1883
topic # out 0
topic # in 0
So is the name bridge-02 given after the “connection” is ip address or host / domain name of broker 2? Do i need to specify that name anywhere in the config file of broker 2? What is that name actually? I am new to mqtt so please answer this question.
No broker 2 sees broker 1 as a client.
Rgds
Steve
sir steve, if i want to use arduino to act as client to subscribe and publish messages
should i install mosquitto broker in it or other broker, or what i could do please
You could use a cloud broker. The advantage of using your own local broker is that it is easier to troubleshoot as you can see the broker console.
Rgds
Steve
hi sir i’m just stuck with mqtt and usage of mosquitto broker, do i need to establish the code for clients and broker in the server means in production level, or i could just write the code for clients and connect it to cloud broker, also what’s the need of mosquitto
Yes you can use local clients and a cloud broker.
Rgds
Steve
Hi Steve,
First of all thank you for your informative homepage.
I have a problem receiving messages when configured bridges cannot be created.
To the setup.
I have a local broker in the production network and bridged the one to several brokers in the plant network.
The configuration on the local broker:
# bridge to plant1
connection bridge-172.16.35.11
address 172.16.35.11:1883
topic # in 2 “” “”
topic 172.16.35.11/# out 2 “” “”
bridge_protocol_version mqttv311
notifications true
cleansession true
restart_timeout 5 30
max_inflight_messages 0
round_robin true
# bridge to plant2
connection bridge-172.16.35.13
address 172.16.35.13:1883
topic # in 2 “” “”
topic 172.16.35.13/# out 2 “” “”
bridge_protocol_version mqttv311
notifications true
cleansession true
restart_timeout 5 30
max_inflight_messages 0
round_robin true
# bridge to plant3
connection bridge-172.16.35.14
address 172.16.35.14:1883
topic # in 2 “” “”
topic 172.16.35.14/# out 2 “” “”
notifications true
bridge_protocol_version mqttv311
cleansession true
restart_timeout 5 30
max_inflight_messages 0
round_robin true
No special configuration is made on the bridged brokers.
The bridges can be created successfully if all brokers running and I can publish messages back and forth.
Well my problem is, if e.g. Plant 1 and Plant 2 are offline, the local broker can’t create a bridge. This is clear. Plant 3 is online, the bridge is successfully created. If I now publish messages from Plant 3, e.g. via MQTT-Explorer, the messages arrive on the local broker with a delay. I have noticed that the delay is caused by the broker while trying to create the bridge connection on Plant1 and Plant2. If the creation of the bridge fails, the messages are received from plant 3 (MQTT-Explorer connected to local broker).
I’m running MQTT Broker version 1.6.9 as a service on Windows 10 (local and remote)
Can you help me?
Best regards,
chris
Hi
Just to clarify the fail condition.
When you publish from plant 3 with 1and2 offline the message doesn’t appear on the local broker until the connection timeout to plants1 and 2 has expired? Is this correct?
What is the delay time?
How about messages going to plant3?
Rgds
Steve
Hi,
The message from Plant 3 doesn’t appear on the local broker while local broker trying to connect to a offline bridge. (plant1)
2020-05-04T08:46:47: Connecting bridge bridge-172.16.35.11 (172.16.35.11:1883)
2020-05-04T08:47:08: Error creating bridge: Unknown error.
After the message “error creating bridge”, the message from plant3 appears on local broker.
Trying to connecting bridge takes about 21 seconds.
There is no other delay time configured. Only restart_timeout 5 30.
Same Problem with messages going to plant3.
many thanks!
Chris
Chris
It looks like the broker is waiting to connection timeout before processing traffic. I need to set it up and test it can you use the ask steve page and send me a quick sketch of the setup and you config files.
Rgds
Steve
I need to configure a mosquitto broker as mqtt bridge where the remote broker has many clients that subscribe to same topic
but the messages are mapped to the respective clients with the help of username and password authentication of the end-point(or subscriber client).
Is there any way to route the mqtt-packets to respective clients of remote broker using several passwords and usernames along each mqtt-packets?
There is a manual way to do this using MQTT API, but ‘mqtt-bridge’ seems easier and automatic to forward messages between two brokers.
The following are the MQTT API for publishing and subscribing the request and responses!
for client1:
mosquitto_pub -h “HOST_URL” -t “REQUEST_TOPIC” -f “REQUEST_MESSAGE” -u USER_NAME1 –
P PASSWORD1
mosquitto_sub -h “HOST_URL” -t “RESPONSE_TOPIC” -u USER_NAME1 –P PASSWORD1
for client2:
mosquitto_pub -h “HOST_URL” -t “REQUEST_TOPIC” -f “REQUEST_MESSAGE” -u USER_NAME2 –
P PASSWORD2
mosquitto_sub -h “HOST_URL” -t “RESPONSE_TOPIC” -u USER_NAME2 –P PASSWORD2
for client3:
………
and so on.
or is there a way to publish same mqtt message on two different brokers (hosts) with same topic?
or any other solutions/suggestions for this problem would be greatly appreaciated.
Looking forward to your kind response.
Regards,
Thangz
Not really sure what you are trying to do.
Messages are routed to clients based on the topic they subscribe to and not the username/password.
When you configure the bridge it is configured to send messages to the remote broker based on topic not username/password.
If you use a python client you can easily send the same message to two brokers.
rgds
Steve
Hi ,
I am not sure I go to the wrong topic or not. I just want to find a solution that can as a client then receive topic data and then restructure it and publish to another MQTT server. Is it the feature of the bridge? Or other solution can fit this feature?
Hi
No the bridge doesn’t restructure it. You need a client App for that. I actually made a start on a Python client tnat does something similar. The Flight logger does this with HTPP to MQTT.
http://www.steves-internet-guide.com/republish-html-data-over-mqtt-flight-arrivals/
What language will you be using?
Rgds
Steve
Rgds
Steve
Hello,
Please have you found a solution to restructure MQTT topics because I a struggling with the same issue and I did not found any helpful solution.
Thank you in advance.
When you say restructure do you have an example
Rgds
Steve
Hi steve!
Suppose that I need to manage 50 small sensor networks. And each network has their own broker on Raspberry Pi.
I have two questions.
1. Could I connect 30 local brokers to cloud such as Amazon cloudMQTT or Google Cloud IoT Core to make one large logical sensor network?
2. What kind of benefits would I get from those cloud? Device management, Data flow, Machine learning?
Yes you could send them to cloud broker by bridging the broker.
However you may want to process data locally and only send processed data to the cloud service. This is especially true of the sensors send lots of redundant information.
Having access to all 50 networks by sending and receiving messages to 1 or 2 brokers is much easier to implement.
As for machine learning I don’t see it makes any difference as long as you get the data.
rgds
steve
Hey, Steve. Appreciate your site a lot! I’ve been going nuts with a problem, and wondering if you have any insights.
I’m trying to bridge a local (Ubuntu-hosted) Mosquitto MQTT instance with Beebotte, so that I can use Beebotte for connecting with IFTTT etc. and use my local instance for running Node-RED, Home Assistant, local control exchanges, etc. I’ve been running my local instance and connections to Beebotte for some time, and both work fine on their own (e.g., I can mosquitto_sub and mosquitto_pub to both just fine; I have apps running Paho that connect fine to both, etc.). So I’m pretty sure that there are no network connectivity, fundamental authentication, etc. issues in play.
However, when I try to bridge from Mosquitto to Beebotte, I never get a successful connection, and just cycle between (in the logs):
– Connecting Bridge (mqtt.beebotte.com:1833)
– Bridge sending CONNECT
– Socket error on client , disconnecting.
When I sniff the network traffic (tcpdump), I see the connection go through to Beebotte, with the authentication token in the header, and I can see the connection established (netstat) for 30 seconds, after which the connection is dropped.
When I compare the network traffic (tcpdump) to a successful mosquitto_sub or mosquitto_pub to Beebotte from the same machine, the only real difference I see (just eyeballing the packets) is:
– mosquitto_sub/pub appear to send a blank password (two additional bytes past the authenticating user ID); I can’t configure a blank password in the Mosquitto bridge configuration, or it throws an error)
– mosquitto_sub/pub follow up the connection with a request, whereas the bridge doesn’t.
I’ve combed the ‘Net, and while people talk about bridging to Beebotte, I haven’t been able to find any live examples, or any discussion of problems bridging.
Below is my configuration (I’ve tried every other variant I could think of, as well). Any ideas? Thanks in advance!
—-
connection beeb-from-local
address mqtt.beebotte.com:1883
remote_user
topic topicparta/topicpartb both 0 “” “”
notifications false
cleansession true
—-
Hi
Never heard of beebotte before it looks a lot like cloudmqtt. I will have to
make a note.
I don’t see the
remote_username
remote_password
options in the bridge conf
Did you also see this page
https://beebotte.com/docs/mqtt#bridge
Let me know if it helps if note I’ll take another look
Rgds
Steve
Hi Steve,
Just been using hours on finding any one whom had made a mosquitto bridge to ThingsBoard IOT service. Your guide is the closest I could find. Thank you for it.
Have you done this (bridge to ThingsBoard) successfully and what would the .conf file look like ?
Thanks in Advance, Sami
Hi
Not sure if you can as thingsboard doesn’t supply a broker. Why can’t you just use a client?
Rgds
Steve
Thanks Steve for quick reply. As relative new to MQTT and Mosquitto …. what do you refere as client?
Just back ground what i’m trying to achive. So I have several Nodemcu + DHT11/CO2 sensors around house and logging indoor /outdoor temps and humidity and CO2 levels in house. All connected via LUA scirpt directly to Thingsboard.com. Works perfectly. Then as started to look for the CO2 levels I started work on project that the CO2 levels would be going also to my HVAC unit (connetec to NodemCU and TRIAC unit) and hence the levels would actually control the unit also. Here I managed to create a system in LUA that it send data to both two broker Thingsboard and Mosquitto. I was just looking would it be easier just to send the data to mosquitto and the re-distribute it to Thingsboard and I would not need to send it into two destinations from the NodeMCU’s.
Thanks !
Are you currently using MQTT to send the data to thingsboard or MQTT?
Rgds
Steve
Not working, when bridge between emqx and mosquitto mqtt.
You need to provide more detail if you want any help.
rgds
steve
Is it possible to run both local broker and the bridge on the same server, or would I need some fancy manger like docker?
Hi
Yes you can they just need to use different ports. This may help
How to Run Multiple mosquitto brokers on the same machine
https://youtu.be/dHH1kSQLhsU
rgds
steve
Thanks Steve, It’s very informative article.
Hi, Steve, I’m beginner here. I try to make the bridge between my local broker and CloudMqtt. Without bridg’s section in mosquitto.conf it start ok:
sudo systemctl status mosquitto
● mosquitto.service – LSB: mosquitto MQTT v3.1 message broker
Loaded: loaded (/etc/init.d/mosquitto)
Active: active (running) since Wed 2019-02-20 21:17:03 +05; 1 day 12h ago
Process: 511 ExecStart=/etc/init.d/mosquitto start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/mosquitto.service
└─531 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
Feb 20 21:17:03 rsb mosquitto[511]: Starting network daemon:: mosquitto.
Feb 20 21:17:03 rsb systemd[1]: Started LSB: mosquitto MQTT v3.1 message broker.
Feb 20 21:17:03 rsb mosquitto[531]: mosquitto version 1.3.4 (build date 2018-09-28 22:21:32+0000) starting
Feb 20 21:17:03 rsb mosquitto[531]: Config loaded from /etc/mosquitto/mosquitto.conf.
Feb 20 21:17:03 rsb mosquitto[531]: Opening ipv4 listen socket on port 1883.
Feb 20 21:17:03 rsb mosquitto[531]: Opening ipv6 listen socket on port 1883.
Feb 22 08:04:29 rsb mosquitto[531]: Saving in-memory database to /var/lib/mosquitto/mosquitto.db.
And when I add next strinngs in mosquitto.conf:
connection cloudmqtt
# address m99.cloudmqtt.com:55555
# try_private false
# notifications false
# start_type automatic
# remote_username uuu
# remote_password ppp
# topic # both
then I get mosquitto stopping:
pi@rsb:~ $ sudo systemctl status mosquitto
● mosquitto.service – LSB: mosquitto MQTT v3.1 message broker
Loaded: loaded (/etc/init.d/mosquitto)
Active: active (exited) since Fri 2019-02-22 09:31:32 +05; 6s ago
Process: 933 ExecStop=/etc/init.d/mosquitto stop (code=exited, status=0/SUCCESS)
Process: 940 ExecStart=/etc/init.d/mosquitto start (code=exited, status=0/SUCCESS)
Feb 22 09:31:32 rsb mosquitto[940]: Starting network daemon:: mosquitto.
Feb 22 09:31:32 rsb systemd[1]: Started LSB: mosquitto MQTT v3.1 message broker.
Uncommenting strings above gives same result.
Help me please. Thanks!
The ping to CloudMqll presents and First the mosquitto starts The contain
hi
you need to enable the topic and address lines
rgds
steve
Thanks for the answer.
Now my mosquitto.conf contains lines:
connection mybridge
address m99.cloudmqtt.com:55555
try_private false
notifications false
start_type automatic
allow_anonymous false
remote_username nazmosq
remote_password *************
topic # both 0 “” “”
and mosquitto stopped:
pi@rsb:~ $ sudo systemctl status mosquitto
● mosquitto.service – LSB: mosquitto MQTT v3.1 message broker
Loaded: loaded (/etc/init.d/mosquitto)
Active: active (exited) since Thu 2019-02-28 23:29:02 +05; 4s ago
Process: 969 ExecStop=/etc/init.d/mosquitto stop (code=exited, status=0/SUCCESS)
Process: 976 ExecStart=/etc/init.d/mosquitto start (code=exited, status=0/SUCCESS)
Feb 28 23:29:02 rsb mosquitto[976]: Starting network daemon:: mosquitto.
Feb 28 23:29:02 rsb systemd[1]: Started LSB: mosquitto MQTT v3.1 message broker.
Using mqtt-spy for connection with cloud-mqtt with such remote user is ok
What I must do?
/var/log/mosquitto/mosquitto.log is empty
I spent so much time for bridge..:(
What version of mosquitto are you using and what OS
rgds
steve
Today I passed to remote broker iot.eclipse.org without authentification. Mosquitto.conf contains lines
connection mybridge
address iot.eclipse.org
notifications false
start_type automatic
topic test001/# both 0 “” “”
But messages from the remote broker are not filtered according to a topic test001:
1551551860: Bridge rsb.mybridge doing local SUBSCRIBE on topic “”test001/#
1551551860: Connecting bridge mybridge (iot.eclipse.org:1883)
1551551860: Bridge rsb.mybridge sending CONNECT
1551551862: Received CONNACK on connection rsb.mybridge.
1551551862: Bridge rsb.mybridge sending SUBSCRIBE (Mid: 39, Topic: “”test001/#, QoS: 0)
1551551863: Received PUBLISH from rsb.mybridge (d1, q1, r0, m69, ‘test001/hum’, … (4 bytes))
1551551863: Sending PUBACK to rsb.mybridge (Mid: 69)
1551551863: Received PUBLISH from rsb.mybridge (d1, q1, r0, m70, ‘test001/heap’, … (5 bytes))
1551551863: Sending PUBACK to rsb.mybridge (Mid: 70)
1551551863: Received PUBLISH from rsb.mybridge (d1, q1, r0, m71, ‘test001/tdht’, … (4 bytes))
1551551863: Sending PUBACK to rsb.mybridge (Mid: 71)
1551551863: Received PUBLISH from rsb.mybridge (d1, q1, r0, m72, ‘test001/heap’, … (5 bytes))
1551551863: Sending PUBACK to rsb.mybridge (Mid: 72)
1551551863: Received PUBLISH from rsb.mybridge (d1, q1, r0, m73, ‘test001/tdht’, … (4 bytes))
1551551863: Sending PUBACK to rsb.mybridge (Mid: 73)
1551551863: Received PUBLISH from rsb.mybridge (d1, q1, r0, m74, ‘test001/hum’, … (4 bytes))
1551551863: Sending PUBACK to rsb.mybridge (Mid: 74)
1551551863: Received PUBLISH from rsb.mybridge (d1, q1, r0, m75, ‘test001/heap’, … (5 bytes))
1551551863: Sending PUBACK to rsb.mybridge (Mid: 75)
1551551863: Received PUBLISH from rsb.mybridge (d0, q0, r1, m0, ‘/034L2Xe68z/000/time’, … (73 bytes))
1551551863: Received PUBLISH from rsb.mybridge (d0, q0, r0, m0, ‘GSM_FLOW’, … (73 bytes))
1551551863: Received PUBLISH from rsb.mybridge (d0, q0, r1, m0, ‘/30c216b0ed/0/motion/perc’, … (11 bytes))
1551551863: Received PUBLISH from rsb.mybridge (d0, q0, r0, m0, ‘fedor’, … (100 bytes))
So I get
Socket error on client rsb.mybridge, disconnecting.
Steve, what I must write in line “topic..” ? Thanks!
Hi
Can you just give me a brief overview of the bridge you are trying to create.Use the ask steve page and I can respond via email
rgds
steve
This my problem is solved. I confused subscribed messages and the messages which are written down in mosquitto.log. For some reason messages come to the mosquitto.log without filtration according to topic. Thanks for your attention!
Hello.
Great tutorial, very helpful.
I want to bridge all topics in to my bridge but only some out. Still then, I couldn’t find a way of using wildcards to avoid havin to specify every topic individually as out…
My config for in topics is (this works beautifully):
topic # in 0
My doubt is with the out topics, which I wanted to filter (for instance, all that contain word Sonoff or sonoff. Something like:
topic *onoff* out 0
or
topic #onoff# out 0
But it doesn’t seem to work properly.
Any ideas on if I can do what I want and what am i doing wrong?
Thanks in advance,
Nuno
hi
you cannot use wild card characters as part of a name e.g string
sonof# doesn’t work but
sonoff/#
should work
* is invalid
I tried to make a bridge with test.mosquitto.org with mosquitto version 1.4.14 on Windows and the results worked. but when I tried it with version 1.5.4 the result failed, the error statement was “unknown error”, this happened when the local broker tried to connect to the remote broker (test.mosquitto.org)
This works when I make a bridge with a remote broker on the same PC
Hi
I have had others complain of bridge problems with mosquitto 1.5.4. I haven’t had chance to test it but it is probably nothing you are doing wromg.
How to establish a TCP Bridge connection.
Trying to TCP Bridge Mosquitto with EMQ.
Included the bridge connection in my mosquitto.conf
connection emq
address 172.22.25.53:2883
topic AB/# both 2
The IP address mentioned above is an IP of a different system. And also setup the port at 2883.
But the very rarely does the connection happen. It fails 95% of the times.
Where am i making a mistake?
Difficult to say. The mosquitto conf file that you included looks ok but missing a line like
connection bridge-emq
If that doesn’t work make sure that you can connect to emq using a normal client as it might have other restriction e.g authentication configured.
Let me know how you get on.
rgds
steve
Hello This solution can use for enterprise ? Anyone use for enterprise ? George
This is a very good post on MQTT bridge. TYou have given tips Encryption which is very useful for the user. They are able to make a bridge to connect with each other.
Hi steve, can we increase client connection limit from default 1024 to 10,000 in mosquitto?
I don’t find a proper solution from internet…so, it would be great for me if you can help in this problem..
Thank you!
☺️
From what I understand you can use -1 and have unlimited connections but the docs say that you might hit problems when you go over 1024.
# The maximum number of client connections to allow. This is
# a per listener setting.
# Default is -1, which means unlimited connections.
# Note that other process limits mean that unlimited connections
# are not really possible. Typically the default maximum number of
# connections possible is around 1024.
#max_connections -1
I am using Mosquitto MQTT Bridge between local gateway & remote MQTT broker server. It is working fine but when internet connection down then all topic published locally will not publish to broker when reconnect to internet even though QOS level is set to 2 and clean session flag is false so how can I ensure high quality of level between local MQTT bridge & remote MQTT broker without making retain topics. Is there any configuration from which local MQTT bridge ensure all topic sends to remote MQTT broker as or when internet connection resume?
It is not just the QOS on the bridge. Yoy need to be publishing to the bridge with a qos of 1 or 2 and subscribing on the remote broker with a qos of 1 or 2 and it should work.
You should also note that mosquitto has a limit of 100 messages (default) setting that it will hold in a queue.
HI Steve
I added following entry in broker2 mosquitto.conf file
connection Bridge1
address 127.11.0.1:1883
topic # both 0 “” b0/
That’s broker-2 as bridge, Broker1 as normal broker
Started normal broker first(broker1) and then bridge broker (broker2) later.
Used mosquitto_pub and mosquitto_sub to verify.
Case 1: Publishing from Broker2
# broker2:~# mosquitto_pub -t chassis/PSU0 -m “PSU0 is ON”
# broker1:~# mosquitto_sub -t b0/chassis/PSU0
PSU0 is ON
Case 2: Publishing from Broker1
#broker1:~# mosquitto_pub -t chassis/PSU0 -m “PSU0 is ON”
#broker2:~# mosquitto_sub -t chassis/PSU0
PSU0 is ON
Broker2 publishes them to broker1 with a prefix of b0/
Broker2 subscribes to all messages from broker1 with no prefix, is that correct ? I thought there would be /b0 this case as well.
Please clarify
No broker2 should subscribe to broker 1 using the prefix.
When you publish a message to broker1 then any message not beginning with the prefix b0/ isn’t sent to broker2.
When you send a message to broker2 it should send it to broker1 with the prefix.
In this case the prefix is both ways in and out.
Hi
Thank you first
I have question about bridge
Assume my broker can accept 10k users
If i use bridge,my brokers can 20k users?
Regards
The bridge will connect the topics to the other broker and The other broker can accept 10k users
So i can increase connection to many users with bridge
Thanks
Yes but you need multiple brokers