Republish HTML Data Over MQTT

HTTP is the protocol that powers the Web,and is the most common protocol on the Internet.

Because http is a request/response protocol a visitor viewing a web page must issue a new request in order to refresh the data.

For most websites this isn’t a problem as the page content is static, but there are a class of sites were the information being displayed is changing, and for which http may not be the best way to deliver it.

Flight Information – The Problem

Data for Live flight arrivals is displayed on a web page and is updated on a regular basis (every 60 seconds is common).

Due to the nature of http the page refresh is usually accomplished automatically by JavaScript code running on the web page.

When the web page is updated then usually the entire contents of the page is sent to the web browser regardless of the changes.

This isn’t a problem with a single client but for 100s or thousands of connected clients this would cause significant server load.

The Solution Using MQTT

The solution using MQTT is to connect a single http client to the web server as before.

The web page is downloaded, processed, and the data for each flight is extracted.

The data is then sent over MQTT with each flight having its own topic, and only changed data is sent to the subscribing client.

An MQTT subscriber could subscribe to the flight that it wanted information for rather than seeing the data for all flights.

This will result in a considerable reduction in website and network traffic.

Python Demo Script Example

This is a demo script written in Python that reads data from a web page processes the data and republishes the data over MQTT.

The example I will be using is flight arrival information published by a local airport.

Because the script needs to parse the HTML of the web page provided by the airport I have created test data from the one I used.

This test data can be used instead of live data from the website.

This means that you can safely run the script without affecting the actual website.

Important script settings

"/arrivals-and-departures/" #add yours here
base_topic="Flights" #base topic for mqtt to publish on
record_flag=False #make copy of web pages then quit
get_from_disk=True #set true to read data from disk not website
scan_interval=10 #should be 60 if reading from website can make any value when getting data from disk

To read from website then set:

record_flag=False #make copy of web pages then quit
get_from_disk=False #set true to read data from disk not website
scan_interval=60

To Just record data to disk set

record_flag=True

To read data from stored pages on disk set:

record_flag=False #make copy of web pages then quit
get_from_disk=False #set true to read data from disk not website
scan_interval=60

Note: This is the Recommended option.

Note: I will shortly be making a video showing the script usage as it is easy to demonstrate using video.

Here is a sample output and you can clearly see the data size difference

Video -How to Republish HTTP Data (web page) Over MQTT

Script and Data

Reading Republished Data In A Web Browser

Although http will likely give way to MQTT the web browser will most like still remain the dominant user Interface.

Fortunately most MQTT brokers support MQTT over websocket and all modern browsers support websockets.

Paho provide a Javascript client that you can use to enable a web browser to use MQTT over websockets.

The video below shows hoe to use this to access our flight Data:

 

Related Tutorials and Resources

• How to Send a File Using MQTT Using a Python Script
• Simple Python MQTT Data Logger
• Logging MQTT Sensor Data to SQL DataBase With Python