Republish HTML Data Over MQTT (Flight Arrivals )

mqtt-republish-iconHTTP is the protocol that powers the Web,and is the most common protocol on the Internet.

Because http is a request/response protocol a visitor viewing a web page must issue a new request in order to refresh the data.



For most websites this isn’t a problem as the page content is static, but there are a class of sites were the information being displayed is changing, and for which http may not be the best way to deliver it.

Flight Information – The Problem

Data for Live flight arrivals is displayed on a web page and is updated on a regular basis (every 60 seconds is common).

Due to the nature of http the refresh is usually accomplished by JavaScript code running on the web page.

When the webpage is updated usually the entire contents of the page is sent regardless of the changes.http-data-request

This isn’t a problem with a single client but for 100s or thousands of connected clients this would cause significant server load.

The Solution Using MQTT

The solution using MQTT is to connect a single http client to the web server as before.

The web page is downloaded, processed, and the data for each flight is extracted.

The data is then sent over MQTT with each flight having its own topic, and only changed data is sent.

An MQTT subscriber could subscribe to the flight that it wanted information for rather than seeing the data for all flights.

republish-http-data-mqtt

This should result in a considerable reduction in website and network traffic.

Demo Script Example

This is a demo script written in Python that reads data from a web page processes the data and republishes the data over MQTT.

The example I will be using is flight arrival information published by a local airport.

Because the script needs to parse the HTML of the web page provided by the airport I have created test data from the one I used which can be used instead of live data from the website.

This means that you safely run the script without affected the actual website.

Important script settings

"/arrivals-and-departures/" #add yours here
base_topic="Flights" #base topic for mqtt to publish on
record_flag=False #make copy of web pages then quit
get_from_disk=True #set true to read data from disk not website
scan_interval=10 #should be 60 if reading from website can make any value when getting data from disk

To read from website then set:

record_flag=False #make copy of web pages then quit
get_from_disk=False #set true to read data from disk not website
scan_interval=60

To Just record data to disk set

record_flag=True

To read data from stored pages on disk set:

record_flag=False #make copy of web pages then quit
get_from_disk=False #set true to read data from disk not website
scan_interval=60

Note: This is the Recommended option.

Note: I will shortly be making a video showing the script usage as it is easy to demonstrate using video. Here is a sample output and you can clearly see the data size difference

republish-script-output

Video

Here is a video showing the script usage.

You can download the script here

I am interested the gauge the popularity of such a script so that I can decide to develop it further from its current draft state. Could you please complete the short survey below and I would be grateful for any comments.

Does this script Interest you?

View Results

Loading ... Loading ...

Related Tutorials and Resources

Please rate? And use Comments to let me know more
[Total: 0    Average: 0/5]

Leave a Reply

Your email address will not be published. Required fields are marked *