Understanding and Using Buffers In Node-Red

When data is read from a file or network it is read byte by byte into a data buffer.

Data Buffers are temporary storage used for transferring data.

To work with binary data we will need access to these buffers.



To work with buffers in nodejs and node-red we use the buffer object.

The following screen shot of the node command line shows how we work with characters using the buffer object.

The first thing we need to do is create a buffer. This can be done in a number of ways:

Create a buffer from a string.

Buffer.from('abcde')

buffer-string

Create a buffer from an array of numbers

var b = Buffer.from([1,2,3,4]);

buffer-numbers

Notice the difference between numbers and strings and numbers as numbers.

numbers-strings-compare

Create an empty buffer of 10 bytes and Initialise it.

var b = Buffer.alloc(10);

buffer-alloc

Working With String Buffers

If you read the tutorial on data and character encoding you will see that Characters and numbers e.g A and 1 are represented as bytes by encoding them.

In the early days of computing ASCII was the encoding standard,and was very simple as a character was a byte.

However due to the lack of support for foreign characters ASCII has been replaced by utf-8.

Below is a screen shot showing a partial ASCII table for the numbers 0-9.

Partial-Ascii-table

1 is encoded as hex 31 and binary 00110001

When we use the Buffer.from method to create a buffer from a string then we can also specify an encoding scheme, utf-8 is the default.

Now utf-8 uses 1 to 4 bytes to encode an character whereas ASCII only uses 1 byte.

However when we encode the characters ‘abcde’ we only have 5 bytes.

buffer-string

This is because the English characters are the same in ASCII as in utf-8.

However what happens if we encode foreign language characters like the French é ?

In the above screen below we see for English characters the encoding is the same in ASCII as UTF-8 and uses a single byte.

However when encoding the French é UTF-8 uses 2 bytes and ASCII can’t encode it and so it substitutes the question mark character.

character-encoding-buffers

Interpreting Data in Buffers

When reading in data from the network or from a file it is important to know what type of data we are working with.

If it is string data then what is the encoding that was used. If we take our simple string example of abcdé encoded as UTF-8.

If we read it using the toString() method, and decode it as ASCII this is what we get:

decode-buffer-ascii

You will notice that ASCII uses 1 byte per character the French character é is encoded as C3 A9.

It is decoded incorrectly in ASCII as C).

Node-Red Example Reading a File

If you look at the file read node you will see that there are 4 options of how the read presents the data to your flow. They are:

  1. A single utf-8 string – In the case the entire file is read and presented to your program as a text string. You need to wait for the read to complete before you can process the data.
  2. One msg per line– Presents the data to your program as lines of text. You can process data while still reading the file.
  3. A single Buffer object – Same as the first option but now you have a sequence of raw bytes. You need to wait for the read to complete before you can process the data.
  4. A stream of buffers -Same as second option but you have a sequence of bytes so you can process data while still reading the file.

File In Options

The first and second options are the most common as files normally contain text data.

If the data in the file isn’t text data or not encoded as utf8 then the single utf8 and 1 msg per line options won’t work as with these options the data is read in, and decoded using utf8 into characters.

Note: See data encoding tutorial for more details

The third and forth options are for when the file contains binary data or text data encoded in another format other than utf8 or ASCII..

These options reads the data into you script/flow without any interpretation i.e as raw data.

Video

In the video below I cover the above and also show you how to use Buffers in node-red to read text data from a file.

Demo flow used in video

download

Encoding and Decoding Integers and Floats

When working with machines you will often be required to encode and decode integer and float data, modbus is a good example.

Early computers used 16 bits to represent an Integer and 32 bits for a floating point number.

Today they can be much larger but the process is the same.

The image below shows a 16 bit Integer and how it is organised into bytes:

Integer-buffer

It is important to understand that because data is sent across a network in bytes it is important to understand whether the MSB (Most Significant Byte) or the LSB (Least Significant Byte) is transmitted first.

This is know as endianness and is discussed in the video.

Most networks use Big endianness (MSB first).

Video -Node-Red Buffers- (Integers and Floats)

Demo flow used in video

download

Worked Examples:

Creating a 2 byte buffer from an Integer (16 bits)

First we create an 2 byte empty buffer and then write the integer into it

var buf=Buffer.alloc(2);
buf.writeInt16BE(17201);

Creating a 4 byte buffer from an Integer (32 bits)

First we create an 4 byte empty buffer and then write the integer into it

var buf=Buffer.alloc(4);
buf.writeInt32BE(17201);

Creating a 16 bit Integer from a 2 byte buffer

We start with our 2 byte buffer containing a 16 bit integer and then we create the integer using the read method.

var buf=Buffer.from([0x43,0x6c])
var res =buf.readInt16BE()

Creating a 4 byte buffer from an Float (32 bits)

First we create an 4 byte empty buffer and then write the Float into it

var buf=Buffer.alloc(4);
buf.writeFloatBE(131.3)

Creating a 32 bit Float from a 4 byte buffer

We start with our 4 byte buffer containing a 32 bit Float and then we create the float using the read method.

var buf=Buffer.from([0x43,0x03,0x4c,0xcd])
var res =buf.readFloatBE();


Related Tutorials and Resources

Click to rate this post!
[Total: 3 Average: 5]

Leave a Reply

Your email address will not be published. Required fields are marked *