What do video processing, 3D and 2D graphics, cryptography, and network protocols have in common? They all use buffers.
It is a low-level abstraction that makes it possible to tap into efficient algorithms and direct memory management. There are no tools in JavaScript that offer the same level of low-level programming as buffers.
This article gives you the answer to the question, "Why do I need to learn buffers?" It walks you through how buffer works on a conceptual level and then goes into detail on how to use it in JavaScript.
Why you need to understand buffers
Buffers became an irreplaceable part of modern JavaScript. Some tasks simply can't be done without using them. Here are just a few areas where buffers play a crucial part.
Working with files
You might not realize it yet, but when you work with files in JavaScript, you work with buffers. Any file on your computer is just a set of 0s and 1s. In other words, it is a raw binary.
The File JavaScript class is just an abstraction over this raw binary to make things easier to deal with. Don't believe me? Let's create a file from absolute zero using ArrayBuffer as a raw binary base for a file.
const buffer = new ArrayBuffer(8);
const file = new File([buffer], 'new-file');
// File size - 8 bytes, File name - new-file
console.log(file.size, file.name);
You can even save this file on your computer.
Image, video, and audio processing
Buffers are irreplaceable when it comes to media file processing because you have access to the raw binary of a file. A raw binary contains a bunch of different information like compression method, color depth, metadata, etc.
The music-metadata library is a JavaScript for Node.js that allows you to get so much additional information from an audio file.
Codec
Bitrate
Frame headers
And so much more. There is simply no ready-to-use API to get all this info except for the raw binary itself.
There is also the AudioBuffer class provided by the Web Platform. All audio data in the audio buffer is stored in the same old ArrayBuffer. How do we know that? Instances of the AudioBuffer class have a method called getChannelData , and it returns the underlying data as a typed array.
Network protocols
On the Internet, we communicate using different network protocols:
Transmission Control Protocol (TCP)
File transfer protocol (FTP)
Hypertext transfer protocol (HTTP)
Notice that all mentioned protocols have a word related to transfer. But what do we transfer? The answer is raw binary. And you already know when there is a raw binary, there is a buffer.
One of the most popular WebSocket libraries, Socket.io, uses buffers heavily across the library. It also makes it possible to send a raw binary. There is no way to support such features without using buffers.
3D and 2D graphics, game development
3D and 2D graphics are never easy to get right. There are many things that you have to be aware of to make a good graphic or game. One of such is the texture. The texture is simply a surface of a model.
Source: https://3d-ace.com/3d-modeling/
You can clearly see the difference between a plain model on the left and a model with texture on the right. There are dedicated 3D libraries in JavaScript that make it possible to work with all the complexity of 3D graphics, and of course, they use buffers for it.
Here is an example of the DDS file format decompression abstraction provided by the BabylonJS library. It is a special file format for model textures. You can instantly notice one thing—it uses buffers a lot. The reason is simple: we're working with decompression of the raw binary.
Cryptography
One of the main requirements in the cryptography industry for code is speed and efficiency. Buffers in JavaScript offer exactly that:
Buffers provide a way to represent fixed-length binary data, which is perfect for crypto keys, hashes, or encrypted data.
It is possible to allocate memory more precisely and in smaller amounts for the same data using buffers.
Working with the raw binary makes it easier to use bit-wise operators, which are faster than regular ones.
In the previous article about bits and bytes in JavaScript, we talked about how much space the V8 JavaScript engine allocates for numbers.
In short, it is either 31 or 64 bits. For numbers without a floating point number, the engine allocates 31 bits. But what if we only work with numbers from 0 to 255? It'll still use 31 bits per each number.
We can do it with almost 4 times less memory using a buffer.
// 92 bits for numbers + array.
const numbers = [1, 2, 3];
// 24 bits for numbers + typed array.
const u8Array = new Uint8Array([1, 2, 3]);
The Uint8Array uses ArrayBuffer under the hood to store the data. We'll talk about it in more detail in the upcoming article on buffer views.
JavaScript buffer is an abstraction
Buffer is an abstraction. This abstraction consists of two specific parts: memory chunk and handle to this memory.
Buffer can allocate a specific amount of memory and store the reference to this memory in JavaScript.
We're talking about direct memory allocation from a high-level language perspective that is supposed to abstract us from direct interaction from memory. Turns out it is not always possible.
The memory handle is the buffer JavaScript object we can directly interact with in our applications. If you're familiar with lower-level programming languages like C or C++, it looks pretty much like a safe version of a memory pointer.
JavaScript buffer representation—ArrayBuffer
We've talked about buffer structure and how it works in theory. The next step is to look at the particular buffer implementation in JavaScript. It is called ArrayBuffer.
The ArrayBuffer is a class that we use to create a buffer.
const buffer = new ArrayBuffer(4);
The number 4 we passed to the constructor indicates how many bytes you want to allocate in memory for this buffer. You can use the byteLength property to check if the buffer is of the right size.
const buffer = new ArrayBuffer(4);
console.log(buffer.byteLength); // Prints 4
By default, ArrayBuffer instances are immutable. We can't directly write any data inside the buffer. But what is the point of having a chunk of memory and not being able to work with it? That's why we have views.
A view is a mechanism that makes it possible to modify a buffer content. We'll talk about views in more detail in the upcoming article.
The only thing we can change when working with ArrayBuffer is the size. To do so, we have to specify the maxByteLength option. The option tells the buffer that it can't grow further than this number.
const buffer = new ArrayBuffer(4, { maxByteLength: 8 });
console.log(buffer.byteLength); // Prints 4
buffer.resize(8);
console.log(buffer.byteLength); // Prints 8
If you try to resize the buffer to more than 8 bytes, the resize function throws a RangeError saying that the value is invalid.
const buffer = new ArrayBuffer(4, { maxByteLength: 8 });
// RangeError: ArrayBuffer.prototype.resize: Invalid length parameter
buffer.resize(10);
Conclusion
Buffers are essential when it comes to low-level things like:
Video, image, and audio processing
3D and 2D graphics and game development
Cryptography
Network protocols
General work with files
A buffer is an abstraction that consists of 2 main parts: memory allocated for this particular buffer and JavaScript handle to the memory. Allocated memory for the buffer is not subject to garbage collection, but the JavaScript handle is. Whenever the JavaScript handle gets destroyed, it frees the allocated memory.
In JavaScript, the lowest possible abstraction of a buffer is called ArrayBuffer. It is an immutable structure that can't be directly modified. That's why we use views to modify a buffer.