# Understanding Node.js Buffer So far, we've become familiar with buffers, typed arrays, data views, and how they all work together. If you missed the previous articles, I highly recommend reading the one dedicated to [buffers](https://pavel-romanov.com/javascript-buffers-explained-why-they-matter-and-how-to-use-them) and the other one on [views](https://pavel-romanov.com/uint8array-vs-dataview-choosing-the-right-buffer-view-in-javascript). Node.js provides a dedicated buffer abstraction called `Buffer`. Why do we need more buffers when we already have `ArrayBuffer` and the different views that come with it? In this article, we'll answer this question and understand the difference between Node.js buffer and all the others. In the end, you'll learn what problems the Node.js buffer has and why some people avoid using it at all costs. ## Difference between Node.js buffer and typed arrays In short, the Node.js buffer is basically `Uint8Array` spiced up with some extra logic. It automatically means two things: 1. Node.js buffer is not "actually" a buffer but a view into an underlying buffer. 2. Wherever you use `Uint8Array`, you can also use Node.js buffer, with minor exceptions that we'll discuss later in this article. One of the core abstractions responsible for buffers in Node.js is called `FastBuffer`. It is a class that extends `Uint8Array`. You can [see it](https://github.com/nodejs/node/blob/d9554978740a75d7150d9b58d232a1de6b88f93c/lib/internal/buffer.js#L956) for yourself. ```typescript class FastBuffer extends Uint8Array { // Using an explicit constructor here is necessary to avoid relying on // `Array.prototype[Symbol.iterator]`, which can be mutated by users. // eslint-disable-next-line no-useless-constructor constructor(bufferOrLength, byteOffset, length) { super(bufferOrLength, byteOffset, length); } } ``` Functions that create Node.js buffer such as `Buffer.from` always [return](https://github.com/nodejs/node/blob/d9554978740a75d7150d9b58d232a1de6b88f93c/lib/buffer.js#L476) the `FastBuffer`. But what is the point of having such a dumb class without any logic? Moreover, whenever we call `Buffer.from` it actually returns `Buffer`, not `FastBuffer`. ```typescript import { Buffer } from 'node:buffer'; const buffer = Buffer.from('hello'); console.log (buffer); // Prints ``` Am I misleading you? Not really. The reason you see `Buffer` instead of `FastBuffer` in the console and why `FastBuffer` doesn't contain any additional logic itself because of the [prototype manipulations](https://github.com/nodejs/node/blob/01cf9bccdfa7fb31a8a1d91ae45e594a730e0427/lib/buffer.js#L130). ```typescript FastBuffer.prototype.constructor = Buffer; Buffer.prototype = FastBuffer.prototype; addBufferPrototypeMethods(Buffer.prototype); ``` What is the point of such a manipulation? One of the reasons is backward compatibility. Many Node.js APIs have been using `Buffer` for a long time, and changing code to `FastBuffer` might result in big problems. That's why it is easier just to swap prototypes and keep the same interfaces for the existing code. The `Buffer` prototype is where all methods like `from`, `alloc`, and others reside. ![](https://cdn.hashnode.com/res/hashnode/image/upload/v1723736378794/8525e8e1-cef9-48a0-807c-cf9b985857b6.jpeg align="center") ## Node.js buffer memory allocation I want to bring your attention to how Node.js allocates the memory for its buffer. That's one of the most important differences between `Buffer` and `Uint8Array`, which you should be aware of. The first thing you have to understand is that typed arrays have separate memory pools. ```typescript const array1 = new Uint8Array('hello'); const array2 = new Uint8Array('world'); console.log(array1.byteOffset); // Prints 0 console.log(array2.byteOffset); // Prints 0 ``` By memory pool, I mean the buffer where data is stored. Every time you create a new typed array, you create a dedicated instance of `ArrayBuffer` with it. At the same time, Node.js buffers **share** the same memory pool. ```typescript import { Buffer } from 'node:buffer'; const buffer1 = Buffer.from('hello'); const buffer2 = Buffer.from('world'); console.log(buffer1.byteOffset); // Prints 16 console.log(buffer2.byteOffset); // Prints 24 ``` If you save anything less than 8 kilobytes in a buffer, then you end up with multiple data chunks in the same shared memory pool. You see `16` in the console because there was already some pre-allocated data inside of a buffer pool. The bytes offset of the second buffer directly depends on how much space the first buffer takes. ```typescript import { Buffer } from 'node:buffer'; const buffer1 = Buffer.from('controversial'); const buffer2 = Buffer.from('opinion'); console.log(buffer1.byteOffset); // Prints 16 console.log(buffer2.byteOffset); // Prints 32 ``` The word *hello* is 5 characters long, and the word *controversial* is 13 characters long. The difference between them is 8 characters, and it is the exact difference in the offset of a second buffer between two examples. I must say that there is a safer way to create a buffer. You can do so by using `alloc` function. However, it doesn't change the fact that the buffer has a shared memory pool. ## Node.js Buffer problems Now, that you understand how Node.js `Buffer` looks like it is time to discuss some of its problems. It's been an ongoing discussion since typed arrays became available. The main question is, "Why do we need to have the `Buffer` at all?" ### It is not compatible with other platforms Imagine you developing a library that you want to be available anywhere: browsers, Node.js, or Deno. If you try to write code using `Buffer` it will break in any other non-Node.js platform. The reason is simple, it is Node.js specific API. Instead, you can use typed arrays like `Uint8Array` or `Int8Array`. They are [part](https://tc39.es/ecma262/multipage/indexed-collections.html#table-49) of ECMAScript standard, and all big platforms that are running JavaScript support them. ### It violates Liskov substitution principle Liskov substitution principle (LSP) is one of the SOLID principles that state that a subclass should be able to substitute for its parent class without causing unexpected behaviors. While Node.js `Buffer` extends `Uint8Array`, it overrides `slice` method, which leads to violation of LSP because they behave differently. When you use `slice` with typed arrays like `Uint8Array` instances, a new typed array with a new underlying buffer is created. ```typescript const array1 = new Uint8Array([1, 2, 3]); const array2 = array1.slice(0, 2); array2[0] = 4; // Prints Uint8Array {0: 4, 1: 2} console.log(array2); // Prints Uint8Array {0: 1, 1: 2, 2: 3} console.log(array1); ``` After we mutate the first element of `array2`, it only affects the underlying buffer of `array2`. We don't see any changes to the first element in `array1`. `slice` method of Node.js Buffer is different. Instead of creating a copy, it creates a view into the same buffer. ```typescript import { Buffer } from 'node:buffer'; const buffer1 = Buffer.from([1, 2, 3]); const buffer2 = buffer1.slice(0, 2); buffer2[0] = 4; // Prints console.log(buffer2); // Prints console.log(buffer1); ``` You can see that `buffer2` has 4 as its first element, which is expected because we directly mutated this buffer. What is not expected is that the first element of `buffer1` has also changed to 4. `slice` method is [officially marked](https://nodejs.org/api/buffer.html#bufslicestart-end) as deprecated. However, as long as it is available as `Buffer` method it violates LSP. ### Buffer has more security implications The last but not least part is `Buffer` security. As we've mentioned before, Node.js buffer has a shared memory pool. If we have some data stored in this shared memory pool, any code that runs in your application can access this memory. Imagine a scenario where you store sensitive data in the buffer, like a person's address, and I, as a bad actor, can access this data. ```typescript import { Buffer } from 'node:buffer'; const addressBuffer = Buffer.from('Person personal addres'); ``` Here we have a buffer with personal information that we don't want to leak anywhere. We can gain access to it through the shared memory pool. ```typescript const hackyBuffer = Buffer.from('1'); // Prints the full underlying buffer console.log(hackyBuffer.buffer); ``` Assuming we don't have access to the initial buffer object itself, we create `hackyBuffer`. The underlying buffer of `hackyBuffer` contains the persons' private data. The only thing left is to interpret the hexadecimal data stored in the buffer into a human-readable format. ```typescript const hackyBuffer = Buffer.from('1'); const typedArray = new Uint8Array(hackyBuffer.buffer) .filter(value => value !== 0); const charCodes = Array.from(typedArray); // Prints "//Person personal addres" console.log(String.fromCodePoint(...charCodes)); ``` We created a view into the underlying buffer to be able to work with its content. Then, we filtered all empty values and created an array of character codes. At this point, it was automatically converted from a hexadecimal to a decimal numeric system. The only thing left is to convert those numbers into a string. Since all of these numbers are basically a number representation of Unicode characters, we can use `fromCodePoint` to convert them back to characters. If you're not comfortable with all these manipulations with character and numeric systems, I highly recommend reading the [article](https://pavel-romanov.com/from-ascii-to-unicode-a-javascript-developers-guide-to-text-encoding) about different encoding schemes in JavaScript. And that's how you "hack" Node.js buffer. ## Why do we need to use Node.js buffer at all After seeing all of these problems, it is reasonable to ask the purpose of using of Node.js buffer. It is a valid question, and it has been raised in Node.js community [before](https://github.com/nodejs/node/issues/41588). We can notice that the question of giving more preference to `Uint8Array` instead of `Buffer` having more and more attention which is a good sign. Despite all the problems, `Buffer` has useful functions that haven't been shipped with typed arrays yet. For example, we can convert buffer to `hex` or `base64` string. ```typescript import { Buffer } from 'node:buffer'; const buffer = Buffer.from('hello'); // Prints 68656c6c6f console.log(buffer.toString('hex')); ``` There are libraries that make it work with `Uint8Array`, but it is one more dependency in your project that is not always worth it. ## Conclusion `Buffer` is Node.js abstraction that enables you to create and manage buffers through a common interface. The `Buffer` extends `Uint8Array` and essentially not a buffer, but view into the underlying `ArrayBuffer`. While `Buffer` extends `Uint8Array`, it provides extra methods on top of the typed array, which makes it unique. Unlike regular typed arrays, Node.js buffer has a shared memory pool. It means that if you create two small buffers, they share the same memory space. This design with a shared memory pool leads to potential vulnerabilities because it is possible to get data from one buffer through a completely unrelated buffer. Other problems of `Buffer` include limited distribution across other platforms because it is Node.js-specific API, and violation of LSP principle from SOLID where superclass method behavior and subclass method behavior are different. Despite all of these problems, the Node.js `Buffer` still might be useful because it provides the functionality that typed arrays simply don't have.