Working with Node.js Streams
What are streams?
Streams are a collection of data that might not be complete and we might not fit them in memory.
When we watch an online video we use a stream. Do we need to download the whole video to watch it? the answer is no. thus the video data is not complete but we can watch the parts we want!
Imagine we are uploading a video with 12GB size. The problem is our memory is only 4GB. So how is it possible to upload such a file? the answer is that with using streams I don’t need to use my memory and load that 12GB file in my memory and I’m going to send this file chunk by chunk in a stream.
Where do we use streams in Node.js?
Some of the Node.js modules implementing a stream interface.
HTTP responses are the most common example of streams. HTTP responses are readable streams on the client and writable streams on the server. We are going to discuss different types of streams.
What are the different types of streams?
- Writable Stream: we can write to this stream.
- Readable Stream: we can read from this stream.
- Duplex Stream: we can read and write to this stream.
- Transform: Transform streams are duplex streams where the output is in some way related to the input.
It is impossible to understand these sentences without proper examples. So let’s see a few examples:
Writable Stream examples
Writing to HTTP response
Here is a simple HTTP server and we want to see if we can write to HTTP response or not :
This is a server listening on port 2000. As you can see we have written a few lines using write method on this writable stream. If we send a request to this port the output will be like this:
Another important part of this code is using end method on a writable stream.
If we don’t use the end() method the client is going to wait until the request becomes timed out.
Writing to a file
We want to write a specific sentence a million times in a file.
We can create a string and we can append this sentence a million times but in the real-world, we cannot do this because some times the size of this variable can be more than our memory size. This was part of the stream definition and we can use a writable stream to fill this file with our sentences.
Readable Stream Examples
Reading from a file
Imagine we want to read from a big file like the file we created using a writable stream and we want to check if a word exists or not. Since this is a bad idea to load the whole file into memory we can use a readable stream instead.
Remember that streams are event emitter and we can register event listeners. Here we used data event to log data.
Two modes of Readable Streams
There are two modes of readable streams:
Paused mode and Flowing mode
When we use the Flowing mode the data is coming and we have to use event handlers and listen to data event. Actually, when we use event handlers we switch the readable stream to the Flowing mode. In order to change the stream mode, we can use paused() method and resume(). In Paused mode, we read data using read() method.
How we can read from a readable stream and write to another writable stream?
Let’s see the answer in a simple example. We know that HTTP response is a writable stream on the server and also we know that we can create a readable stream from a file. So let’s give the content of the file as a response to the user. We have a readable stream and we want to write it to a writable stream (response).
The first way we can do this is by using event handlers:
As you can see we have two event listeners one for writing data to res and one for ending the response.
The second way is using pipe and it is just like pipes in Linux.
Readable streams have a pipe method that we can use to pass a writable stream.
Example of Duplex Streams
We can create any kind of stream using the stream module.
In order to have a duplex stream, we need to implement both read and write methods:
I used process.stdin which is a readable stream and process.stdout which is a writable stream so when I write something on my console I’m writing to that duplex stream.
Example of Transform stream and its usage
We need to use stream.Transform to create a transform stream and all we need to do is implementing the transform method.
For example, I want to replace the numbers of a stream with *:
As you can see we must use push method to add something to a readable stream and we have access to chunk which comes from a writable stream.
Now let’s see what is the result:
I entered the first line and the second line is written by transform stream to process.stdout.
