Stream Buffering Visualized

stream-examples is an attempt to visually represent the state of the internal buffers of a read and write stream during a .pipe operation.

The code that is examined in this post (backpressure) at its essence is just: myReadStream.pipe(myWriteStream). The other relevant fact is that myWriteStream processes data more slowly than myReadStream.

There are still a few interesting moments to observe:

1. Nothing is queued up in the read buffer until the write buffer is full. Chunks are passed directly to the write buffer as they are read:

2. Once the write buffer fills up, it stops accepting new chunks until it empties completely, and emits a 'drain' event. Chunks will accumulate in the read buffer as the write buffer empties:

3. If the read buffer fills completely before the write buffer drains, read operations pause. This is signaled by this.push in the read stream returning false. Note that even though there is room in the write buffer as chunks are processed, nothing is transferred from the read buffer yet:

4. Once the write buffer empties and the 'drain' event is emitted, as much of the read buffer as possible is transferred to the write buffer. At this point the source starts reading in new chunks:

This process continues until the read stream pushes all of its data, and all of the chunks in the read and write buffers are processed.