Node.js › File Uploads & Media Processing
How would you stream a large file upload directly to processing (like virus scanning or transcoding) without first saving the entire file to disk?
IntermediateAnswer
Rather than using multer's disk storage (which writes the complete file to disk first) you can use multer's memory storage for smaller files, or for genuinely large files, use a library that exposes the upload as a readable stream directly (like busboy, which multer itself is built on), piping that stream directly into a processing pipeline (a virus scanner, a transcoder, cloud storage) as data arrives, without ever needing the complete file to exist on local disk at any point.
const busboy = require('busboy');
app.post('/upload', (req, res) => {
const bb = busboy({ headers: req.headers });
bb.on('file', (name, fileStream) => {
fileStream.pipe(s3UploadStream); // streams directly through to S3, no local disk write
});
req.pipe(bb);
});Real-world example A media-processing service streams uploaded video files directly into an ffmpeg transcoding pipeline as they're received, never writing the potentially multi-gigabyte original file to local disk at all, reducing both disk I/O and processing latency.