All 15 File Uploads & Media Processing questions
More in File Uploads & Media Processing
What is multipart/form-data, and why does a Node.js server need special middleware (like multer) to handle file uploads? What is the difference between multer's diskStorage and memoryStorage engines? Why should an application typically avoid storing uploaded files directly on the same server instance that runs the web application? How would you validate and restrict uploaded file types and sizes using multer to prevent malicious or oversized uploads? Why should CPU-intensive media processing like image resizing or video transcoding typically not run directly inside an Express request handler? How would you validate an uploaded image's actual dimensions and reject ones that don't meet minimum or maximum size requirements? How would you implement a progress indicator for a large file upload from the client's perspective, given the Node.js server side? What is the difference between upload.single(), upload.array(), and upload.fields() in multer? How would you process and resize an uploaded image in Node.js using the sharp library, and why is sharp generally preferred over pure-JavaScript alternatives? How would you implement direct-to-S3 file uploads from a client, avoiding routing large file uploads through your Node.js server entirely?
Advertisement
Node.js › File Uploads & Media Processing

How would you stream a large file upload directly to processing (like virus scanning or transcoding) without first saving the entire file to disk?

Intermediate

Answer

Rather than using multer's disk storage (which writes the complete file to disk first) you can use multer's memory storage for smaller files, or for genuinely large files, use a library that exposes the upload as a readable stream directly (like busboy, which multer itself is built on), piping that stream directly into a processing pipeline (a virus scanner, a transcoder, cloud storage) as data arrives, without ever needing the complete file to exist on local disk at any point.
const busboy = require('busboy');

app.post('/upload', (req, res) => {
  const bb = busboy({ headers: req.headers });
  bb.on('file', (name, fileStream) => {
    fileStream.pipe(s3UploadStream); // streams directly through to S3, no local disk write
  });
  req.pipe(bb);
});
Real-world example A media-processing service streams uploaded video files directly into an ffmpeg transcoding pipeline as they're received, never writing the potentially multi-gigabyte original file to local disk at all, reducing both disk I/O and processing latency.

Common follow-ups: What's the tradeoff of this streaming approach if a processing step later in the pipeline needs to seek backward within the file, which streams don't naturally support?;How would you handle validating a file's type before processing begins, given you only have the first chunk of a stream to inspect initially?

Streams & Buffers;Cloud & DevOps
All 15 File Uploads & Media Processing questions