All 15 File System & File Processing questions
More in File System & File Processing
How would you copy a large file efficiently in Node.js, and what's the difference between fs.copyFile() and manually piping streams? What is the purpose of the 'flag' option in fs.writeFile(), and what's the difference between 'w', 'a', and 'wx' flags? How would you implement a file-watching feature in Node.js using fs.watch(), and what are its known cross-platform limitations? How would you implement atomic file writes in Node.js to avoid leaving a file in a corrupted, half-written state if the process crashes mid-write? How would you implement file locking in Node.js to prevent two processes from writing to the same file simultaneously? How would you implement a directory-size calculator in Node.js that recursively sums the size of all files within a directory tree? How would you implement a simple in-process file-based cache in Node.js, and what are the tradeoffs versus using Redis?
Advertisement
Back to results for “SQL/Sequelize)”
Node.js › File System & File Processing

How would you process a very large CSV file in Node.js without loading the entire file into memory?

Advanced

Answer

Combining a readable file stream with a line-by-line or CSV-specific parsing library (like csv-parse in streaming mode) lets you process each row as it's read, rather than waiting for and holding the entire file in memory -- this is essential for files too large to fit comfortably in memory, and lets processing begin immediately rather than waiting for the entire file to load first.
const fs = require('node:fs');
const { parse } = require('csv-parse');

fs.createReadStream('huge-data.csv')
  .pipe(parse({ columns: true }))
  .on('data', (row) => { processRow(row); })
  .on('end', () => console.log('Finished processing'));
Real-world example A data-import tool processes a multi-gigabyte CSV file of customer records by streaming it through csv-parse and inserting each row into the database as it's parsed, rather than attempting to load and parse the entire file into an array first, which had previously caused out-of-memory crashes.

Common follow-ups: How would you add backpressure-aware batching, inserting rows in batches of 1000 rather than one at a time, to this streaming pipeline?;What happens if a single malformed row in the middle of a huge file causes a parsing error?

Streams & Buffers;Databases & ORMs (MongoDB/Mongoose SQL/Sequelize)
All 15 File System & File Processing questions