Upload file to an Express Server

I was working on creating a proof of concept RabbitMQ data pipeline in Node where a web app would upload a large csv file to an Express server and the server would stream its content into the pipeline in JSON.

There are two possibilities of uploading a file
1. Send entire file
2. Stream file

Send Entire File

Send entire csv file from browser

  fetch('http://localhost:3000/upload', {
    method: 'POST',
    headers: {
      'Content-Type': 'text/csv' 
    },
    body: file 
  })
  .then(success => console.log(success))
  .catch(error => console.log(error))

The two important points int the server are

1. How to handle request
2. How to stream csv file content as json into pipeline

To get a stream of JSON objects from the csv file, create a stream and pipe that stream into fast-csv.

  const app = require('express')()
  const textBodyParser = require('body-parser').text
  const csv = require('fast-csv')
  const { Readable } = require('stream')
  
  // Handle very large file
  app.use(text({ type: 'text/csv', limit: '500mb' }))
  
  app.post('/upload', (req, res) => {
    const content = Readable.from(req.body)
    content
      .pipe(csv.parse({ headers: true }))
      .on('data', (data) => {
        console.log(data) // Handle JSON object
      })
    res.sendStatus(200)
  })

Stream File

Stream csv file from browser

  //Important that file is sent as FormData
  const data = new FormData()
  data.append('file', file)
  fetch('http://localhost:3000/upload', {
    method: 'POST',
    body: data,
  })
  .then((success) => console.log(success)) 
  .catch((error) => console.log(error))
  

For the server to handle the stream, the HTTP request must have the header Content-Type: multipart/form-data; boundary=aBoundaryString, more info found here.
By sending the file as form data we can avoid having to specify this header.The browser will take care of it.

Use busboy to get the file stream and pipe that to fast-csv to get a stream of JSON objects.

The resulting code

  app.post('/upload', (req, res) => {
    const busboy = new Busboy({ headers: req.headers })
    // Busboy gives us a lot information regarding the file
    busboy.on('file', (__, file) => {
      file.pipe(csv.parse({ headers: true })).on('data', (row) => {
        // Handle data here. Row is a csv row in JSON
        console.log('Row in JSON', row) 
      })
      file.on('end', function () {
        // Handle end case here
        console.log('file ended')
      })
    })
    busboy.on('finish', function () {
      res.writeHead(303, { Connection: 'close', Location: '/' })
      res.end()
    })
    req.pipe(busboy)
  })