Feb 27, 2024 • 4 min read

Node Firestore Import

A quick aide memoir on how to import data from a CSV formatted files into Firestore. If you have used the Pet Theory lab, this should be familar, but if not here is the code.

The general idea is to create a csv and import it directly into Cloud Firestore. To do this, the code will read the csv, then connect to Firestore and add a new collection containing the csv records.

If you havent worked with Firestore before, this is a nice short cut to get to grips with the Firestore API.

Requirements:

Node.js installation
Firestore package

Overview

In this blog post learn how to parse a CSV, and add the information to Firestore. The following content can also be applied to a different formats if required.

To transition between JSON and CSV, check out the JQ to CSV blog post.
To create fake CSV test data, check out the Faker blog post.

Run app

The npm packages are required:

NODE_TYPE // bash

npm install csv-parse @google-cloud/firestore

Run the application

Add the csv file e.g. customers_10.csv as the input.

NODE_TYPE // bash

node index.js customers_10.csv

Expected Output:

NODE_TYPE // bash

Wrote 10 records

Refresh the Cloud Firestore database to see the updated records.

Application Code

The template application will be based on Node.js. If you are unfamiliar with Node.js, this is a standard step to create the files required.

The package.json file provides the information relating to packages used and general descriptions. In addition it has a helpful script section, for testing that can be generalised.

Add a package.json

            NODE_TYPE // bash
        
{
  "name": "fs-import",
  "version": "1.0.0",
  "description": "Import CSV into Cloud Firestore",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "Rich Rose",
  "license": "MIT",
  "dependencies": {
    "@google-cloud/firestore": "^7.3.0",
    "csv-parse": "^5.5.3",
  }
}

Create an index.js file

The code is required to do the following things:

Include package dependencies
Add validation for command line arguments
Add a writeToFirestore function
Add an importCSV function

Package Dependencies

A couple of declarations are added for handling the data entry. The npm packages are required:

csv-parse
@google-cloud/firestore

            NODE_TYPE // JavaScript
        
const csv = require('csv-parse');
const fs  = require('fs');
const { Firestore } = require("@google-cloud/firestore");

const projectId    = 'PROJECT_ID'
const collectionId = 'customers'

const resource = {
  type: "global",
};

Validate Command Line Arguments

The application expects the name of a CSV file to be added. If a filename is not present, the application will exit.

            NODE_TYPE // javascript
        
if (process.argv.length < 3) {
  console.error('Please include a path to a csv file');
  process.exit(1);
}

importCsv(process.argv[2]).catch(e => console.error(e));

Read the CSV file

Use the csv-parse package to read a csv filename. The csv is comma separated and is expected to include a header. Each item read becomes a record with the header values used as field names.

            NODE_TYPE // javascript
        
async function importCsv(csvFilename) {
  const parser = csv.parse({ columns: true, delimiter: ',' }, async function (err, records) {
    if (err) {
      console.error('Error parsing CSV:', err);
      return;
    }
    try {
      console.log(`Call write to Firestore`);
      await writeToFirestore(records);
      console.log(`Wrote ${records.length} records`);
    } catch (e) {
      console.error(e);
      process.exit(1);
    }
  }); 

  await fs.createReadStream(csvFilename).pipe(parser);
}

Write to Firestore

The writeToFirestore function is asynchronous and reads each record presented. The record parameter is a data object, and each item is written to the named collection.

Note:

The code does not include pagination, so loading of huge files is not supported.
Assumes the application code can communicate to Cloud Firestore database.

            NODE_TYPE // javascript
        
async function writeToFirestore(records) {
  const db = new Firestore({  
    // projectId: projectId 
  });
  const batch = db.batch()

  records.forEach((record)=>{
    console.log(`Write: ${record}`)
    const docRef = db.collection(collectionId).doc(record.email);
    // batch.create(docRef, record)
    batch.set(docRef, record, { merge: true })
  })

  batch.commit().then(() => {
    console.log('Batch executed')
  }).catch(err => {
    console.log(`Batch error: ${err}`)
  })
  return 
}

Complete Code

            NODE_TYPE // javascript
        
const fs  = require('fs');
const csv = require('csv-parse');
const { Firestore } = require("@google-cloud/firestore");

const projectId    = 'PROJECT_ID'
const collectionId = 'customers'

const resource = {
  type: "global",
};

async function writeToFirestore(records) {
  const db = new Firestore({  
//        projectId: projectId 
  });
  const batch = db.batch()

  records.forEach((record)=>{
    console.log(`Write: ${record}`)
    const docRef = db.collection(collectionId).doc(record.email);
    // Create new document
    // batch.create(docRef, record)
    // Merge or create new doc
    batch.set(docRef, record, { merge: true})
  })

  batch.commit().then(() => {
    console.log('Batch executed')
  }).catch(err => {
    console.log(`writeToFirestore: ${err}`)
  })
  return 
}

async function importCsv(csvFilename) {
  const parser = csv.parse({ columns: true, delimiter: ',' }, async function (err, records) {
    if (err) {
      console.error('csv-parse:', err);
      return;
    }
    try {
      console.log(`Call write to Firestore`);
      await writeToFirestore(records);
      console.log(`Wrote ${records.length} records`);
    } catch (err) {
      console.error(`importCsv: ${err}`);
      process.exit(1);
    }
  }); 

  await fs.createReadStream(csvFilename).pipe(parser);
}

if (process.argv.length < 3) {
  console.error('Please include a path to a csv file');
  process.exit(1);
}

importCsv(process.argv[2]).catch(err => console.error(err));