Node Firestore Import
A quick aide memoir on how to import data from a CSV formatted files into Firestore. If you have used the Pet Theory lab, this should be familar, but if not here is the code.
The general idea is to create a csv and import it directly into Cloud Firestore. To do this, the code will read the csv, then connect to Firestore and add a new collection containing the csv records.
If you havent worked with Firestore before, this is a nice short cut to get to grips with the Firestore API.
Requirements:
- Node.js installation
- Firestore package
Overview
In this blog post learn how to parse a CSV, and add the information to Firestore. The following content can also be applied to a different formats if required.
- To transition between JSON and CSV, check out the JQ to CSV blog post.
- To create fake CSV test data, check out the Faker blog post.
Run app
The npm packages are required:
1npm install csv-parse @google-cloud/firestore
Run the application
Add the csv file e.g. customers_10.csv as the input.
1node index.js customers_10.csv
Expected Output:
1Wrote 10 records
Refresh the Cloud Firestore database to see the updated records.
Application Code
The template application will be based on Node.js. If you are unfamiliar with Node.js, this is a standard step to create the files required.
The package.json file provides the information relating to packages used and general descriptions.
In addition it has a helpful script section, for testing that can be generalised.
- Add a package.json
1{
2 "name": "fs-import",
3 "version": "1.0.0",
4 "description": "Import CSV into Cloud Firestore",
5 "main": "index.js",
6 "scripts": {
7 "test": "echo \"Error: no test specified\" && exit 1"
8 },
9 "keywords": [],
10 "author": "Rich Rose",
11 "license": "MIT",
12 "dependencies": {
13 "@google-cloud/firestore": "^7.3.0",
14 "csv-parse": "^5.5.3",
15 }
16}
- Create an index.js file
The code is required to do the following things:
- Include package dependencies
- Add validation for command line arguments
- Add a writeToFirestore function
- Add an importCSV function
Package Dependencies
A couple of declarations are added for handling the data entry. The npm packages are required:
- csv-parse
- @google-cloud/firestore
1const csv = require('csv-parse');
2const fs = require('fs');
3const { Firestore } = require("@google-cloud/firestore");
4
5const projectId = 'PROJECT_ID'
6const collectionId = 'customers'
7
8const resource = {
9 type: "global",
10};
11
Validate Command Line Arguments
The application expects the name of a CSV file to be added. If a filename is not present, the application will exit.
1if (process.argv.length < 3) {
2 console.error('Please include a path to a csv file');
3 process.exit(1);
4}
5
6importCsv(process.argv[2]).catch(e => console.error(e));
Read the CSV file
Use the csv-parse package to read a csv filename.
The csv is comma separated and is expected to include a header.
Each item read becomes a record with the header values used as field names.
1async function importCsv(csvFilename) {
2 const parser = csv.parse({ columns: true, delimiter: ',' }, async function (err, records) {
3 if (err) {
4 console.error('Error parsing CSV:', err);
5 return;
6 }
7 try {
8 console.log(`Call write to Firestore`);
9 await writeToFirestore(records);
10 console.log(`Wrote ${records.length} records`);
11 } catch (e) {
12 console.error(e);
13 process.exit(1);
14 }
15 });
16
17 await fs.createReadStream(csvFilename).pipe(parser);
18}
Write to Firestore
The writeToFirestore function is asynchronous and reads each record presented.
The record parameter is a data object, and each item is written to the named collection.
Note:
- The code does not include pagination, so loading of huge files is not supported.
- Assumes the application code can communicate to Cloud Firestore database.
1async function writeToFirestore(records) {
2 const db = new Firestore({
3 // projectId: projectId
4 });
5 const batch = db.batch()
6
7 records.forEach((record)=>{
8 console.log(`Write: ${record}`)
9 const docRef = db.collection(collectionId).doc(record.email);
10 // batch.create(docRef, record)
11 batch.set(docRef, record, { merge: true })
12 })
13
14 batch.commit().then(() => {
15 console.log('Batch executed')
16 }).catch(err => {
17 console.log(`Batch error: ${err}`)
18 })
19 return
20}
Complete Code
1const fs = require('fs');
2const csv = require('csv-parse');
3const { Firestore } = require("@google-cloud/firestore");
4
5const projectId = 'PROJECT_ID'
6const collectionId = 'customers'
7
8const resource = {
9 type: "global",
10};
11
12async function writeToFirestore(records) {
13 const db = new Firestore({
14// projectId: projectId
15 });
16 const batch = db.batch()
17
18 records.forEach((record)=>{
19 console.log(`Write: ${record}`)
20 const docRef = db.collection(collectionId).doc(record.email);
21 // Create new document
22 // batch.create(docRef, record)
23 // Merge or create new doc
24 batch.set(docRef, record, { merge: true})
25 })
26
27 batch.commit().then(() => {
28 console.log('Batch executed')
29 }).catch(err => {
30 console.log(`writeToFirestore: ${err}`)
31 })
32 return
33}
34
35async function importCsv(csvFilename) {
36 const parser = csv.parse({ columns: true, delimiter: ',' }, async function (err, records) {
37 if (err) {
38 console.error('csv-parse:', err);
39 return;
40 }
41 try {
42 console.log(`Call write to Firestore`);
43 await writeToFirestore(records);
44 console.log(`Wrote ${records.length} records`);
45 } catch (err) {
46 console.error(`importCsv: ${err}`);
47 process.exit(1);
48 }
49 });
50
51 await fs.createReadStream(csvFilename).pipe(parser);
52}
53
54if (process.argv.length < 3) {
55 console.error('Please include a path to a csv file');
56 process.exit(1);
57}
58
59importCsv(process.argv[2]).catch(err => console.error(err));