Merge multiple JSON files with JQ

Oct 29, 2023 linux jq json

Merging JSON files with JQ.

jq is like sed for JSON data - you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text.

If you have multiple JSON files that you want to combine, it can be tempting to write a program to perform this task. However, did you know that our old friend jq can do this without too much hassle.

In the following blog post learn how to use jq to transform multiple JSON data files into a single file.

The information below cover merging items with different array labels. I will write a short follow up to cover merging the same labels.

The example assumes you have a linux environment.
An example JSON file is provided to illustrate the commands required.

System Requirements

jq download link

Installation

Install the jq utility

1sudo apt install -y jq

Multiple JSON files and JQ

Lets look at an example file

Create a new JSON file named maths.json. Use the content below:

 1{
 2  "Maths": [
 3    {
 4      "topic": "Numerics",
 5      "data": [
 6        {
 7          "name": "Addition"
 8        },
 9        {
10          "name": "Subtraction"
11        },
12        {
13          "name": "Multiplication"
14        }
15      ]
16    }
17  ]
18}

Create a new JSON file named lang.json. Use the content below:

 1{
 2  "Languages": [
 3    {
 4      "topic": "Comprehension",
 5      "data": [
 6        {
 7          "name": "English"
 8        },
 9        {
10          "name": "Spanish"
11        },
12        {
13          "name": "French"
14        }
15      ]
16    }
17  ]
18}

Merge multiple JSON files

With the files defined, we can use jq to perform a merge on the two files.

Merge the files using the command below:

1jq -s '.' *.json > test-one.json

Which will provide an output similar to that below:

 1[
 2  {
 3    "Maths": [
 4      {
 5        "topic": "Numerics",
 6        "data": [
 7          {
 8            "name": "Addition"
 9          },
10          {
11            "name": "Subtraction"
12          },
13          {
14            "name": "Multiplication"
15          }
16        ]
17      }
18    ]
19  },
20  {
21    "Languages": [
22      {
23        "topic": "Comprehension",
24        "data": [
25          {
26            "name": "English"
27          },
28          {
29            "name": "Spanish"
30          },
31          {
32            "name": "French"
33          }
34        ]
35      }
36    ]
37  }
38]

If this output suits, awesome - you are done. Personally I dont like unlabelled arrays, within JSON.

If you would like the output to be more practical, then read on.

Merge and format multiple JSON files

The JSON files contain arrays of information, so it might make sense to add another layer to the output. In this section, I want to add another array to represent the lessons.

To achieve this, you need to tell jq that the content should be wrapped. In the example below, I create a new element lessons and map the original values into to it.

Merge the files using the command below:

1jq -s '. | { "lessons": map(.) }' *.json > test-two.json

Which will provide an output similar to that below:

 1{
 2  "lessons": [
 3    {
 4      "Languages": [
 5        {
 6          "topic": "Comprehension",
 7          "data": [
 8            {
 9              "name": "English"
10            },
11            {
12              "name": "Spanish"
13            },
14            {
15              "name": "French"
16            }
17          ]
18        }
19      ]
20    },
21    {
22      "Maths": [
23        {
24          "topic": "Numerics",
25          "data": [
26            {
27              "name": "Addition"
28            },
29            {
30              "name": "Subtraction"
31            },
32            {
33              "name": "Multiplication"
34            }
35          ]
36        }
37      ]
38    }
39  ]
40}

Hopefully the above is useful and either one of the solutions works for you!