Boost your Productivity with JQ | Play with JSON in unique way

ওয়েব ডেভেলপমেন্ট
,
1

A few days back, I conducted a session in the office on boosting our productivity using JQ when we work with large JSON files. Writing here as well. Hope it will help some developers.

Boosting Productivity with JQ
JQ is a command-line JSON processor, which could be helpful when we playing with JSON in our day-to-day work.

Inside JQ

  • CLI tool to manipulate, filter, and transform JSON data.
  • Written in C, no external dependencies.
  • Works right from the terminal, no setup needed on many systems
  • A lightweight, flexible command-line JSON processor
  • jq 1.3 released on 19 May 2013

What JQ Can Do?

  • Parse & Format JSON: Clean, readable output with proper indentation
  • Filter & Transform: Extract specific data points from complex structures
  • Map & Reduce: Apply transformations across arrays
  • Combine & Merge: Join data from multiple sources
  • Validate & Test: Check JSON structure validity
  • Calculate & Analyze: Perform operations on numeric data

When & Where To Use

  • Debug API responses: Instant filter & format
  • CI/CD scripts: Automate condition checks
  • CI/CD scripts: Automate condition checks
  • Log inspection: Extract nested values
  • JSON diff or audit: Clean, scriptable comparisons
  • Data transformation: Map, reduce, filter easily

What am I gonna do?
Example of how to play with JQ

  • I have created a Node.js application
  • Two API that return JSON data
  • CURL those APIS using JQ
  • Also, read the JSON file directly

Let’s say we have this JSON

[
    {
        "name": "ABC",
        "email": "[email protected]",
        "age": 51,
        "gender": "Male"
    },
    {
        "name": "ABC",
        "email": "[email protected]",
        "age": 53,
        "gender": "Male"
    },
    {
        "name": "CDE",
        "email": "[email protected]",
        "age": 34,
        "gender": "Male"
    },
    {
        "name": "CDE",
        "email": "[email protected]",
        "age": 60,
        "gender": "Female"
    },
    {
        "name": "FGH",
        "email": "[email protected]",
        "age": 19,
        "gender": "Male"
    },
    {
        "name": "HIJ",
        "email": "[email protected]",
        "age": 68,
        "gender": "Male"
    },
    {
        "name": "KLI",
        "email": "[email protected]",
        "age": 24,
        "gender": "Male"
    },
    {
        "name": "MNO",
        "email": "[email protected]",
        "age": 58,
        "gender": "Male"
    },
    {
        "name": "PQRS",
        "email": "[email protected]",
        "age": 46,
        "gender": "Male"
    },
    {
        "name": "TVU WXYZ",
        "email": "[email protected]",
        "age": 60,
        "gender": "Female"
    }
]

** BASIC USE CASE**
Example 1: This is only an example, but think about it, if we have a large-scale dataset and we need to find out unique properties, we can easily use that command.

  • Copy this JSON and create a JSON file. (users.json) on your local PC
  • Run this command on your terminal: jq 'map(.gender) | unique' users.json
  • Observe the output, since we have only two genders, it will print
  • [ "Female", "Male" ]

Example 2: Let’s say you wanna print the length of the JSON

  • Run this command on your terminal: jq '. | length' users.json
  • Observe the output, since we have 10 datasets, it will print: 10

*Example 3: Not only directly file we can also play with JQ when we curl the API Let’s say this API returns the same data as the response that we have in our users.json file: *
curl http://localhost:3000/api/users

  • To run this example, create a sample NodeJS app on your local and return the JSON file with localhost and port 3000:

  • Run this command on your terminal: curl http://localhost:3000/api/users | jq '. | length'

  • Observe the output, since we have 10 datasets, it will print: 10

Example 4: Similarly with curl, let’s select some specific data

  • This step remains the same: To run this example, create a sample NodeJS app on your local and return the JSON file with localhost and port 3000:

  • Run this command on your terminal: curl http://localhost:3000/api/users | jq '.[] | {name, gender}'

  • Observe the output; it will print only the name and gender.

Apart from that, I also created a GitHub Repo to show advanced JQ technique, if you are interested can find this here: Play With JQ - Meher Ullah Khan Raj

Resources for Learning More

1 Like