Skip to content

Cookbook

iq renders JSON-family formats only, no CSV and no tables, by design. Its output is meant to compose, so you can pipe it into other tools.

Piped input

Implicit stdin source
cat dump.jsonl | iq '.[]'

Stdin becomes the source only when nothing else resolves. An active source wins over the pipe.

iq reads the pipe once, as a stream. It does not hold the whole input in memory. A command that must read the data twice fails with piped stdin can be read only once; save it to a file and read the file instead. In that case, save the data to a file and read the file.

--from-format forces the decode when the content cannot be sniffed (a gzipped or oddly-shaped dump).

Compare piped data with a source

The pipe is one source only. iq diff, iq combine, and source() name registered sources, so they cannot read the pipe. Write the pipe to a fixed file path and register that path once. Then rewrite the file before each run.

iq add needs the file to exist, so write it first. Plain JSON rows from sq -J are not in iq's typed dump format. iq --typed --key-field id converts them to typed records keyed by id. Without --key-field, iq --typed rejects the rows with record has no key.

The examples set pipefail. Without it, a pipeline reports only the status of iq. A failed sq or curl can then leave a partial file, and the next command reads it. pipefail needs bash, zsh, or another shell that supports it.

Keep the file in a directory that only you can write to, such as your home directory. Do not use a shared directory such as /tmp. There, another user can create a symbolic link at the path, and the redirect then overwrites the file that the link points to.

One time: write the file, then register it
set -o pipefail
sq -J @pg.users | iq --typed --key-field id > "$HOME/in.jsonl" &&
  iq add -n in "file://$HOME/in.jsonl"
Diff piped data against a source
set -o pipefail
sq -J @pg.users | iq --typed --key-field id > "$HOME/in.jsonl" && iq diff in prod
Combine piped data with a source
set -o pipefail
curl -sf https://api.example.com/users | iq --typed --key-field id > "$HOME/in.jsonl" &&
  iq combine --with '$in + $prod' 'in=.[]' 'prod=.[]'

For a diff, the keys must match the keys of the other source.

Because in is a registered file source, iq reads it from disk page by page. The data and schema diffs and every combine feature work on it. A file source has no native statistics, so iq diff --stats does not.

The file changes on every run, so a decode cache of it is never used again. For a large file, add --no-cache, so that each run does not leave an old cache entry behind.

Scripts that run at the same time must use different paths and handles.

sq

sq is a command-line tool giving jq-style access to SQL databases and files like CSV or Excel.

SQL → NoSQL

Load a table from sq to iq
sq -J @pg.actor | iq --insert cache --key-field actor_id

sq emits one JSON object per row (-J). iq keys each by the named field. Foreign JSON always needs --key-field or --key (see Key mapping).

NoSQL → SQL

Load documents from iq to sq
iq --src books '.[]' --jsonl | sq --insert @dw.books .data

Piped data is sq's .data table, and the destination table is created if missing.

CSV or Excel

Render data into CSV or Excel with sq
iq --src books '.[]' --jsonl | sq -C .data
iq --src books '.[]' --jsonl | sq -x .data -o books.xlsx

iq itself renders no CSV. sq is the tabular bridge.

quicktype

quicktype generates strongly-typed models and serializers from JSON, JSON Schema, TypeScript, and GraphQL queries.

Create typed models with quicktype from live documents
iq --src books '.[]' --jsona | quicktype -l typescript --top-level Book -o book.ts

The schema-fed variant (iq schema … | quicktype -s schema, see Schema) types the whole inferred shape. This one types what a query returned. Pipe a bounded result, not a scan of a huge keyspace.

gron

gron transforms JSON into discrete assignments to make it easier to grep for what you want and see the absolute path to it.

Grep the flattened view then unflatten it
iq --src books '.[]' -G | grep -i title | gron --ungron

-G (grona) indexes results as json[N], so gron's --ungron rebuilds a JSON array. -g roots every result at json and is for grepping only.

Miller

mlr is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed.

JSONL → CSV

Convert JSONL to CSV with Miller
iq --src books '.[]' --jsonl | mlr --ijsonl --ocsv cat

Drift guard

Schema drift check
iq diff prod.orders staging.orders --schema || echo "schema drift"

iq diff follows diff(1): exit 0 when the sources match, 1 when they differ. As a result, it slots into a cron job or a pre-deploy check as-is.