Cookbook¶
iq renders JSON-family formats only, no CSV and no tables, by design. Its
output is meant to compose, so you can pipe it into other tools.
Piped input¶
Stdin becomes the source only when nothing else resolves. An active source wins over the pipe.
iq reads the pipe once, as a stream. It does not hold the whole input in
memory. A command that must read the data twice fails with piped stdin can be
read only once; save it to a file and read the file instead. In that case, save
the data to a file and read the file.
--from-format forces the decode when the content cannot be sniffed (a gzipped
or oddly-shaped dump).
Compare piped data with a source¶
The pipe is one source only. iq diff, iq combine, and source() name
registered sources, so they cannot read the pipe. Write the pipe to a fixed file
path and register that path once. Then rewrite the file before each run.
iq add needs the file to exist, so write it first. Plain JSON rows from
sq -J are not in iq's typed dump format. iq --typed --key-field id converts
them to typed records keyed by id. Without --key-field, iq --typed rejects
the rows with record has no key.
The examples set pipefail. Without it, a pipeline reports only the status of
iq. A failed sq or curl can then leave a partial file, and the next
command reads it. pipefail needs bash, zsh, or another shell that supports it.
Keep the file in a directory that only you can write to, such as your home
directory. Do not use a shared directory such as /tmp. There, another user
can create a symbolic link at the path, and the redirect then overwrites the
file that the link points to.
set -o pipefail
sq -J @pg.users | iq --typed --key-field id > "$HOME/in.jsonl" &&
iq add -n in "file://$HOME/in.jsonl"
set -o pipefail
sq -J @pg.users | iq --typed --key-field id > "$HOME/in.jsonl" && iq diff in prod
set -o pipefail
curl -sf https://api.example.com/users | iq --typed --key-field id > "$HOME/in.jsonl" &&
iq combine --with '$in + $prod' 'in=.[]' 'prod=.[]'
For a diff, the keys must match the keys of the other source.
Because in is a registered file source, iq reads it from disk page by page.
The data and schema diffs and every combine feature work on it. A file source
has no native statistics, so iq diff --stats does not.
The file changes on every run, so a decode cache of it is never used again. For
a large file, add --no-cache, so that each run does not leave an old cache
entry behind.
Scripts that run at the same time must use different paths and handles.
sq¶
sq is a command-line tool giving jq-style access to SQL
databases and files like CSV or Excel.
SQL → NoSQL¶
sq emits one JSON object per row (-J). iq keys each by the named field.
Foreign JSON always needs --key-field or --key (see
Key mapping).
NoSQL → SQL¶
Piped data is sq's .data table, and the destination table is created if
missing.
CSV or Excel¶
iq --src books '.[]' --jsonl | sq -C .data
iq --src books '.[]' --jsonl | sq -x .data -o books.xlsx
iq itself renders no CSV. sq is the tabular bridge.
quicktype¶
quicktype generates strongly-typed
models and serializers from JSON, JSON Schema, TypeScript, and GraphQL queries.
iq --src books '.[]' --jsona | quicktype -l typescript --top-level Book -o book.ts
The schema-fed variant (iq schema … | quicktype -s schema, see
Schema) types the whole inferred shape. This one
types what a query returned. Pipe a bounded result, not a scan of a
huge keyspace.
gron¶
gron transforms JSON into discrete
assignments to make it easier to grep for what you want and see the absolute
path to it.
-G (grona) indexes results as json[N], so
gron's --ungron rebuilds a JSON array.
-g roots every result at json and is for grepping only.
Miller¶
mlr is like awk, sed, cut, join, and
sort for data formats such as CSV, TSV, JSON, JSON Lines, and
positionally-indexed.
JSONL → CSV¶
Drift guard¶
iq diff follows diff(1): exit 0 when the sources match, 1 when they differ.
As a result, it slots into a cron job or a pre-deploy check as-is.