Query data¶
The default action, a jq filter run against the active source (see
Sources).
The top-level paths name the keys to fetch. The result is printed as pretty JSON by default (see Output formats)
iq '[ .["book:1"].title, .["book:2"].title ]'
Escaping
Always wrap the filter in single quotes. jq syntax is full of characters
that the shell otherwise expands or splits: brackets ([ ]), whitespace,
|, *, $. Bracket-quoting a colon key like .["book:1"] reads as a
glob to zsh (no matches found) or bash unless quoted.
Cross-source queries¶
Both Compose and Combine reduce per source, then combine. Pick what the query needs.
Compose source() |
Combine iq combine |
|
|---|---|---|
| Shape | a single jq filter |
a positional spec per source, then one --with program |
| Correlated reads (B keyed by A's rows) |
✓ nest source() |
✗ specs are independent |
| Quoting | sub-filter is a quoted string inside the filter | each spec is its own argument |
| Memory | each reduced result held until combine, plus a correlated source() re-runs per row |
each reduced result held until combined |
Use source() when a read depends on another source's values, or to keep
everything in one composable filter.
Use iq combine for a straightforward join, union, or aggregate across a
few sources.
Compose source()¶
source("name"; "<jq>") runs <jq> against source name (reduced, streamed
and pushed down like any query) and yields its results as a stream. A
one-argument source("name") yields the whole source.
Both arguments of source() are strings, so they must be quoted. It also
yields a stream. Collect it before indexing with INDEX(source(…); .id) or
[source(…)], not source(…) | INDEX(.id).
A filter that calls source() runs over a null input. Every read is an
explicit source() call, and there is no implicit primary source. As a
result, it needs no active source. Names resolve through the registry like --src, active-group
namespacing included.
iq 'INDEX(source("users"; ".[]"); .id) as $u
| source("orders"; ".[] | select(.total > 99)")
| {name: $u[.userId].name, total}'
Correlated lookups re-run
A source() opened inside a stream runs its sub-filter once per element.
The connection is reused, but the sub-filter re-executes.
Hoist a constant lookup into a binding INDEX(source("users"; ".[]"); .id)
as $u | … and index $u per element instead.
Optimize memory usage
Binding source() to a jq variable materializes that call's whole result
set in memory (jq indexing needs a concrete array), even though the read
itself streams.
Push the reduction into the sub-filter, source("orders"; ".[] |
select(.total > 99)"), not source("orders"; ".[]") filtered outside, so
only the rows you need are held.
A bare source("big") over a large source buys no streaming benefit.
When each side is large and independent, prefer iq combine.
Combine combine¶
iq combine <source>[=<jq>]... --with <jq> [flags]
Query several sources and combine their results with one jq program.
Each positional is a source spec (<source>[=<jq>]) reduced at the source
(bounded reads, streaming scans, and predicate pushdown all still apply). The
spec's results bind to a jq variable named after the source, with /, .
and - becoming _ (so prod/books=.[] binds $prod_books). --with is
the final program, so it can join, union ($a + $b), aggregate, or fan across
any number of sources.
Each source reduces at the source, and a pushable filter pushes down. As a
result, this never copies whole datasets to join them. A spec with no filter
binds the whole keyspace, which is a whole-keyspace read. It needs
--unbounded, exactly as the same expression does on a plain query.
| short | long | default | description |
|---|---|---|---|
--no-compile |
✗ | disable server-side predicate pushdown. Run each spec's filter client-side | |
--with <string> |
required | final jq over the bound source results (each spec's results bound to $name), run over a null input |
iq combine 'users=.[] | {id, name}' \
'orders=.[] | select(.total > 99)' \
--with '($users | INDEX(.id)) as $u | $orders[] | . + {name: $u[.userId].name}'
Reduce, then combine
Each spec is evaluated independently. Its (already reduced) result is held
in memory before --with runs. For this reason, keep a stage's output
small with select/projection/aggregation.
A spec that must materialize its whole source (keys, ., map(...))
still needs --unbounded, exactly like a single-source query. A
.[]-rooted spec streams without it.
Unbounded --unbounded¶
Permit a filter that loads the whole dataset into memory. It also materializes a
.[]-rooted filter instead of streaming it. As a result, a filter that
collapses the keyspace into one value (keys, ., map(...)) runs only with
it (see Read strategies).
The flag also lets a read load a Redis key with more than 1,000,000 elements. Without it, the read fails
and names the key, its type, and its size (see Redis). It does the same for an
HBase row with more than 1,000,000 cells in one column family (see HBase). A key in an
RDB dump follows the same limit, and the flag lifts it there too (see File dumps). The same
flag does this for iq diff, iq schema, and the copy and move modes.
iq combine and iq data carry their own copies of the flag where they apply.
Client-side --no-compile¶
Disable pushdown.
| short | long | default | description |
|---|---|---|---|
--no-compile |
✗ | disable predicate pushdown. Run the full .[] | select() filter client-side |
Native form --native¶
Read each item in the native form of the source driver instead of normalized JSON.
DynamoDB, MongoDB and HBase have a native form today. DynamoDB JSON gives every attribute value a
type tag, for example {"price": {"N": "1.50"}}. MongoDB canonical Extended JSON tags each BSON
type, for example {"year": {"$numberInt": "2020"}}. The HBase raw form holds the row key, each
qualifier and each cell value as base64.
| short | long | default | description |
|---|---|---|---|
--native |
✗ | read items in the native form of the source (DynamoDB JSON, MongoDB Extended JSON or HBase raw bytes). No predicate is pushed down, and the filter must start with .[] |
iq --native --src books '.[] | select(.year.N == "2020") | .price'
iq --native --src mbooks '.[] | select(.year["$numberInt"] == "2020") | .title'
iq --native --src hbooks '.[] | .cf'
The filter sees the tags, so it reads .year.N and not .year. iq scans the whole table and runs
the filter on the client. --native works with --json, --jsona, --jsonl, and --yaml. See
Native form for the copy and the dump.