Sources¶
iq connects only through saved sources, a named connection you register
once, then select by name or as the default.
Configuration
Sources live in a TOML file at <os user config dir>/iq/iq.toml (for example
~/.config/iq/iq.toml), written 0600 because a URI can carry a password.
Override the path with IQ_CONFIG, or per run with the global
--config <path> flag (which wins over IQ_CONFIG).
By default, a source added with a password keeps no password in this file.
The password lives in the OS keyring (Secret Service on Linux, Keychain on
macOS, Credential Manager on Windows). iq splices it back into the URI
only when connecting.
Add add¶
iq add <uri> [flags]
Register a source from a connection URI. The URI is the only positional argument.
Each driver supports a different set of URI parameters (see Drivers).
| short | long | default | description |
|---|---|---|---|
-a |
--active |
✗ | make the new source the active source |
-d <string> |
--driver <string> |
auto-detect | expected backend driver. It must match the URI scheme |
-n <string> |
--handle <string> |
the keyspace the URI names | handle for the source, derived from the URI when omitted |
-p |
--password |
✗ | prompt for the URI password or read it from stdin |
--skip-verify |
✗ | skip the post-add reachability check | |
--store <string> |
keyring |
where the URI's password is kept, keyring (OS keyring) or inline (in the config file) |
iq add -a 'cassandra://localhost:9042/shop?table=orders'
iq add -p 'mongodb://iq@localhost:27018/iq?collection=books'
iq add -p 'mongodb://root@localhost:27018/iq?authSource=admin&collection=books'
When the URI has a password, iq add puts the password in the OS keyring and
saves the URI without it. When no keyring is available, for example on a
server with no Secret Service, iq add saves the password in the config file
and prints a warning. Use --store inline to save the password in the config
file without the warning. Use --store keyring to make a missing keyring an
error.
A config file can have a keyring ID, a field named keyring_id that holds 16
random hex characters. The keyring entry of a source is named by the ID and the
handle, so two config files that use the same handle keep separate entries.
The first command that writes the keyring in a file gives the file its ID. That
command copies the existing entry of each keyring-backed source of the file to
its new name, then saves the ID. A command that only reads never changes the
file. A file with no ID uses the handle alone as the entry name, as older
versions of iq did.
A read in a file with an ID looks for the new name first. If that entry does
not exist, it reads the old handle-only entry. Writes and deletes touch the new
name only. The old entries stay, because another config file without an ID can
still use them. To delete them, run iq config keyring prune --legacy.
If the file already holds a password for the handle, iq add stops with an
error and does not replace the password. Choose another handle with -n. The
check and the write are two keyring calls, and iq saves the config file
without a lock. So two commands that edit the same config file at the same time
can overwrite each other.
If you copy a config file, the copy keeps the keyring_id of the original. The
two files then share their keyring entries. To give the copy its own entries,
delete the keyring_id line from the copy and run a keyring write in it.
Escaping the source URI
If the URI contains ? (and &), your shell can interpret it as a
wildcard, operator or separator. As a result, you must escape it (for
example iq add 'mongodb://localhost:27017/iq?collection=books').
"add" shadowing
iq add shadows jq's built-in add filter at the top level. To sum with
jq, write it inside a larger expression, for example iq '[ .a, .b ] | add'.
Group group¶
iq group [name] [flags]
Show, set, or clear the active group.
A / in a name groups sources (prod/books, dev/books). Set an active group with
iq group prod. An unqualified name then resolves inside it. For example, iq src books selects
prod/books. If the group has none, iq src books falls back to a top-level books.
| short | long | default | description |
|---|---|---|---|
--clear |
✗ | clear the active group |
Listing groups
You can list the already created groups with iq ls -g.
List ls¶
iq ls [group] [flags]
List saved sources, the active one marked with *.
An optional [group] limits the listing to sources in that group.
| short | long | default | description |
|---|---|---|---|
-v |
--verbose |
✗ | show a header and a FORMAT (a file source's detected dump format) and an OPTIONS (the source's stored option defaults) column |
-g |
--group |
✗ | lists groups instead of sources |
-j |
--json |
✗ | emit machine-readable JSON output |
-y |
--yaml |
✗ | emit machine-readable YAML output |
--reveal |
✗ | prints a password stored inline in the config verbatim | |
--expand |
✗ | resolves a keyring-backed source's stored password and inlines it (combine with --reveal to print verbatim) |
Move mv¶
iq mv <old> <new> [flags]
Rename a source to a new full handle, or move it into a group by giving a
group-qualified target (iq mv books prod/books).
When <old> names a group, every source under it is re-prefixed
(iq mv prod staging renames prod/* to staging/*).
The active source and group follow the move. A keyring-backed source's stored credential moves with it.
Ping ping¶
iq ping [name...] [flags]
Open each source and round-trip a cheap command (for example Redis PING or
MongoDB {ping:1}), reporting its driver and the round-trip time, or the error.
With no arguments, iq ping pings the active source. Otherwise, each argument
is a source handle or a group (pinging every member). --all pings every saved
source and takes no arguments.
--timeout bounds each check. Exits with a non-zero exit code if any
source is unreachable.
| short | long | default | description |
|---|---|---|---|
--all |
✗ | ping every saved source, rejects source arguments | |
--timeout |
5s |
per-query timeout |
Remove rm¶
iq rm <name>... [flags]
Remove saved sources or whole groups. Each argument is a source handle or a group name (which removes every source under it).
The removal is atomic. If any argument names neither a source nor a group, nothing is removed. A keyring-backed source's stored credential is deleted too.
Show/Set src¶
iq src [name] [flags]
Show or set the active source.
--src is global. Every command accepts it (see
Global flags).
Once a source is active, every query runs against it. Select a different source for a single
command with --src/-s, without changing the active one. Address a MongoDB collection or a
Cassandra table with a dotted handle.collection / handle.table suffix. With no active source
and no --src, the command errors. There is no ambient URI or environment fallback.
Per command source selection
You can run commands against a specific source which can be different from
the active one, for example iq --src books '.["2"]'.
Inspect inspect¶
iq inspect [source] [flags]
Show a source's native server/database introspection.
The positional argument names the source, like iq inspect books. With none, it
uses --src or the active source.
Certain sources accept sq-style addressing (for example MongoDB
<source>.<collection> or Cassandra <source>.<table>) to pick the
keyspace, overriding the default in the source's URI (for example
?collection= or ?table=).
| short | long | default | description |
|---|---|---|---|
--list |
✗ | list the subcommands/sections available for the source | |
--only <strings> |
✗ | narrow to these sections or subcommands | |
--reveal |
✗ | print an inline-stored password verbatim in the location header instead of redacting it | |
--expand |
✗ | resolves a keyring-backed source's stored password and inlines it (combine with --reveal to print verbatim) |
|
-j |
--json |
✗ | emit machine-readable JSON |
-y |
--yaml |
✗ | emit machine-readable YAML |
iq inspect shop --only dbStats,serverStatus
Diff diff¶
iq diff <a>[=<jq>] <b>[=<jq>] [flags]
Compare two saved sources across the layers a schemaless store can meaningfully
compare. Selecting no layer defaults to --data. Layers combine.
Each side can carry a jq filter, so a diff can be scoped to part of a
keyspace. Give it per side as <source>=<jq>, or for both sides at once with
--filter. A side's own filter takes precedence.
Exits non-zero when the sources differ and zero when they match
(diff(1)-style2), so scripts can branch on the exit status. Bounded by
--timeout.
| short | long | default | description |
|---|---|---|---|
--data |
✓ | diff items key by key (cross-driver allowed, for example MongoDB _id and Redis key) |
|
--filter <string> |
none | jq filter you root at .[], scoping both sides. A spec's own source=<jq> overrides it for that side |
|
--patch |
✗ | emit an RFC 6902 JSON Patch1 that transforms the left source into the right (single layer only). For --data, the pointers read /<key>/<field> over the whole keyspace map. --patch excludes --json/--yaml/--set-arrays |
|
--schema |
✗ | diff an inferred field/type shape (cross-driver allowed) based on a sample (use --sample to change the sample size) |
|
--sample <int> |
1000 |
max items sampled per side for --schema (0 = all) | |
--unbounded |
✗ | permit a Redis key or an RDB dump key with more than 1,000,000 elements, or an HBase row with more than 1,000,000 cells in one column family, to be loaded into memory | |
--stats |
✗ | diff native introspection trees (same driver only). Use with --section to narrow it down |
|
--section <strings> |
full set | introspection section(s) for --stats (comma-separated or repeatable) |
|
--set-arrays |
✗ | compare arrays order-insensitively as multisets (duplicates counted). Membership deltas are reported at the array's own path with no index segment. A pure reorder becomes no difference | |
-j |
--json |
✗ | emit machine-readable delta in JSON |
-y |
--yaml |
✗ | emit machine-readable delta in YAML |
iq diff prod staging --filter '.[] | select(.status == "new")'
iq diff 'prod=.[] | select(.type == "order")' 'staging=.[] | select(.kind == "ORDER")'
iq diff 'prod=.["orders:42"]' 'staging=.["orders:42"]'
Filtering
Write the filter .[]-rooted, as on a plain query (iteration is not
implicit here, unlike --insert). There are two reasons. A keyed diff
needs each item to keep its key. A pushable select(...) narrows the read
at the backend rather than merely narrowing the report.
iq diff prod staging --filter '.[] | select(.status == "new")'
A filter can instead name a single key (prod=.["orders:42"]) to compare
one document. An absent key then reports as removed rather than as a
change to null.
A filter that collapses the keyspace (keys, map(...)) or fans one item
out into several values (.[] | .tags[]) is refused. Neither leaves a key
to match on.
Data diff
--data without filtering reads both keyspaces fully into memory. As a
result, it costs memory proportional to the two sources. This is a
deliberate tradeoff, because an added/removed diff needs both key sets at
once.
A filter narrows that read, and a pushable one narrows it at the backend.
Cross driver can be useful for verifying a migration. But the identity match is only as meaningful as the keys lining up. It is a power-user tool, not a schema comparison.
Schema diff
--schema used across drivers gives each field path (with []
array-element and {} map-value wildcards) a canonical type
(integer/number/string(fmt)/map/array) and a required/optional
presence. As a result, it measures logical shape rather than sampling luck
and stays quiet under resampling.
The shape is sampled (with size set with --sample) and inferred, never
declared, so a wider sample yields a truer shape.
Two backends that genuinely normalize a native type differently (a timestamp as an RFC33393 string vs an epoch number) still diff. That is the JSON each serves back. The format tags make the row legible rather than mysterious.
Arrays are aligned by a longest common subsequence, so a single insertion reports one addition rather than a cascade at every later index.
Schema schema¶
iq schema [source[=<jq>]] [flags]
Sample a source and project a schema inferred from its values (the same
inference diff --schema uses).
Unlike inspect, which shows a backend's native
introspection, schema infers a driver-agnostic shape, never declared. The
field/type structure is sampled (with a sample size defined with --sample)
from the values themselves. As a result, a wider sample yields a truer shape.
It describes values, not keys. A non-object keyspace is legal (a string
keyspace emits {"type":"string"}). A filter that fans one item out into
several values is fine here, even though a keyed diff must
refuse it.
| short | long | default | description |
|---|---|---|---|
--filter <string> |
none | jq filter scoping which items the shape is inferred from. The spec form source=<jq> sets it per source |
|
--format <string> |
jsonschema |
picks the contract dialect the shape projects into: JSON Schema draft 2020-124 (jsonschema) or Open Data Contract Standard v3.2.05 (odcs) |
|
--sample <int> |
1000 |
max items sampled (0 = all) | |
--unbounded |
✗ | permit a Redis key or an RDB dump key with more than 1,000,000 elements, or an HBase row with more than 1,000,000 cells in one column family, to be loaded into memory | |
-y |
--yaml |
✗ | emit YAML instead of JSON (jsonschema only, odcs is always YAML) |
iq schema 'dev.orders=.[] | select(.active)'
iq schema dev.orders > orders.schema.json
iq schema dev.orders --format odcs > orders.odcs.yaml # emit an ODCS v3.2.0 contract
Scoped schema
A source=<jq> spec (or --filter) scopes which items the shape is
inferred from. For example, iq schema 'prod=.[] | select(.active)'
describes only the active ones. The sample cap then applies to the
survivors, so a selective filter walks further into the keyspace to fill
it.
JSON Schema
JSON Schema draft 2020-12 (jsonschema, the default) can be used for
interop with code generators, like
quicktype.
iq schema prod.orders > orders.schema.json && quicktype -s schema
orders.schema.json -l go
Open Data Contract Standard
Open Data Contract Standard v3.2.0 (odcs) is a YAML data contract for
tools like
datacontract-cli,
Soda
and Great Expectations.
It has eleven logical types, with no binary or decimal member. iq never
writes vector, because sampled values cannot prove an embedding.
A date-time becomes logicalType: timestamp with no format option. A
calendar date becomes logicalType: date with the format yyyy-MM-dd. A
UUID stays string with the uuid format. Base64-binary and exact-decimal
values stay plain string.
An id-keyed object becomes logicalType: map with a string key and the
value element. A string field with a small set of repeated values gets an
enum.
A field is required only when every object holds it and it never held
null. The root must hold objects. If the sample holds other values, such as
strings or arrays, iq schema --format odcs fails and writes no output.
Use --format jsonschema for those sources.
The contract's identifiers (id, name, schema-object name) derive
deterministically from the source handle and keyspace. They use no
timestamps or random ids.
-
JSON Patch defines a JSON document structure for expressing a sequence of operations to apply to a JavaScript Object Notation (JSON) document. It is suitable for use with the HTTP PATCH method. The "application/json-patch+json" media type is used to identify such patch documents. https://datatracker.ietf.org/doc/html/rfc6902 ↩
-
diff(1)is a classic Unix command-line utility that compares two files (or directories) line by line and outputs the differences between them. https://pubs.opengroup.org/onlinepubs/9699919799/utilities/diff.html ↩ -
Date and time format for use in Internet protocols that is a profile of the ISO 8601 standard for representation of dates and times using the Gregorian calendar https://datatracker.ietf.org/doc/html/rfc3339 ↩
-
JSON Schema is a declarative language for defining structure and constraints for JSON data. https://json-schema.org/draft/2020-12 ↩
-
The Open Data Contract Standard (ODCS) is an open-source, vendor-neutral specification (maintained under the Linux Foundation's LF AI & Data) for defining data contracts in machine-readable format (YAML or JSON). https://bitol-io.github.io/open-data-contract-standard ↩