Get started¶
iq runs jq1 filters to query, dump, copy, diff and write data across NoSQL
databases, and their dump files, from a single static binary. See
Drivers for supported databases.
Why iq¶
- Query any supported store with one language. The same jq filter runs on Redis, MongoDB,
Cassandra, DynamoDB and the other drivers, and gives the same result on
each. One filter can also
join data from several sources, with
source()oriq combine. - A filter that names keys fetches only those keys. A
.[]filter scans the source in pages. - Query and compare dumps without a database server. Restore them to a live store.
- Compare two sources, or a source and a dump, by data, stats, or inferred schema.
iq diffreturnsdiff(1)exit codes, so a CI job can fail on drift. - Infer a schema from a sample of any source with
iq schema, as JSON Schema draft 2020-12 or an Open Data Contract Standard v3.2.0 contract. - Copy typed data across stores in one command, with no export script and no conversion script.
- Give an AI agent access to every store with one Agent Skill and one MCP server. The skill works with any agent that reads the Agent Skills format. The server is read-only by default.
iq normalizes fetched values to JSON and runs the filter on the client. The URI scheme2
chooses the backend. The selector walks the parsed jq AST5. Based on the AST, it
runs a bounded read, a streaming scan (with pushdown6), or a materialized scan.
See How it works.
iq is inspired by sq, whose command set it
deliberately follows to make the tool feel familiar. Read
Comparison for how iq relates to other tools.
Note
iq is built with AI assistance. Every change passes the full test suite,
container-backed integration tests for every backend, and a mutation gate
before it lands (see
CONTRIBUTING.md).
Feedback and bug reports are very welcome. Report security problems privately, see the security policy.
Installation¶
iq ships as a single static binary (no runtime dependencies, no CGO3).
Version & Location
The script downloads the release for your OS/arch, verifies its SHA-2564
against the release checksums, and installs the binary. IQ_VERSION
pins a version and IQ_INSTALL_DIR picks the target directory.
You can also download a .deb, .rpm, .apk, or Arch .pkg.tar.zst from the
releases.
Verify a release¶
Every release signs checksums.txt with a keyless cosign
signature from the release workflow, and carries SLSA build provenance for every
artifact (multiple.intoto.jsonl). The install script checks the SHA-256 of the
archive it downloads. To check a download yourself, first make sure that
checksums.txt comes from the iq release workflow:
cosign verify-blob checksums.txt \
--bundle checksums.txt.sigstore.json \
--certificate-identity-regexp '^https://github.com/zsltg/iq/\.github/workflows/release\.yml@refs/tags/v' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com
sha256sum --ignore-missing -c checksums.txt
To check the build provenance of an archive, use slsa-verifier:
slsa-verifier verify-artifact iq_0.37.1_linux_amd64.tar.gz \
--provenance-path multiple.intoto.jsonl \
--source-uri github.com/zsltg/iq --source-tag v0.37.1
Releases before v0.37.1 carry checksums.txt only, with no signature or
provenance.
Each binary archive carries a third-party-licenses/ tree with the license and
notice texts of every module that iq links. Each Linux package installs the
same texts in /usr/share/licenses/iq/third-party/. The source archive does not
contain these texts.
Building from source¶
Quick start¶
You need Docker, and iq on your PATH (see Installation). If the
installer prints a note about PATH, add the directory that the note names.
Run each command on its own. If a command fails, stop and read its error.
Start a disposable MongoDB that only this machine can reach:
Register the database as the active source. iq add connects to make sure that the source
works, and --timeout 30s gives MongoDB time to start:
Write two sample books. iq reads piped stdin as a source and takes each key from the id
field:
printf '%s\n' '{"id":1,"title":"Dune","year":1965,"price":9.99}' \
'{"id":2,"title":"Project Hail Mary","year":2021,"price":14.5}' |
iq --insert books --key-field id
Read one book by its key. iq fetches only that document:
The output is similar to the following:
Filter a scan, and show the query plan:
The output is similar to the following, where server filter shows the part of the filter
that MongoDB runs. The full filter then runs again on the client, so the result is the same
with or without pushdown:
query plan
source: books (mongo)
...
mongo calls:
find(<filter>): server-side pre-filter, then the full jq re-runs client-side
cursor streamed in batches of 100
server filter (mongo find):
{ — .year > 2015
"$or": [
{"year": {"$gt": 2015}},
{"year": {"$type": ["string", "array", "object"]}}
]
}
...
Without --explain, the same command prints {"price": 14.5, "title": "Project Hail Mary"}.
When you are done, remove the container and the source:
The basics¶
Register a source for each store you work with. Then make one of them active:
iq add -n orders 'mongodb://localhost:27017/shop?collection=orders'
iq add -n staging 'mongodb://staging:27017/shop?collection=orders'
iq add -n cache redis://localhost:6379/0
iq add -n snap file:///backups/prod.rdb
iq src orders
iq --src orders '.[] | select(.status == "new") | {id, total}'
Writes
Queries are read-only. --insert, --replace, iq data clear,
iq data drop and iq data delete write to the target. iq exec forwards
a native command to the database, so it can write too. Use --explain to
see the query plan or --dry-run to report the
effect of a write, without changing anything. iq exec has no dry run.
You can find detailed examples in Sources, Query data and Write data.
For more advanced usage check Output, Query plan, Cookbook and Loading exports.
For debugging, see Diagnostics & Logging.
Supported data sources are listed in Drivers.
Shell completions¶
The .deb, .rpm, .apk and .pkg.tar.zst packages install
Bash,
Zsh and fish completions for you.
For a brew, scoop,
go-install or source build,
iq completion <shell> prints a script to install by hand.
Completions cover the commands, their sub-subcommands and flags. They also cover
the saved source handles, groups, and config-option keys, which they read live
from your config. As a result, iq --src <TAB> offers the sources iq ls lists.
A flag that takes a closed
set offers that option's own values.
In zsh, fish, and PowerShell, a candidate shows a short description next to it.
Bash shows the descriptions when it lists more than one candidate. A source
handle shows its driver, its keyspace, and active for the active source, for
example prod/books mongo, orders, active. It never shows the host, the user,
or the URI. A group shows its number of sources, and a config key shows the help
text of its flag. To turn the descriptions off, generate the script with
iq completion <shell> --no-descriptions.
iq inspect --only <TAB> and iq diff --section <TAB> offer the
introspection subcommands of the selected source's backend, worked out from its
saved URI.
iq add <TAB> completes the connection URI part by part. It offers the schemes,
then the names of the URI options that iq itself reads, then the values of an
option that has a closed set of values. A shell gives ? and & a special
meaning, so type them with a backslash, for example
iq add cassandra://host/ks\?consistency=lo<TAB>, which offers local_quorum
and local_one. Completion does not work inside quotes, because the bash and
zsh completion scripts pass the quote character to iq. A quoted URI still runs
correctly. Completion does not offer the host, the path, or the options that a
backend SDK reads, such as authSource for MongoDB. It also offers nothing when
the typed text holds a password, so a password never appears in a candidate.
The jq filter itself is a program, not a completable value. As a result, iq
offers no candidates there (and never falls back to filenames). iq also offers
no candidates for the backend verb of iq exec and its operands.
Every completion is offline. It reads your config file and nothing else. As a
result, a <TAB> never opens a connection, never reads the OS
keyring7, and cannot hang. That is why a collection suffix does not
complete. iq --src shop.<TAB> offers nothing, because listing collections
needs a connection.
Man page¶
The .deb, .rpm, .apk, and .pkg.tar.zst packages also install an iq(1) manual page, so man iq works
after a package install. For any other install, pipe it into your man path:
-
jqis a widely-used command-line utility and very high-level, functional, domain-specific programming language designed for processing JSON data. https://jqlang.org ↩ -
RFC3986 proposes a generic URI syntax and a process for resolving URI references that might be in relative form, along with guidelines and security considerations for the use of URIs on the Internet. https://datatracker.ietf.org/doc/html/rfc3986 ↩
-
Cgo enables the creation of Go packages that call C code. https://pkg.go.dev/cmd/cgo ↩
-
SHA-256 is a Secure Hash Algorithm with a message digest size of 256. https://nvlpubs.nist.gov/nistpubs/fips/nist.fips.180-4.pdf ↩
-
An abstract syntax tree is the tree that a parser builds from the source of a program. Here, it is the parsed
jqfilter that the key selector inspects to decide how to read (see How it works). https://en.wikipedia.org/wiki/Abstract_syntax_tree ↩ -
Predicate pushdown hands part of the filter to the database, so that the database returns only matching items instead of everything for client-side filtering. Each driver page lists what the driver can push (see Drivers). https://en.wikipedia.org/wiki/Predicate_pushdown ↩
-
The operating system's credential store (macOS Keychain, Windows Credential Manager, the Secret Service on Linux), where
iq addkeeps source passwords by default (see Configuration). https://pkg.go.dev/github.com/zalando/go-keyring ↩