Skip to content

Write data

Data movement lives on the query command, like in sq. The jq filter is the transform. --insert names a destination. Piped stdin is an implicit source. There is no separate copy command. iq exec remains the untyped escape hatch for anything the typed path does not cover.

Insert --insert

With --insert/--typed, the filter transforms each item (its key is preserved). Iteration over the source is implicit, so you do not write .[].

short long default description
--insert <source> ✗ write each item into this destination source (copy/restore/import) instead of rendering
--typed ✗ emit typed {"key":…, "type":…, "value":…} records, a re-importable dump (needed for Redis, a document store's plain output already restores)
--native ✗ read each item in the native form of the source (DynamoDB JSON, MongoDB Extended JSON or HBase raw bytes). Needs a DynamoDB, MongoDB or HBase source. See Native form

Source → Source, key/id preserving
iq --src books --insert books2
Cross-driver (Redis → Mongo), object values only
iq --src cache --insert docs
Back up Redis losslessly (typed dump)
iq --src cache --typed -o dump.jsonl
Back up Mongo with plain output (self-describing)
iq --src books --jsonl -o dump.jsonl
Add a dump file, then restore it into a live source
iq add file:///dump.jsonl -n snap
iq --src snap --insert cache
Move plan, no connection
iq --src books --insert books2 --explain

Implicit iteration

This is the one place where iq reads a filter per item. Everywhere else (a plain query and the <source>=<jq> specs iq combine, iq diff and iq schema take), the filter is rooted at the whole keyspace, and you write .[] yourself.

The split is deliberate. A keyspace-rooted filter can aggregate across items ([.[] | .total] | add) and can name a single key (.["orders:42"]). A per-item filter can express neither. But when items arrive from piped stdin, the write path has no keyspace at all.

Nothing can tell the two apart automatically (.name means the key named "name" keyspace-rooted and the field name per item). For this reason, they stay separate rather than guessing.

Existing keys are overwritten (upsert) unless --no-overwrite makes the run insert-only. --replace empties the destination first (with confirmation, or --force).

A document store (Mongo, CouchDB, Couchbase, Elasticsearch) stores each value exactly as given and so requires it be a JSON object. A bare scalar (for example a Redis string value) is rejected with a hint rather than silently wrapped as {"value": …}. As a result, a successful copy round-trips exactly. Shape it explicitly first, for example iq 'if type == "object" then . else {value: .} end' --insert <dest>.

Typed format

--typed serializes the records in the chosen format: --jsonl (default), --json, --jsona, or --yaml. All of these re-import through a file:// source or a piped --insert. iq auto-detects them from content by their typed {"key":…, "type":…, "value":…} envelope.

The sniff reads the first complete record, at most 16 MiB. A first record over that size stops the read with an error. A .yaml/.yml name or an explicit ?format= / --from-format overrides the sniff (jsonl, yaml, mongoexport, bson, rdb, dynamodb-json, cassandra-csv, or neo4j-json). Aliases like json are accepted.

The renderings that cannot carry a record back are rejected rather than written. --raw (a scalar cannot hold the envelope), --format parquet (columnar) and --gron/--grona (flattened assignments no source decodes) drop --typed to grep or export the value stream instead.

A backend that stores text as JSON cannot keep a byte sequence that is not valid UTF-8. It would change each invalid byte to U+FFFD, and two different keys or names could become one. So Elasticsearch, OpenSearch, CouchDB, Couchbase, DynamoDB, and RedisJSON refuse a record with such a string. The error names the field path, and no part of that batch is written. iq data delete refuses such a key too. The source can hold these bytes in a Redis string, in an RDB dump, or in the output of a jq filter such as @base64d. Encode the value first, for example with @base64.

Redis has no empty hash, list, set, sorted set, or stream, and a native Redis value has no null. So a Redis write refuses an empty one of these types and a null in a native value. A RedisJSON value keeps its JSON null. The error names the key, and no part of that batch is written. Redis stores a number or a bool as its text, and a set keeps one copy of each member.

Native form --native

A normalized value can be plainer than the native type that it came from. A DynamoDB fractional number reads as an exact string, a binary value as base64 text, and a set as a list. A copy into DynamoDB then writes an S or an L. A MongoDB int32 reads as a number and writes back as an int64, and a date or a Decimal128 writes back as a string. An HBase cell that is not UTF-8 reads as base64 text and writes back as the bytes of that text. The copy does not restore the type.

--native reads each item in the typed JSON form of its driver instead. Today, DynamoDB, MongoDB and HBase have one. DynamoDB JSON looks like {"price": {"N": "1.50"}}. MongoDB canonical Extended JSON looks like {"year": {"$numberInt": "2020"}}. HBase raw bytes look like {"cf": {"dGl0bGU=": "RHVuZQ=="}}. Each record carries its form. A --typed dump holds it in an optional form field, for example {"key":"a","type":"item","form":"dynamodb-json","value":{…}}. A dump of normalized records has no form field, so old dumps read as before. A reader refuses a form that it does not know.

DynamoDB to DynamoDB, every type kept
iq --native --src books --insert books2
Native dump of a table, and the same table from an export file
iq --native --typed --src books -o books.jsonl
iq --native --typed --src 'file:///export.json.gz?format=dynamodb-json&keys=pk' -o books.jsonl
MongoDB to MongoDB, every type kept
iq --native --src mbooks --insert mbooks2
iq --native --typed --src mbooks -o mbooks.jsonl
HBase to HBase, every byte kept
iq --native --src hbooks --insert hbooks2
iq --native --typed --src hbooks -o hbooks.jsonl
Query with the type tags
iq --native --src books '.[] | select(.price.N == "1.50") | .title.S'

The rules of --native:

  • Only the destination that owns the form can write it. A DynamoDB destination reads a native record strictly and refuses a malformed type tag, a number that it cannot store exactly, or an empty set. A MongoDB destination reads a native record as canonical Extended JSON and refuses a malformed tag. It keeps the type of _id when the display form of _id equals the key. Any other destination refuses the record, and the error names the key and both forms. An HBase destination decodes a raw record strictly. It refuses a key, a qualifier or a cell value that is not canonical base64, a family that is not an object, and a cell that is not a string. The error names the key and the cell. A record in normalized JSON still goes to every destination.
  • A jq filter sees the typed form, so a filter that edits a value must keep the tags. A filter that breaks a tag makes the destination refuse the record.
  • No predicate is pushed down. iq scans the whole source and runs the filter on the client. A native read cannot fetch a named key, so the filter must start with .[]. --explain shows this.
  • --native works with --json, --jsona, --jsonl, and --yaml. The other renderings flatten the tags, so iq refuses them.
  • --native does not work with --key-field, because foreign JSON has no native form. It does not work with source(). iq diff and iq schema have no --native.
  • A source without a native form refuses --native. That includes a typed dump file. Read a typed dump without --native, because each of its records already names its form.
  • --native does not use the file decode cache. Each run reads the dump again.
  • A piped typed dump already names the form of each record. --native has no effect on it.
  • With --replace, iq clears the destination only after the first page passes the checks.

Key mapping --key*

On a plain copy, each item carries its source key, so no key flag is needed. Foreign JSON (piped stdin, a reshaped stream) has no natural key. --key-field takes it from an object field. --key computes it with a jq expression. --key-prefix prepends a namespace either way. A key is a string, a number, or a boolean. A float key must be a whole number and prints as plain digits (1000000, not 1e+06). A fractional, NaN, or infinite float key is refused. So is a float key whose value a float cannot hold exactly, such as 1e23. Use an integer or a string key.

iq combine --insert is the one write that always requires --key or --key-field. A combine's results come out of one program over a null input, so no value has a key to inherit. As a result, the run is refused up front rather than failing partway through a copy. For the same reason, combine has no --typed. A typed dump is a stream of {key,type,value} records and needs the same key.

The record key replaces the identity that the value carries. MongoDB, Elasticsearch, OpenSearch, and CouchDB drop the _id of the value and use the record key instead. CouchDB also drops the _rev of the value. DynamoDB writes the record key over the key attributes of the item. Neo4j merges on the record key and writes it as the key property. This is how --key, --key-field, and --key-prefix take effect. For a record with no key, MongoDB, Elasticsearch, OpenSearch, and CouchDB also drop the _id of the value, and the backend mints an identity. DynamoDB writes such a record as it is, including an _id attribute, and the item must hold the key attributes of the table.

CouchDB refuses a value that holds a _deleted or _attachments member. A read returns an attachment as a stub, so iq does not copy attachments.

A write flag used without --insert is an error, never silently ignored.

short long default description
--key <string> none jq expression yielding each written item's key (--insert/--typed)
--key-field <string> none object field to take each written item's key from (--insert)
--key-prefix <string> none string prepended to every written key (--insert/--typed)
Persist the joined rows into a third source
iq combine 'users=.[] | {id, name}' 'orders=.[] | select(.total > 99)' \
   --with '($users | INDEX(.id)) as $u | $orders[] | . + {name: $u[.userId].name}' \
   --insert joined --key '.userId | tostring' --key-prefix 'j:'

import foreign JSON from stdin, keyed by id
cat foreign.json | iq --insert books --key-field id
reshape + re-key while copying
iq '{t: .title}' --src books --insert kv --key '.t'

Type mapping --type

A plain copy carries each item's native type along (a Redis hash lands as a hash), so no type flag is needed. A filter that reshapes the value drops that type. The output is plain JSON, and --type names the native type that the destination stores it as.

Left unset, a typed destination infers it from the shape (Redis writes a scalar as a string and an object or array as json). As a result, --type is only required when you want something else:

  • A hash built from an object
  • A list or set from an array
  • A zset from [{member, score}] pairs.

A document store ignores it. Every value is a document there. With --typed, the same flag stamps the type field of each dump record instead.

short long default description
--type <string> none native type stamped on each written value, for example hash, list, json (--insert/--typed)

Reshape Mongo documents into Redis hashes
iq '{title, year: (.year | tostring)}' --src books --insert cache --type hash
Project one field per item into a Redis list
iq '[.tags[]]' --src books --insert cache --type list --key-prefix 'tags:'
Same reshape, no --type: an object lands as RedisJSON
iq '{title, year}' --src books --insert cache
Dump reshaped items with an explicit type tag
iq '{title}' --src books --typed --type json -o titles.jsonl

Replace --replace

By default, a write upserts. Existing keys are overwritten, and every other key in the destination stays. For a key that exists, the result depends on the driver. MongoDB, Elasticsearch, OpenSearch, CouchDB, Couchbase, DynamoDB, Redis and Neo4j replace the whole value of the key. Cassandra and HBase merge. They write the columns or cells that the record holds, and the other columns or cells of the row stay. --replace turns the copy into a restore. It empties the destination first (the same operation as iq data clear, a Redis FLUSHDB, a Mongo deleteMany({})) and then writes. As a result, the destination ends up holding exactly the values of the source. In Neo4j, the clear also deletes the relationships of the nodes, and the copy writes only the nodes of the label with their properties. So relationships and other labels are not restored.

Because it destroys data, it prompts (clear <destination> before writing). --force answers yes. --dry-run reports the copy without clearing anything.

A source cannot be its own --replace destination. The copy reads the source after the clear, so it would find nothing. iq refuses the run, also with --dry-run, before it opens the destination, so nothing is cleared or written. To rewrite a source in place, dump it first with --typed -o <file>, register the file as a source, and insert from that file.

iq compares what the two URIs name, not how they are spelled. Two handles for redis://h and redis://h:6379/0 name one container, so iq refuses them. The comparison ignores user info and the case of the host. It also treats localhost, 127.0.0.1, and ::1 as one host, and fills in the default port. Two host lists match when they share one host. The check compares the URI text and does not resolve names. It can miss two names for one server, such as two DNS names, a CNAME, or mongodb+srv://h and mongodb://h. Treat it as a safety net.

A destination that cannot be cleared (the read-only file dump) is refused up front. --no-overwrite is the opposite choice, insert-only, so the two are mutually exclusive. Neither applies to a --typed dump.

short long default description
--replace ✗ empty the destination before writing, with confirmation (--insert)
--force ✗ skip the confirmation prompt for --replace
--no-overwrite ✗ skip keys that already exist (--insert)

Restore a dump so the destination matches it exactly
iq --src snap --insert cache --replace
Same, unattended (no prompt)
iq --src snap --insert cache --replace --force
Preview the restore: reports the copy, clears nothing
iq --src snap --insert cache --replace --dry-run
Fill gaps only, never touch an existing key
iq --src books --insert books2 --no-overwrite

Lifecycle previews

Every iq data subcommand (delete, clear, drop) shares two previews. clear and drop also prompt before destroying data. --force skips the prompt.

short long default description
--explain ✗ print the access plan and exit without connecting or changing anything
--dry-run ✗ connect and report the real effect without changing anything

Plan only: which operation, and whether the driver supports it
iq data drop cache --explain
Connect and count what a clear would remove, then stop
iq data clear shop.orders --dry-run
Check which of the named keys exist before deleting
iq data delete cache book:1 book:2 --dry-run

Delete data delete

iq data delete <target> <key> removes a named set of keys, keeping the container. It is the typed, capability-gated, explainable counterpart of a raw per-key delete (HBase's exec delete verb, a Redis DEL).

Each key uses the same spelling as a Get: a bare string (book:1) or a JSON array for a composite key (["shop",42]). A key already absent is not an error (delete is idempotent). The report states it (deleted N key(s), M already absent).

Unlike data clear/data drop, it does not prompt. The explicit key list you typed is the confirmation (use --dry-run to preview). A backend with no per-key identity (the read-only file dump) rejects it (like Redis rejects data drop).

"deleted 2 key(s), 0 already absent"
iq data delete cache book:1 book:2
a composite-key row, by its JSON-array spelling
iq data delete shop.orders '["eu",42]'

Clear data clear

Empties a container but keeps it (for example MongoDB deleteMany({}) or Redis FLUSHDB). Both this and iq data drop are distinct from iq rm, which only unregisters a saved source. These destroy stored data, and prompt for confirmation unless --force is used.

Empty a Redis source called "cache" (FLUSHDB)
iq data clear cache
Empty two MongoDB collections "shop.orders" and "shop.users"
iq data clear shop.orders shop.users

Drop data drop

Removes the container entirely and prompts for confirmation unless --force is used. Sources without a droppable container are rejected (for example Redis).

Reports drop as unsupported for Redis source called "cache"
iq data drop cache --explain
Drop one Mongo collection called "shop.orders"
iq data drop shop.orders
Drop containers "shop.orders" and "shop.users"
iq data drop shop.orders shop.users

Dry run --dry-run

Reports the effect without writing. It does everything except the final mutation. It connects, opens source and destination, and runs the real scan. It applies the real transform and probes capabilities. Then it suppresses the write.

short long default description
--dry-run ✗ report the effect of --insert without writing anything

Dry run for a copy into "books2"
iq --src books --insert books2 --dry-run
Dry run for clearing source "books"
iq data clear books --dry-run