Write data¶
Data movement lives on the query command, like in sq. The jq filter is the
transform. --insert names a destination. Piped stdin is an implicit source.
There is no separate copy command. iq exec remains the untyped escape
hatch for anything the typed path does not cover.
Insert --insert¶
With --insert/--typed, the filter transforms each item (its key is
preserved). Iteration over the source is implicit, so you do not write .[].
| short | long | default | description |
|---|---|---|---|
--insert <source> |
✗ | write each item into this destination source (copy/restore/import) instead of rendering | |
--typed |
✗ | emit typed {"key":…, "type":…, "value":…} records, a re-importable dump (needed for Redis, a document store's plain output already restores) |
|
--native |
✗ | read each item in the native form of the source (DynamoDB JSON, MongoDB Extended JSON or HBase raw bytes). Needs a DynamoDB, MongoDB or HBase source. See Native form |
iq add file:///dump.jsonl -n snap
iq --src snap --insert cache
Implicit iteration
This is the one place where iq reads a filter per item. Everywhere else
(a plain query and the <source>=<jq> specs iq combine, iq diff and
iq schema take), the filter is rooted at the whole keyspace, and you
write .[] yourself.
The split is deliberate. A keyspace-rooted filter can aggregate across
items ([.[] | .total] | add) and can name a single key
(.["orders:42"]). A per-item filter can express neither. But when items
arrive from piped stdin, the write path has no keyspace at all.
Nothing can tell the two apart automatically (.name means the key named
"name" keyspace-rooted and the field name per item). For this reason,
they stay separate rather than guessing.
Existing keys are overwritten (upsert) unless --no-overwrite makes the
run insert-only. --replace empties the destination first (with
confirmation, or --force).
A document store (Mongo, CouchDB, Couchbase, Elasticsearch) stores each
value exactly as given and so requires it be a JSON object. A bare scalar
(for example a Redis string value) is rejected with a hint rather than
silently wrapped as {"value": …}. As a result, a successful copy
round-trips exactly. Shape it explicitly first, for example
iq 'if type == "object" then . else {value: .} end' --insert <dest>.
Typed format
--typed serializes the records in the chosen format: --jsonl (default),
--json, --jsona, or --yaml. All of these re-import through a
file:// source or a piped --insert. iq auto-detects them from content
by their typed {"key":…, "type":…, "value":…} envelope.
The sniff reads the first complete record, at most 16 MiB. A first record
over that size stops the read with an error. A .yaml/.yml name or an
explicit ?format= / --from-format overrides the sniff (jsonl, yaml,
mongoexport, bson, rdb, dynamodb-json, cassandra-csv, or
neo4j-json). Aliases like json are accepted.
The renderings that cannot carry a record back are rejected rather than
written. --raw (a scalar cannot hold the envelope), --format parquet
(columnar) and --gron/--grona (flattened assignments no source
decodes) drop --typed to grep or export the value stream instead.
A backend that stores text as JSON cannot keep a byte sequence that is not valid
UTF-8. It would change each invalid byte to U+FFFD, and two different keys or
names could become one. So Elasticsearch, OpenSearch, CouchDB, Couchbase,
DynamoDB, and RedisJSON refuse a record with such a string. The error names the
field path, and no part of that batch is written. iq data delete refuses such
a key too. The source can hold these bytes in a Redis string, in an RDB dump, or
in the output of a jq filter such as @base64d. Encode the value first, for
example with @base64.
Redis has no empty hash, list, set, sorted set, or stream, and a native Redis value has no null. So a Redis write refuses an empty one of these types and a null in a native value. A RedisJSON value keeps its JSON null. The error names the key, and no part of that batch is written. Redis stores a number or a bool as its text, and a set keeps one copy of each member.
Native form --native¶
A normalized value can be plainer than the native type that it came from. A DynamoDB
fractional number reads as an exact string, a binary value as base64 text, and a set as a list.
A copy into DynamoDB then writes an S or an L. A MongoDB int32 reads as a number
and writes back as an int64, and a date or a Decimal128 writes back as a string. An HBase cell
that is not UTF-8 reads as base64 text and writes back as the bytes of that text. The copy does not
restore the type.
--native reads each item in the typed JSON form of its driver instead. Today, DynamoDB,
MongoDB and HBase have one. DynamoDB JSON looks like {"price": {"N": "1.50"}}. MongoDB canonical
Extended JSON looks like {"year": {"$numberInt": "2020"}}. HBase raw bytes look like
{"cf": {"dGl0bGU=": "RHVuZQ=="}}. Each record carries its form. A
--typed dump holds it in an optional form field, for example
{"key":"a","type":"item","form":"dynamodb-json","value":{…}}. A dump of normalized records has
no form field, so old dumps read as before. A reader refuses a form that it does not know.
iq --native --typed --src books -o books.jsonl
iq --native --typed --src 'file:///export.json.gz?format=dynamodb-json&keys=pk' -o books.jsonl
iq --native --src mbooks --insert mbooks2
iq --native --typed --src mbooks -o mbooks.jsonl
iq --native --src hbooks --insert hbooks2
iq --native --typed --src hbooks -o hbooks.jsonl
The rules of --native:
- Only the destination that owns the form can write it. A DynamoDB destination reads a native
record strictly and refuses a malformed type tag, a number that it cannot store exactly, or an
empty set. A MongoDB destination reads a native record as canonical Extended JSON and refuses a
malformed tag. It keeps the type of
_idwhen the display form of_idequals the key. Any other destination refuses the record, and the error names the key and both forms. An HBase destination decodes a raw record strictly. It refuses a key, a qualifier or a cell value that is not canonical base64, a family that is not an object, and a cell that is not a string. The error names the key and the cell. A record in normalized JSON still goes to every destination. - A jq filter sees the typed form, so a filter that edits a value must keep the tags. A filter that breaks a tag makes the destination refuse the record.
- No predicate is pushed down.
iqscans the whole source and runs the filter on the client. A native read cannot fetch a named key, so the filter must start with.[].--explainshows this. --nativeworks with--json,--jsona,--jsonl, and--yaml. The other renderings flatten the tags, soiqrefuses them.--nativedoes not work with--key-field, because foreign JSON has no native form. It does not work withsource().iq diffandiq schemahave no--native.- A source without a native form refuses
--native. That includes a typed dump file. Read a typed dump without--native, because each of its records already names its form. --nativedoes not use the file decode cache. Each run reads the dump again.- A piped typed dump already names the form of each record.
--nativehas no effect on it. - With
--replace,iqclears the destination only after the first page passes the checks.
Key mapping --key*¶
On a plain copy, each item carries its source key, so no key flag is needed.
Foreign JSON (piped stdin, a reshaped stream) has no natural key. --key-field
takes it from an object field. --key computes it with a jq expression.
--key-prefix prepends a namespace either way. A key is a string, a number, or
a boolean. A float key must be a whole number and prints as plain digits
(1000000, not 1e+06). A fractional, NaN, or infinite float key is refused.
So is a float key whose value a float cannot hold exactly, such as 1e23. Use an
integer or a string key.
iq combine --insert is the one write that always requires --key or
--key-field. A combine's results come out of one program over a null input,
so no value has a key to inherit. As a result, the run is refused up front
rather than failing partway through a copy. For the same reason, combine has
no --typed. A typed dump is a stream of {key,type,value} records and needs
the same key.
The record key replaces the identity that the value carries. MongoDB,
Elasticsearch, OpenSearch, and CouchDB drop the _id of the value and use the
record key instead. CouchDB also drops the _rev of the value. DynamoDB writes
the record key over the key attributes of the item. Neo4j merges on the record
key and writes it as the key property. This is how --key,
--key-field, and --key-prefix take effect. For a record with no key,
MongoDB, Elasticsearch, OpenSearch, and CouchDB also drop the _id of the value,
and the backend mints an identity. DynamoDB writes such a record as it is,
including an _id attribute, and the item must hold the key attributes of the
table.
CouchDB refuses a value that holds a _deleted or _attachments member. A
read returns an attachment as a stub, so iq does not copy attachments.
A write flag used without --insert is an error, never silently ignored.
| short | long | default | description |
|---|---|---|---|
--key <string> |
none | jq expression yielding each written item's key (--insert/--typed) |
|
--key-field <string> |
none | object field to take each written item's key from (--insert) | |
--key-prefix <string> |
none | string prepended to every written key (--insert/--typed) |
iq combine 'users=.[] | {id, name}' 'orders=.[] | select(.total > 99)' \
--with '($users | INDEX(.id)) as $u | $orders[] | . + {name: $u[.userId].name}' \
--insert joined --key '.userId | tostring' --key-prefix 'j:'
Type mapping --type¶
A plain copy carries each item's native type along (a Redis hash lands as a
hash), so no type flag is needed. A filter that reshapes the value drops that
type. The output is plain JSON, and --type names the native type that the
destination stores it as.
Left unset, a typed destination infers it from the shape (Redis writes a scalar
as a string and an object or array as json). As a result, --type is only
required when you want something else:
- A
hashbuilt from an object - A
listorsetfrom an array - A
zsetfrom[{member, score}]pairs.
A document store ignores it. Every value is a document there. With --typed,
the same flag stamps the type field of each dump record instead.
| short | long | default | description |
|---|---|---|---|
--type <string> |
none | native type stamped on each written value, for example hash, list, json (--insert/--typed) |
iq '{title, year: (.year | tostring)}' --src books --insert cache --type hash
iq '[.tags[]]' --src books --insert cache --type list --key-prefix 'tags:'
iq '{title}' --src books --typed --type json -o titles.jsonl
Replace --replace¶
By default, a write upserts. Existing keys are overwritten, and every other key
in the destination stays. For a key that exists, the result depends on the
driver. MongoDB, Elasticsearch, OpenSearch, CouchDB, Couchbase, DynamoDB, Redis
and Neo4j replace the whole value of the key. Cassandra and HBase merge. They
write the columns or cells that the record holds, and the other columns or cells
of the row stay. --replace turns the copy into a restore. It empties
the destination first (the same operation as iq data clear, a Redis
FLUSHDB, a Mongo deleteMany({})) and then writes. As a result, the
destination ends up holding exactly the values of the source. In Neo4j, the
clear also deletes the relationships of the nodes, and the copy writes only the
nodes of the label with their properties. So relationships and other labels are
not restored.
Because it destroys data, it prompts (clear <destination> before writing).
--force answers yes. --dry-run reports the copy without clearing anything.
A source cannot be its own --replace destination. The copy reads the source
after the clear, so it would find nothing. iq refuses the run, also with
--dry-run, before it opens the destination, so nothing is cleared or written.
To rewrite a source in place, dump it first with --typed -o <file>, register
the file as a source, and insert from that file.
iq compares what the two URIs name, not how they are spelled. Two handles for
redis://h and redis://h:6379/0 name one container, so iq refuses them. The
comparison ignores user info and the case of the host. It also treats
localhost, 127.0.0.1, and ::1 as one host, and fills in the default port.
Two host lists match when they share one host. The check compares the URI text
and does not resolve names. It can miss two names for one server, such as two
DNS names, a CNAME, or mongodb+srv://h and mongodb://h. Treat it as a
safety net.
A destination that cannot be cleared (the read-only file dump) is refused up
front. --no-overwrite is the opposite choice, insert-only, so the two are
mutually exclusive. Neither applies to a --typed dump.
| short | long | default | description |
|---|---|---|---|
--replace |
✗ | empty the destination before writing, with confirmation (--insert) | |
--force |
✗ | skip the confirmation prompt for --replace |
|
--no-overwrite |
✗ | skip keys that already exist (--insert) |
iq --src snap --insert cache --replace --dry-run
Lifecycle previews¶
Every iq data subcommand (delete, clear, drop) shares two previews.
clear and drop also prompt before destroying data. --force skips the
prompt.
| short | long | default | description |
|---|---|---|---|
--explain |
✗ | print the access plan and exit without connecting or changing anything | |
--dry-run |
✗ | connect and report the real effect without changing anything |
Delete data delete¶
iq data delete <target> <key> removes a named set of keys, keeping the
container. It is the typed, capability-gated, explainable counterpart of a raw
per-key delete (HBase's exec delete verb, a Redis DEL).
Each key uses the same spelling as a Get: a bare string (book:1) or a JSON
array for a composite key (["shop",42]). A key already absent is not an error
(delete is idempotent). The report states it
(deleted N key(s), M already absent).
Unlike data clear/data drop, it does not prompt. The explicit key list you
typed is the confirmation (use --dry-run to preview). A backend with no
per-key identity (the read-only file dump) rejects it (like Redis rejects
data drop).
Clear data clear¶
Empties a container but keeps it (for example MongoDB deleteMany({}) or Redis
FLUSHDB). Both this and iq data drop are distinct from iq rm, which only
unregisters a saved source. These destroy stored data, and prompt for
confirmation unless --force is used.
Drop data drop¶
Removes the container entirely and prompts for confirmation unless --force is
used. Sources without a droppable container are rejected (for example Redis).
Dry run --dry-run¶
Reports the effect without writing. It does everything except the final mutation. It connects, opens source and destination, and runs the real scan. It applies the real transform and probes capabilities. Then it suppresses the write.
| short | long | default | description |
|---|---|---|---|
--dry-run |
✗ | report the effect of --insert without writing anything |