Converting between formats with jq, yq and csvkit

Reshape JSON with jq, convert YAML and TOML with yq, and turn CSV into queryable tables with csvkit and miller — the pipelines that replace a throwaway script.

jq for reshaping

jq is a filter language: you describe the output shape and it streams the input. Most conversion tasks are three operations — select, map, construct.

# pick and rename fields, drop nulls
jq '[.[] | select(.active) | {id, name: .full_name}]' users.json

# flatten a nested array into rows
jq -r '.orders[] | [.id, .customer.name, .total] | @csv' orders.json > orders.csv

# group and aggregate instead of converting at all
jq 'group_by(.country) | map({country: .[0].country, n: length})' users.json

# stream a large file instead of building an array in memory
jq -c --stream 'select(length == 2)' big.json

• -r prints raw strings without JSON quotes — essential when feeding @csv or a shell.
• -c emits one compact object per line, which is NDJSON.
• --slurp reads the whole input as a single array; avoid it on large files.
• @csv and @tsv do the quoting rules for you.

yq for YAML, TOML and XML

# YAML to JSON
yq -o=json '.' config.yaml

# JSON to YAML, safely quoted
yq -P '.' config.json > config.yaml

# TOML to JSON
yq -p=toml -o=json '.' pyproject.toml

# edit in place rather than converting
yq -i '.spec.replicas = 3' deployment.yaml

The mikefarah yq (Go) and the Python yq wrapper have different flags. Check yq --version before copying a command from a blog post — this is a common source of mysterious failures in CI.

csvkit and miller for tabular work

# inspect instead of guessing
csvstat sales.csv
head -3 sales.csv | csvlook

# filter and select by column name, not index
csvgrep -c country -m DE sales.csv | csvcut -c date,amount > de.csv

# run SQL directly against CSV files
csvsql --query "select country, sum(amount) from sales group by country" sales.csv

# miller: one tool for CSV, TSV, JSON and more
mlr --icsv --ojson head -n 5 sales.csv
mlr --csv sort -nr amount sales.csv

csvsql and miller are the fastest route from "a colleague sent a spreadsheet" to an answer, and both keep the transformation as a one-line, reviewable command instead of a notebook cell.

FAQ

Related

CSV in depth: quoting, delimiters and Excel traps Dates, numbers and encoding pitfalls across formats

Last refreshed 2026-09-18.