Data Formats cheat sheet

A scannable Data Formats reference: 28 short snippets across 14 topics, each linking back to the lesson it came from.

At a glance

Quick snippets

JSON basics

Rules that break parsers

{
  "name": "Ada",
  "age": 36,
  "skills": ["math", "logic"],
  "active": true,
  "score": null
}

Reading and writing it

// Parse (throws on invalid JSON)
const obj = JSON.parse(text);

// Serialize (replacer + indent for readability)
const text = JSON.stringify(obj, null, 2);

Reading and writing it

import json
obj = json.loads(text)        # parse
text = json.dumps(obj, indent=2)  # pretty-print

Full lesson: JSON basics →

CSV vs JSON

Converting safely

import csv, json

with open('data.csv', newline='', encoding='utf-8') as f:
    rows = list(csv.DictReader(f))

with open('data.json', 'w', encoding='utf-8') as f:
    json.dump(rows, f, indent=2, ensure_ascii=False)

Full lesson: CSV vs JSON →

CSV in depth: quoting, delimiters and Excel traps

What the format actually allows

name,note,price
Widget,"ships in 2-3 days",9.99
Gadget,"he said ""hello""",19.99
Cable,"multi-line
description",4.50

Parsing and writing safely

import csv

with open("data.csv", newline="", encoding="utf-8-sig") as f:
    reader = csv.reader(f, dialect="excel")
    rows = [r for r in reader]

# writing: the writer quotes only when it must
with open("out.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.writer(f, quoting=csv.QUOTE_MINIMAL, lineterminator="\r\n")
    w.writerow(["id", "note"])
    w.writerow(["1", 'he said "hi", loudly'])

Full lesson: CSV in depth: quoting, delimiters and Excel traps →

YAML for configuration

Structure without brackets

# block style
service:
  name: api
  ports:
    - 8080
    - 8443
  env:
    LOG_LEVEL: info

# the same document in flow style
service: {name: api, ports: [8080, 8443], env: {LOG_LEVEL: info}}

The type inference trap

version: "1.10"     # string, not float
country: "NO"       # string, not boolean
zip: "01234"        # string, not octal
enabled: true       # the only boolean you should write

Reuse and multiple documents

defaults: &defaults
  retries: 3
  timeout: 30

prod:
  <<: *defaults        # merge key
  timeout: 60          # overrides the anchor

---
# a second document in the same file
kind: ConfigMap

Full lesson: YAML for configuration →

TOML and INI files

INI is a family, not a standard

; classic ini
[server]
host = 0.0.0.0
port = 8080

[database]
host = localhost
port = 5432

Reading them in code

import tomllib          # read-only, Python 3.11+
import configparser

with open("app.toml", "rb") as f:
    cfg = tomllib.load(f)          # must be opened in binary mode
print(cfg["server"]["port"])       # already an int

parser = configparser.ConfigParser()
parser.read("legacy.ini")
port = parser.getint("server", "port")   # you must coerce by hand

Full lesson: TOML and INI files →

Line-delimited data: NDJSON and log formats

NDJSON: one object per line

{"ts":"2026-09-18T10:00:00Z","level":"info","msg":"started","port":8080}
{"ts":"2026-09-18T10:00:01Z","level":"warn","msg":"slow query","ms":812}
{"ts":"2026-09-18T10:00:02Z","level":"info","msg":"ready"}

NDJSON: one object per line

# count warnings without loading the file
grep '"level":"warn"' app.ndjson | wc -l

# jq streams line by line by default
jq -c 'select(.ms > 500) | {ts, ms}' app.ndjson

Full lesson: Line-delimited data: NDJSON and log formats →

Schema definition and validation

Composition and its traps

{
  "oneOf": [
    { "properties": { "kind": { "const": "card" }, "last4": { "type": "string", "pattern": "^[0-9]{4}$" } }, "required": ["kind", "last4"] },
    { "properties": { "kind": { "const": "bank" }, "iban":  { "type": "string" } }, "required": ["kind", "iban"] }
  ]
}

Validating in CI

import json
from jsonschema import Draft202012Validator

schema = json.load(open("schema/user.json"))
validator = Draft202012Validator(schema)

errors = sorted(validator.iter_errors(payload), key=lambda e: e.path)
for e in errors:
    print("/".join(map(str, e.path)) or "<root>", "-", e.message)
assert not errors, f"{len(errors)} validation error(s)"

Full lesson: Schema definition and validation →

Tabular formats: TSV, Parquet, Arrow and Avro

Two ways to lay out the same table

row store (CSV, Avro)
  [id=1, name=Ada,   city=London] [id=2, name=Alan, city=Manchester]

column store (Parquet, Arrow)
  id:   [1, 2]
  name: [Ada, Alan]
  city: [London, Manchester]

Why Parquet is fast

import pyarrow.parquet as pq
import pyarrow.compute as pc

table = pq.read_table("events.parquet", columns=["user_id", "amount"])
filtered = table.filter(pc.greater(table["amount"], 100))

pq.write_table(table, "out.parquet", compression="zstd", row_group_size=128_000)

Full lesson: Tabular formats: TSV, Parquet, Arrow and Avro →

Binary formats: Protocol Buffers, MessagePack, CBOR and BSON

Protocol Buffers: the tag is the identity

# generate and inspect
protoc --python_out=. order.proto
echo '{"id":1,"currency":"EUR","items":[{"sku":"A1","qty":2}]}' \
  | protoc --encode=shop.Order order.proto > order.bin
protoc --decode=shop.Order order.proto < order.bin

The schema-less alternatives

import msgpack, cbor2, bson

payload = {"id": 1, "tags": ["a", "b"], "ok": True}
packed = msgpack.packb(payload, use_bin_type=True)
assert msgpack.unpackb(packed, raw=False) == payload

# BSON embeds type and length prefixes per field: larger, but queryable
doc = bson.BSON.encode(payload)

Full lesson: Binary formats: Protocol Buffers, MessagePack, CBOR and BSON →

Converting between formats with jq, yq and csvkit

jq for reshaping

# pick and rename fields, drop nulls
jq '[.[] | select(.active) | {id, name: .full_name}]' users.json

# flatten a nested array into rows
jq -r '.orders[] | [.id, .customer.name, .total] | @csv' orders.json > orders.csv

# group and aggregate instead of converting at all
jq 'group_by(.country) | map({country: .[0].country, n: length})' users.json

# stream a large file instead of building an array in memory
jq -c --stream 'select(length == 2)' big.json

yq for YAML, TOML and XML

# YAML to JSON
yq -o=json '.' config.yaml

# JSON to YAML, safely quoted
yq -P '.' config.json > config.yaml

# TOML to JSON
yq -p=toml -o=json '.' pyproject.toml

# edit in place rather than converting
yq -i '.spec.replicas = 3' deployment.yaml

Full lesson: Converting between formats with jq, yq and csvkit →

Dates, numbers and encoding pitfalls across formats

Dates: pick one representation

{
  "createdAt": "2026-09-18T10:00:00Z",
  "createdAtEpoch": 1789123200,
  "birthday": "1990-04-01"
}

Numbers that do not survive the trip

JSON.parse('{"id": 9223372036854775807}')
// -> { id: 9223372036854775808 }   wrong, precision lost

JSON.stringify({ x: 0.1 + 0.2 })
// -> '{"x":0.3}', but the value is 0.30000000000000004

// money as a float accumulates error
0.1 + 0.2 === 0.3            // false

Full lesson: Dates, numbers and encoding pitfalls across formats →

Choosing a format: size, speed, readability and tooling

The decision table

# a cheap way to compare real payloads on real data
wc -c payload.json
jq -c . payload.json | gzip -9 | wc -c
# then compare against the same data in your candidate binary format

Full lesson: Choosing a format: size, speed, readability and tooling →

Data interchange in APIs: negotiation and versioning

Negotiating the representation

GET /v1/orders/42 HTTP/1.1
Accept: application/json;q=1.0, application/vnd.acme.order+json;q=0.9, */*;q=0.1

HTTP/1.1 200 OK
Content-Type: application/vnd.acme.order+json; charset=utf-8
Vary: Accept
Cache-Control: private, max-age=60

{"id":42,"total":{"amount":"19.99","currency":"EUR"}}

Versioning and compatibility

{
  "id": 42,
  "total": "19.99",
  "totalAmount": "19.99",
  "currency": "EUR"
}

Envelope or bare payload, and the error contract

{
  "data": { "id": 42, "total": "19.99" },
  "meta": { "requestId": "b7f1", "durationMs": 12 },
  "errors": []
}

Full lesson: Data interchange in APIs: negotiation and versioning →

Validating and testing data pipelines

Fixtures and round trips

tests/fixtures/
  users.valid.json
  users.missing-required.json
  users.nan.json
  legacy.bom.csv
  legacy.crlf.csv
  legacy.tab-delimited.csv
  legacy.single-column.csv
  events.partial-line.ndjson

Full lesson: Validating and testing data pipelines →

FAQ

Related cheat sheets

Algorithms Data Structures Computer Networks Operating Systems Character Encodings Hashing & Checksums

Last refreshed 2026-09-27.