Choosing a format: size, speed, readability and tooling
A decision table that maps real requirements to a concrete format, plus the factors that actually change the answer — schema evolution, human editing and ecosystem support.
The five criteria that matter
- Readability — can a human review it in a pull request? Favours JSON, YAML, TOML, CSV.
- Size — how much bandwidth or storage? Favours Parquet, Protobuf, MessagePack.
- Parse cost — how much CPU per record? Text formats are the expensive ones.
- Schema evolution — can old readers survive new writers? Favours Avro, Protobuf, JSON with tolerant parsing.
- Ecosystem — is there a maintained library in every language you use, and a CLI for debugging?
Most teams over-weight size and speed, and under-weight ecosystem and debuggability. A format your team cannot inspect at 3am is a format that will cost more than the bandwidth it saves.
The decision table
| Situation | Choose | Because |
|---|---|---|
| Public REST API request/response | JSON | Universal clients, readable in logs |
| Hand-edited tool config | TOML | Typed, no indentation or type surprises |
| Large structured deployment config | YAML | Mature tooling in the Kubernetes ecosystem |
| Event stream between services | Protobuf or Avro | Small payloads plus a registry for schemas |
| Analytical scan of a data lake | Parquet | Column pruning and compression |
| In-process exchange between languages | Arrow | Zero-copy shared memory layout |
| Human-facing spreadsheet export | CSV | Opens everywhere, even when it breaks |
| Append-only log or cache value | NDJSON or MessagePack | Streamable, one record per line |
| Config generated by a machine | JSON | One canonical form, easy to emit |
# a cheap way to compare real payloads on real data
wc -c payload.json
jq -c . payload.json | gzip -9 | wc -c
# then compare against the same data in your candidate binary formatSwitching later
Format choice is rarely permanent, but it is expensive to reverse if you let it leak into every layer. Contain it: convert at the boundary, keep one internal representation, and never let a wire format define your domain model.
- Introduce a codec module that converts between the wire format and your internal types.
- Serve both formats behind content negotiation during the transition.
- Publish the new format to a small consumer first, and measure.
- Remove the old format only when traffic reaches zero for a full retention window.
💡
The most common mistake is not choosing the wrong format — it is choosing three. Pick one per boundary (client API, internal events, storage) and write the decision down, or every new service will invent its own.
FAQ
Is JSON always slow?
Parsing is the cost, not the format. For most APIs JSON parsing is a small fraction of request time. It becomes the bottleneck in high-throughput pipelines, where binary or columnar layouts win.
What about XML?
Still the right answer for documents with mixed content, for XSD-validated regulated interchange, and for legacy SOAP integrations. It is verbose for simple records, which is what JSON replaced.
Related
Tabular formats: TSV, Parquet, Arrow and Avro Data interchange in APIs: negotiation and versioning
Last refreshed 2026-09-18.