Tabular formats: TSV, Parquet, Arrow and Avro

Row versus columnar storage, how Parquet gets its speed from encoding and predicate pushdown, what Arrow changes in memory, and when Avro is the right record format.

Two ways to lay out the same table

A row store keeps all fields of a record together. A column store keeps all values of one field together. The layout decides which queries are cheap.

row store (CSV, Avro)
  [id=1, name=Ada,   city=London] [id=2, name=Alan, city=Manchester]

column store (Parquet, Arrow)
  id:   [1, 2]
  name: [Ada, Alan]
  city: [London, Manchester]

Format	Layout	Best at	Weak at
TSV	Row, text	Paste into shell tools; simple streaming	Any quoting need; no types
Avro	Row, binary	Write-heavy event streams with schema evolution	Analytical scans of a few columns
Parquet	Column, binary	Analytics; reads only the columns needed	Single-record appends and updates
Arrow	Column, in memory	Zero-copy exchange between processes and languages	Long-term storage and file interchange

Why Parquet is fast

Column pruning — a query touching two of fifty columns reads two column chunks.
Dictionary and run-length encoding turn repeated values into tiny integer codes.
Row groups with statistics let a reader skip a whole block when the min/max range excludes the filter.
Predicate pushdown applies the filter during the scan instead of after it.

import pyarrow.parquet as pq
import pyarrow.compute as pc

table = pq.read_table("events.parquet", columns=["user_id", "amount"])
filtered = table.filter(pc.greater(table["amount"], 100))

pq.write_table(table, "out.parquet", compression="zstd", row_group_size=128_000)

Row group size is the main tuning knob. Larger groups compress better; smaller groups let a scan skip more, and increase metadata overhead.

Arrow in memory, Avro on the wire

import pyarrow as pa

schema = pa.schema([
    ("ts", pa.timestamp("us", tz="UTC")),
    ("user_id", pa.int64()),
    ("amount", pa.decimal128(12, 2)),
])
batch = pa.record_batch([[1700000000000000], [42], [19_99]], schema=schema)

# Arrow IPC: hand this buffer to another process without re-encoding
sink = pa.BufferOutputStream()
with pa.ipc.new_stream(sink, schema) as w:
    w.write_batch(batch)

Arrow is a memory format with a defined layout, so a Python producer and a Rust consumer share the same bytes. Parquet is an on-disk format that Arrow buffers can be written into and read out of.

⚠️

Parquet files are immutable. An overwrite writes a whole new file, so naive streaming creates thousands of tiny files and destroys scan performance. Compact small files on a schedule, and never treat Parquet as a queue.

FAQ

Can I append to a Parquet file cheaply?

Not really. You can add a row group to a new file, but the format is designed for immutable, bulk-written datasets. For per-event writes land NDJSON or Avro first, then compact into Parquet.

Do these formats preserve types?

Parquet and Arrow carry a typed schema, so timestamps, decimals and nulls survive round trips. TSV is text only, so types must be re-declared by the reader.

Line-delimited data: NDJSON and log formats Binary formats: Protocol Buffers, MessagePack, CBOR and BSON

Last refreshed 2026-09-18.

Tabular formats: TSV, Parquet, Arrow and Avro

Two ways to lay out the same table

Why Parquet is fast

Arrow in memory, Avro on the wire

FAQ

Related