Line-delimited data: NDJSON and log formats

One JSON object per line for streaming pipelines, plus the logfmt, Apache and syslog shapes you will actually parse, and how to survive a half-written line.

NDJSON: one object per line

Newline-delimited JSON removes the biggest operational problem of a JSON array: you cannot stream it. With NDJSON you can process, filter and append records without ever holding the whole file in memory.

{"ts":"2026-09-18T10:00:00Z","level":"info","msg":"started","port":8080}
{"ts":"2026-09-18T10:00:01Z","level":"warn","msg":"slow query","ms":812}
{"ts":"2026-09-18T10:00:02Z","level":"info","msg":"ready"}

• No outer array and no commas between records — appending a record is a single write.
• Each line must be a complete, valid JSON value; a newline inside a string is forbidden.
• Files compress extremely well and parallelise trivially by splitting on line boundaries.

# count warnings without loading the file
grep '"level":"warn"' app.ndjson | wc -l

# jq streams line by line by default
jq -c 'select(.ms > 500) | {ts, ms}' app.ndjson

The log shapes you will meet

Prefer emitting JSON where you control the writer. Structured logs avoid the regular expressions that break the moment a message contains a quote.

Reading a stream that is still being written

import json

def read_ndjson(path, start_offset=0):
    """Yield (offset, record). Skip a trailing partial line."""
    with open(path, "rb") as f:
        f.seek(start_offset)
        buf = b""
        while chunk := f.read(65536):
            buf += chunk
            lines = buf.split(b"\n")
            buf = lines.pop()          # last element is incomplete or empty
            for line in lines:
                if line.strip():
                    yield json.loads(line)
        if buf.strip():                # file ended without a newline
            try:
                yield json.loads(buf)
            except json.JSONDecodeError:
                pass                   # a genuinely truncated write

Two details make a stream reader reliable: always keep the un-terminated tail in the buffer, and remember the byte offset so a restart resumes without reprocessing.

FAQ

Related

Tabular formats: TSV, Parquet, Arrow and Avro Validating and testing data pipelines

Last refreshed 2026-09-18.