JSON Lines, streaming and large documents

Use NDJSON for event and log data, parse it line by line, and process files too large to load without running out of memory.

One object per line

{"ts":"2026-09-18T11:00:00Z","level":"info","msg":"started","pid":412}
{"ts":"2026-09-18T11:00:01Z","level":"warn","msg":"retry","attempt":1}
{"ts":"2026-09-18T11:00:04Z","level":"info","msg":"ready","port":3000}

// writing: one object, one newline, no pretty printing
for (const event of events) {
  stream.write(JSON.stringify(event) + "\n");
}

// reading: split on newlines and parse each line
import { createReadStream } from "node:fs";
import { createInterface } from "node:readline";

const rl = createInterface({ input: createReadStream("events.ndjson") });

rl.on("line", (line) => {
  if (!line.trim()) return;               // tolerate a trailing blank line
  const event = JSON.parse(line);
  handle(event);
});

• A streaming file has no enclosing array and no commas between records, which is what makes it appendable and resumable.
• Every record must be on exactly one line: a raw newline inside a string has to be escaped, which JSON.stringify does for you.
• The content type is application/x-ndjson; some APIs also accept application/jsonl.
• A blank line at the end is common - skip it rather than failing the parse.

Files that do not fit in memory

// this loads the whole file, then doubles it while parsing
const all = JSON.parse(await readFile("big.json", "utf8"));   // 2 GB file, 4 GB+ of RAM

// line-oriented processing stays flat
const rl = createInterface({ input: createReadStream("big.ndjson") });
let count = 0;
for await (const line of rl) {
  if (!line.trim()) continue;
  if (JSON.parse(line).level === "error") count++;
}
console.log(count);

# inspect a large NDJSON file without loading it all
head -c 500 big.ndjson
wc -l big.ndjson

# filter and count with jq, streaming
jq -c 'select(.level == "error")' big.ndjson | wc -l

# sample the first 20 records
head -20 big.ndjson | jq -s .

import ijson

# iterate the elements of a top-level array without loading it
with open("big.json", "rb") as f:
    for order in ijson.items(f, "orders.item"):
        process(order)

Streaming HTTP and pagination

// consume a streaming response as it arrives
const res = await fetch("https://api.example.com/events/stream", {
  headers: { Accept: "application/x-ndjson" },
});

const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split("\n");
  buffer = lines.pop();              // keep the incomplete last line

  for (const line of lines) {
    if (line) handle(JSON.parse(line));
  }
}

if (buffer.trim()) handle(JSON.parse(buffer));

1. A chunk boundary can fall in the middle of a line, so a buffer that keeps the incomplete tail is required, not optional.
2. For a large result set, paginate rather than streaming one enormous response: it bounds memory on both sides and gives a natural resume point.
3. A cursor is better than an offset for anything that changes while you read it, because inserts do not shift the pages.
4. Record the last processed id when you finish a batch, so a restarted job resumes rather than starting again.
5. Compress the stream. NDJSON compresses very well, since every record repeats the same keys.

# follow a growing file, and keep the byte offset for a resume
tail -f events.ndjson | jq -c 'select(.level == "error")'

# count records without decompressing to disk
zstd -dc archive.ndjson.zst | wc -l

FAQ

Related

JSON in HTTP APIs JSON in databases and storage

Last refreshed 2026-09-18.