Data Formats

JSON and CSV: shape, rules and conversion.

Data Formats is taught here in 14 lessons, running from JSON basics through to Validating and testing data pipelines. Each lesson takes one topic, shows the working code, and links onward to the next, so the course can be read straight through in order.

Lessons

1. JSON basicsThe shape of JSON, how it maps to language types, and the easy mistakes that produce invalid JSON.
2. CSV vs JSONWhen a flat table beats nested documents, and how to convert between them safely.
3. CSV in depth: quoting, delimiters and Excel trapsRFC 4180 in practice — quoted fields, doubled quotes, embedded newlines — plus the BOM, delimiter and locale quirks that break real spreadsheet exports.
4. YAML for configurationBlock and flow styles, the type-inference surprises that turn version numbers into floats, anchors and aliases, multi-document files, and the indent rules that cost hours.
5. TOML and INI filesSections, dotted keys, typed values and arrays without ambiguity, plus why TOML replaced INI for tooling config and how to migrate a file between them.
6. Line-delimited data: NDJSON and log formatsOne JSON object per line for streaming pipelines, plus the logfmt, Apache and syslog shapes you will actually parse, and how to survive a half-written line.
7. Schema definition and validationJSON Schema keywords from required to oneOf, how to avoid the composition traps, Avro schema basics, and wiring validation into CI so bad data never reaches production.
8. Tabular formats: TSV, Parquet, Arrow and AvroRow versus columnar storage, how Parquet gets its speed from encoding and predicate pushdown, what Arrow changes in memory, and when Avro is the right record format.
9. Binary formats: Protocol Buffers, MessagePack, CBOR and BSONField tags instead of names, how schema evolution stays compatible, where MessagePack and CBOR fit, what BSON adds for MongoDB, and real numbers on size and speed.
10. Converting between formats with jq, yq and csvkitReshape JSON with jq, convert YAML and TOML with yq, and turn CSV into queryable tables with csvkit and miller — the pipelines that replace a throwaway script.
11. Dates, numbers and encoding pitfalls across formatsWhy dates should be strings or epochs but never both, how big integers and floats lose precision in JSON, and the null, bytes and Unicode differences between formats.
12. Choosing a format: size, speed, readability and toolingA decision table that maps real requirements to a concrete format, plus the factors that actually change the answer — schema evolution, human editing and ecosystem support.
13. Data interchange in APIs: negotiation and versioningContent-Type and Accept negotiation, envelope versus bare payloads, how to version a format without breaking clients, and an error contract worth keeping.
14. Validating and testing data pipelinesFixture files, round-trip and property tests, schema drift detection, a malformed-input suite, and the metrics that catch bad data before a customer does.

More in Computer Science Fundamentals

Algorithms Data Structures Computer Networks Operating Systems Character Encodings Hashing & Checksums Dates & Time Regular Expressions

FAQ