Content addressing: Git, IPFS and deduplication

How naming data by its hash enables deduplication and verification, how Git builds a tree of objects, what a Merkle tree proves, and where content addressing breaks down.

Name the content, not the location

In content-addressed storage the key is a hash of the value. Two identical files have the same key automatically, verification is a recomputation, and immutability is a consequence rather than a policy.

import hashlib

def digest(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()

a = b"shared header" + b"body A"
b = b"shared header" + b"body B"
print(digest(a))
print(digest(b))

# identical content, identical key, stored once
assert digest(b"same bytes") == digest(b"same bytes")

Git is a hash-addressed object store

# the object id is the hash of "blob <len>\0" plus the content
printf 'hello\n' | git hash-object --stdin
# ce013625030ba8dba906f756967f9e9ca394464a

git cat-file -t ce013625030ba8dba906f756967f9e9ca394464a    # blob
git cat-file -p ce013625030ba8dba906f756967f9e9ca394464a    # hello

# a tree references blobs by hash; a commit references a tree and parents
git cat-file -p HEAD | head -4

• A commit is a hash over the tree, the parents, the author, the committer and the message — so editing history changes every subsequent id.
• Deduplication is automatic: the same file content in two branches is one blob.
• Git's original SHA-1 choice is why SHA-1 collision research caused a real integrity concern and drove the migration to SHA-256 repositories.
• A hash of an empty directory is not the empty hash; Git stores an empty tree as a distinct object.

Merkle trees and what they buy

import hashlib

def h(b: bytes) -> bytes:
    return hashlib.sha256(b).digest()

def merkle_root(chunks):
    level = [h(c) for c in chunks]
    while len(level) > 1:
        if len(level) % 2:
            level.append(level[-1])          # duplicate the odd node
        level = [h(level[i] + level[i + 1]) for i in range(0, len(level), 2)]
    return level[0].hex()

root = merkle_root([b"a", b"b", b"c", b"d"])
print(root)

# proof: a verifier needs only the sibling path, not the whole data
def proof(chunks, index):
    level = [h(c) for c in chunks]
    path = []
    while len(level) > 1:
        if len(level) % 2:
            level.append(level[-1])
        sibling = index ^ 1
        path.append(level[sibling])
        level = [h(level[i] + level[i + 1]) for i in range(0, len(level), 2)]
        index //= 2
    return path

• One root hash commits to an entire dataset; changing any leaf changes the root.
• A proof for one leaf is logarithmic in the number of leaves, which is what makes light clients possible.
• Deduplication in backups uses content-defined chunking so that an insertion early in a file does not shift every subsequent chunk boundary.
• Deleting data from a content-addressed store is hard when an object is referenced by many keys.

FAQ

Related

Collisions, birthday attacks and length extension Cache keys, ETags and shard keys

Last refreshed 2026-09-18.