---
title: "Build object-storage lookups with byte ranges"
description: "Pair a compact index with HTTP range reads to fetch individual records from large immutable objects without a database."
canonical_url: "https://fanout.sh/system/archive/byte-range-indexed-object-storage"
md_url: "https://fanout.sh/system/archive/byte-range-indexed-object-storage.md"
access: "public"
---

# Build object-storage lookups with byte ranges

Pair a compact index with HTTP range reads to fetch individual records from large immutable objects without a database.

## Public overview

Byte-Range Indexed Object Storage

Byte-range reads let an application fetch only part of a large object. With a separate index, object storage can support point-lookups without behaving like a database.

This is the core trick behind the lecture's dictionary design.

Object storage is good at durable blob storage, high aggregate throughput, and simple object reads. It is not a query engine.

The index supplies the missing query plan:

S3 supports fetching part of an object using a byte range. HTTP also defines range requests for GET , with 206 Partial Content responses when a satisfiable range is served.

large dictionary/catalog lookup, reading media/video chunks, reading indexed binary file sections, parallel download of huge objects, serving static immutable data with a small hot index.

Latency And Cost Model

A point lookup is cheap only if the index lookup avoids extra network calls.

If the index lives in a remote database, the design becomes two network hops before any bytes return. If the index fits in process memory, the object store is the only remote read. That is why this pattern works best for static catalogs, model artifacts, media manifests, or dictionary-like datasets whose index can be loaded at startup.

Choice Effect --- --- one record per range simple reads, many small object requests block of records per range fewer requests, extra local scan/decode local hot-value cache lower p95 for popular keys, cache invalidation by version

Is the object immutable while this index is active? Is the index version tied to the object version? Does the range read include exactly one logical value or a block containing many values? Are offsets byte offsets after compression or before compression? What happens if object storage returns partial/failing reads? Can API servers cache hot values locally?

Compression changes range-read math.

If the whole file is compressed as one stream, arbitrary byte ranges may be useless because decompressing a value can require earlier bytes. Prefer:

per-record compression, block compression with a block index, or no compression for the data section if direct ranges matter more than space.

Do not expose raw object paths and byte ranges as an authorization bypass. The API should still enforce:

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md