Build object-storage lookups with byte ranges

Pair a compact index with HTTP range reads to fetch individual records from large immutable objects without a database.

Byte-range indexed object storage

A byte-range read retrieves part of an object. Add an index from logical keys to byte locations, and a large dictionary or catalog can answer a point lookup without transferring its entire data file.

The application supplies the lookup logic; object storage supplies bytes. We'll translate an index entry into a range, verify a local read, then handle response validation, compression and access control.

Name the object as well as the position

The wiki/file-backed-dictionary-storage-engine dictionary design needs an entry shaped like this:

Object identity must select the version used to build the index. An unchanged offset in a replaced object can return a different value with a perfectly valid length. Use immutable object names or an explicit retained version; wiki/immutable-versioned-data-files the next lesson covers publication and cleanup.

Our exercise uses a separate color pack: red\n , green\n , then purple\n . The newline is one byte, making the values four, six and seven bytes long. This is not the previous lesson's word-meaning file.

Key Half-open byte slice and length --- --- apple [0:4] : 4 bytes pear [4:10] : 6 bytes plum [10:17] : 7 bytes

A program slice excludes its right endpoint. HTTP byte ranges include it, so pear's six bytes become Range: bytes=4-9 . For a nonempty indexed value, the last byte is start + length - 1 . An empty value needs no body read; do not turn zero length into an invalid reversed range.

Exercise the boundary

The object-storage download writes this 17-byte pack in a private temporary directory. It compares a full read with a seek to pear's offset, then tries invalid metadata and an actually shortened source file.

The helper rejects a negative offset or an interval beyond the recorded complete length. It also checks the returned byte count. A valid-looking index cannot make a shortened source return all its promised bytes.

This mode performs local file reads. Its HTTP line is calculated syntax, not a network response or an S3 measurement. Returned bytes also do not tell us the number of physical disk reads.

Check what the server actually returned

A served single HTTP range uses 206 Partial Content and a Content-Range describing the interval. An unsatisfiable range can produce 416 . A server may also ignore Range and send the full representation, so sending the header does not guarantee a partial response.

For our six-byte lookup, require the expected interval and body length before decoding. Handle a full response explicitly within a bounded fallback policy, rather than passing its first bytes off as the requested value. A short body, wrong version or unexpected encoding is a failed read.