Simple binary vector format reader

Reads the vecs family of binary vector formats (e.g. .bvecs, .ivecs, .fvecs), defined at INRIA LEAR around 2008-2009 and used by the TEXMEX reference datasets.

Supports int8, int16, int32, float32, int64, and float64 elements.

All vectors in the file should be of the same type, but do not have to be of the same dimensions.

Each vector should be of the form:

[length][element:0]...[element:length-1]

Where length is an int, and element is one of the supported vector component types.

The file should then be a contiguous array of the above vectors.

[vector][vector][vector]...

For example:

Path path = Path.of("/path/to/vectors.fvecs");

try (CloseableIterator<float[]> vectors = VectorFileIterators.floats(path)) {
    while (vectors.hasNext()) {
        float[] vector = vectors.next();
        // process vector
    }
} catch (IOException exception) {
    // handle exception
}

Composable API

VectorFileIterators is a thin convenience over a composable API with sensible defaults. For control over where the bytes come from, how they are buffered, and their byte order, compose a reader from a source, a buffering strategy, and the element type:

Path path = Path.of("/path/to/vectors.bvecs");

try (CloseableIterator<short[]> vectors = VectorReader.from(Source.file(path)) // read from a file
        .fixed(Buffers.heap(1 << 20)) // 1 MiB read-ahead buffer
        .as(ElementType.INT8) // of 8-bit vectors
        .map(VectorConverters::unsignedBytesToShorts) // interpreted as unsigned
        .filter(vector -> vector.length == 128) // keeping only 128d vectors
        .limit(10_000_000)) { // but only the first 10M

    while (vectors.hasNext()) {
        short[] vector = vectors.next();
        // process vector
    }

} catch (IOException exception) {
    // handle exception
}

Source: where the bytes come from: Source.file(path), Source.directFile(path) (direct IO), Source.stream(inputStream), Source.channel(channel), or Source.bytes(array).
Buffering: fixed(Buffers.heap(capacity)) reads ahead into one shared buffer, while growable(order) reads each record exactly into a buffer that grows as records require. Buffers.heap(capacity, order) sets the byte order; Buffers.direct(capacity, order) allocates a block-aligned direct buffer for use with Source.directFile.
Type: as(type) names the element type the file was written with, one of ElementType.INT8, INT16, INT32, FLOAT32, INT64, or FLOAT64. No format records its own element type, so naming the wrong one decodes into meaningless values. The named bytes(), shorts(), ints(), longs(), floats(), and doubles() are deprecated since 1.1.0.
Operations: the returned CloseableIterator is lazy - chain filter, map, peek, limit, skip, takeWhile, or dropWhile. Each returns a CloseableIterator, so a try-with-resources on the final link still closes the source.
VectorConverters supplies element mappers for map, such as unsigned widenings (unsignedBytesToShorts) and the ...ToFloats narrowings.