vector-reader release notes

New public API

ElementType names the element type a file was written with, and Decoding#as(ElementType) decodes vectors as that type. Nothing in the vecs family records its own element type, so the caller still names it, but the vocabulary is now the stored width and interpretation rather than the Java primitive it lands in.

ElementType.INT8

signed 8-bit two's complement, decoded into byte[]

ElementType.INT16

signed 16-bit two's complement, decoded into short[]

ElementType.INT32

signed 32-bit two's complement, decoded into int[]

ElementType.FLOAT32

IEEE 754 binary32, decoded into float[]

ElementType.INT64

signed 64-bit two's complement, decoded into long[]

ElementType.FLOAT64

IEEE 754 binary64, decoded into double[]

try (CloseableIterator<float[]> vectors = VectorReader.from(Source.file(path))
        .fixed(Buffers.heap(1 << 16))
        .as(ElementType.FLOAT32)) {
    while (vectors.hasNext()) {
        float[] vector = vectors.next();
        // process vector
    }
}

Naming the element type as a value rather than as a method keeps one terminal per family as further formats are added, and lets the vocabulary grow to stored widths that have no matching Java primitive.

Deprecated for removal

The named element terminals on VectorReader.Decoding are deprecated since 1.1.0, for removal in 2.0.0. Each has an exact replacement returning the same type:

• bytes() becomes as(ElementType.INT8)
• shorts() becomes as(ElementType.INT16)
• ints() becomes as(ElementType.INT32)
• floats() becomes as(ElementType.FLOAT32)
• longs() becomes as(ElementType.INT64)
• doubles() becomes as(ElementType.FLOAT64)

VectorFileIterators is unaffected, its own method names are unchanged.

javac enables removal warnings by default, unlike plain deprecation warnings, so a build compiling with -Werror fails on these calls until they are migrated or suppressed with @SuppressWarnings("removal").

Removed

Types and members deprecated for removal in 0.7.0 and earlier are gone:

• VectorConvertors, the misspelled duplicate of VectorConverters
• Options
• CloseableIterable, Resource.Iterator, and Resource.Iterable
• the named per-type file-iterator classes (FloatVectorFileIterator and friends), replaced by the VectorFileIterators factory methods
• the VectorConverters iterator adapters
• ThrowingSupplier and LazyThrowingSingleton

Helpers that were public but never part of the reader API are withdrawn:

• Mutable, MutableInt, and MutableReference moved to test scope
• PrefetchingIterator moved to an encapsulated internal package
• ThrowingRunnable is now package-private

Behavior

• the jar is a JPMS module, module uk.co.parnmatt.vector, exporting the reader, convert, and util packages and encapsulating its internals. It requires transitive org.jspecify, so consumers see the nullness annotations
• the public API is @NullMarked (JSpecify), so every type, parameter, and return has a specified nullness. It is declared on each package, so both module-path and classpath consumers observe it
• the iterator types accept null elements, taking <E extends @Nullable Object>:

Iterator<@Nullable String> values = Iterator.adapt(source);

• Buffers is now a functional interface, an opaque allocation strategy. The heap and direct factories are unchanged, but its former concrete surface is removed: the BLOCK_ALIGNMENT constant, the Placement enum, the public constructor, and the capacity(), order(), placement(), and alignment() accessors. A strategy can be supplied directly, such as a lambda:

Buffers buffers = () -> ByteBuffer.allocate(4096).order(ByteOrder.LITTLE_ENDIAN);

• the resource-returning API carries @MustBeClosed and the pure, builder, and fluent API carries @CheckReturnValue, enforced at compile time for consumers that run Error Prone. The annotations are compile-time only, adding no runtime dependency
• the on-disk record format and decode semantics are unchanged

Deprecated for removal

Internal helpers, still functional:

• Mutable, MutableInt, and MutableReference, moving to test scope
• ThrowingRunnable, becoming package-private
• PrefetchingIterator, moving to an internal module

Behavior

• the jar declares Automatic-Module-Name: uk.co.parnmatt.vector, giving module-path consumers a stable module name
• the on-disk record format and decode semantics are unchanged

New public API

Iterator<E>, ResourceIterator<E, X>, and CloseableIterator<E>

gain lazy filter, peek, limit, skip, takeWhile, and dropWhile, alongside the existing map. Each returns an iterator of the same kind, evaluated as elements are drawn. They close through their source, so a derived iterator still releases the original.

try (CloseableIterator<float[]> vectors = VectorReader.from(Source.file(path))
        .fixed(Buffers.heap(1 << 16))
        .ints()
        .map(VectorConverters::intsToFloats)
        .filter(vector -> vector.length == 128)
        .limit(1_000)) {
    while (vectors.hasNext()) {
        float[] vector = vectors.next();
    }
}

Deprecated for removal

Still functional:

• MutableInt#increment(), which in 1.0.0 will return the value before incrementing. Do not rely on the increment()void binary symbol.

Behavior

• the operations are lazy, applied only as elements are drawn from the iterator
• the on-disk record format and decode semantics are unchanged

New public API

Iterator<E> and ResourceIterator<E, X>

library iterators over java.util.Iterator, adding a lazy map that applies a function to each element. CloseableIterator gains the same map. These close through their source, so a mapped iterator still releases the original. Iterator#adapt(java.util.Iterator) lifts an existing iterator into the fluent API.

try (CloseableIterator<float[]> vectors = VectorReader.from(Source.file(path))
        .fixed(Buffers.heap(1 << 16))
        .bytes()
        .map(VectorConverters::bytesToFloats)) {
    while (vectors.hasNext()) {
        float[] vector = vectors.next();
    }
}

VectorConverters

element-wise converters (unsignedBytesToShorts, bytesToFloats, shortsToFloats, intsToFloats, longsToFloats, doublesToFloats) that convert a single vector, for use as map functions. The former iterator-to-iterator methods are deprecated.

Deprecated for removal

Still functional, delegating to the new API:

• the Buffers canonical constructor, its capacity, order, placement, and alignment accessors, the Placement enum, and BLOCK_ALIGNMENT, replaced by building with Buffers::heap or Buffers::direct and reading the capacity or order from the allocated buffer
• the iterator-to-iterator VectorConverters methods, replaced by the element-wise converters applied through Iterator::map
• Resource.Iterator, replaced by ResourceIterator
• Resource.Iterable, CloseableIterable, ThrowingSupplier, and LazyThrowingSingleton, no longer part of the public API

Behavior

• Buffers::direct now guarantees the requested capacity for buffers smaller than the block alignment, where an aligned slice could previously collapse to fewer bytes than requested (down to zero) depending on the allocation's base address
• the deprecated surface produces identical output by delegation, and the on-disk record format and decode semantics are unchanged

New public API

VectorReader

fluent entry point. from(Source) chooses the source, then fixed(Buffers) or growable(ByteOrder) chooses buffering, then bytes, shorts, ints, longs, floats, or doubles() returns a CloseableIterator and throws IOException.

try (CloseableIterator<float[]> vectors = VectorReader.from(Source.file(path))
        .fixed(Buffers.heap(1 << 16))
        .floats()) {
    while (vectors.hasNext()) {
        float[] vector = vectors.next();
    }
}

Source

functional interface opening a ReadableByteChannel, with factories file, directFile, stream, channel, and bytes. file, directFile, and bytes are re-openable, while stream and channel are single-use.

Buffers

record describing buffer allocation (capacity, order, placement, alignment), with heap and direct factories and a public BLOCK_ALIGNMENT constant.

CloseableIterator<E>

Closeable resource iterator, the return type of the reader factories.

VectorConverters

correctly spelled replacement for VectorConvertors, with the same conversions as static factories.

VectorFileIterators

gained static factories bytes, shorts, ints, longs, floats, and doubles(Path) returning a CloseableIterator with a default 256 KiB little-endian heap buffer. VectorFileIterators itself is not deprecated.

Deprecated for removal

Still functional, delegating to the new API:

• Options, replaced by Source and Buffers
• VectorConvertors and its nested iterators, replaced by VectorConverters
• the nested VectorFileIterators per-type classes taking (Path, Options)

Behavior

The deprecated surface produces identical output by delegation. The on-disk record format and decode semantics are unchanged.

Internal

ByteSource was renamed to ByteWindow, and VectorSourceIterator to VectorRecordIterator.

Internal

The decoding path was decoupled from raw channel and buffer access, separating the physical read, the buffered view, and record framing:

• decoding is independent of the buffering strategy, allowing either a shared read-ahead buffer or a per-record growable buffer
• physical reads sit behind their own abstraction, with clearer blocking-mode handling

Testing expanded with an integration test across all vector types and byte orders, plus unit tests for the new internal abstractions.

Behavior

• direct-IO buffers are now block-aligned, fixing a direct-IO read bug
• a negative dimension prefix is rejected with IOException
• read-failure messages were split into distinct cases

Internal

Converters no longer use reflection, and decoding moved behind a package-private abstraction, with no public impact. Test coverage expanded across decoders, iterators, converters, and options, including a Linux-only direct-IO read.

New public API

VectorFileIterators

provides six iterators, one per component type: ByteVectorFileIterator, ShortVectorFileIterator, IntVectorFileIterator, LongVectorFileIterator, FloatVectorFileIterator, DoubleVectorFileIterator. Each iterates the matching primitive array, takes (Path, Options), and is AutoCloseable (closing releases the FileChannel).

Options options = new Options(1 << 20, ByteOrder.LITTLE_ENDIAN, false);
try (FloatVectorFileIterator vectors = new FloatVectorFileIterator(path, options)) {
    while (vectors.hasNext()) {
        float[] vector = vectors.next();
    }
}

Options

record Options(int capacity, ByteOrder order, boolean directIO) for buffer capacity, byte order, and optional direct IO.

VectorConvertors

provides lazy iterator adapters: unsigned-byte reinterpretation, and byte, short, int, long, and double to float.

util types

Resource (an AutoCloseable base with Iterator and Iterable variants), PrefetchingIterator, and LazyThrowingSingleton, used by the file iterators.

Behavior

• lazy iteration with prefetching and lazy buffer allocation
• read failures during iteration surface as UncheckedIOException
• a vector larger than the buffer capacity or the remaining data raises IOException

Internal

Javadoc is provided on all public types.