wireform-arrow

wireform-arrow implements Apache Arrow IPC and the Arrow columnar data model in Haskell. Arrow is the interchange format between analytics engines, dataframe libraries, and columnar storage. Use this package when you need typed record batches, schema-aware encoding, or a shared column vocabulary that Parquet and ORC readers in wireform can target.

Key features

• Schema framing and IPC message encode/decode via Arrow.IPC
• Record batch encode and decode for in-memory columnar data
• Typed record API with Template Haskell and Generic support
• Table projection and subsetting to read only the columns you need
• Optional compression (Zstd and LZ4 behind Cabal flags)
• Zero-copy decode: columns are Arrow-native buffers that alias the input; only offsets, UTF-8 and dictionary keys are validated (C kernels)
• Single-allocation encode, plus lazy encoders whose chunks alias the column buffers
• Streaming reader for framed IPC streams in Arrow.Stream

Basic usage

Define a record type, build a Table, encode a vector of records to an Arrow IPC stream with Arrow.Stream, and decode it back:

{-# LANGUAGE DerivingStrategies #-}
{-# LANGUAGE OverloadedStrings #-}
module Trades where

import Arrow.Record
import Arrow.Stream (decodeArrowStream, defaultWriteOptions, encodeArrowStream)
import Data.ByteString (ByteString)
import Data.Int (Int32)
import Data.Text (Text)
import qualified Data.Vector as V

data Trade = Trade
  { tradeSym  :: !Text
  , tradeQty  :: !Int32
  , tradeNote :: !(Maybe Text)
  }
  deriving stock (Show, Eq)

tradeTable :: Table Trade
tradeTable = table enc dec
  where
    enc =
      fieldE "sym"  tradeSym  utf8E
        <> fieldE "qty"  tradeQty  int32E
        <> fieldE "note" tradeNote (nullable utf8E)
    dec =
      Trade
        <$> columnD "sym"  utf8D
        <*> columnD "qty"  int32D
        <*> columnD "note" (nullableD utf8D)

-- | One record batch in an IPC stream. 'Left' when the columns do not
-- fit the schema (they always do for a 'Table').
encodeTrades :: V.Vector Trade -> Either String ByteString
encodeTrades trades =
  let (schema, columns) = encodeTable tradeTable trades
  in encodeArrowStream defaultWriteOptions schema [columns]

-- | Every record of every batch in the stream.
decodeTrades :: ByteString -> Either String (V.Vector Trade)
decodeTrades bytes = do
  (schema, batches) <- decodeArrowStream bytes
  V.concat <$> traverse (decodeTable tradeTable schema) batches

roundTripTrades :: V.Vector Trade -> Either String (V.Vector Trade)
roundTripTrades trades = encodeTrades trades >>= decodeTrades

encodeArrowFile / decodeArrowFile do the same for the Arrow file format. The writers check every batch against the schema and return Left for a mismatch, or for dictionaries they cannot write (an emit-once dictionary larger than its index type can address).

Project column batches down to a subset of fields when the full schema is larger than what your query needs:

{-# LANGUAGE OverloadedStrings #-}
import Arrow.Column (ColumnArray)
import Arrow.Record (projectTable)
import Arrow.Types (Schema)
import qualified Data.Vector as V

projectSymQty :: Schema -> V.Vector ColumnArray -> Maybe (Schema, V.Vector ColumnArray)
projectSymQty schema cols =
  projectTable ["sym", "qty"] schema cols

Arrow.File reads the file format and exposes the raw record batches; Arrow.Stream.openStreamReader reads a stream one batch at a time.

Columns

A ColumnArray holds Arrow’s own buffers: storable vectors for fixed-width values and offsets, LSB-first bitmaps for validity and booleans, and ByteString regions for variable-length data. Nullability is per value: every array with a validity slot carries Maybe Validity, Nothing meaning no nulls. The patterns (ColPrim, ColInt64, ColUtf8, …) only match; build with primColumn, the from* conversions, the builders, or the validating mk* constructors:

{-# LANGUAGE OverloadedStrings #-}
module Build where

import Arrow.Column
import Control.Monad.ST (runST)
import qualified Data.Vector as V
import qualified Data.Vector.Storable as VS

-- O(1): the vector becomes the values buffer.
prices :: ColumnArray
prices = primColumn PDouble (VS.fromList [101.5, 99.25, 100])

-- A builder appends into pinned, growable buffers; the validity bitmap
-- is allocated on the first null.
quantities :: ColumnArray
quantities = runST $ do
  b <- newPrimBuilder PInt64 3
  appendPrim b 10
  appendNull b
  appendPrim b 30
  freezeBuilder b

-- Convenience constructors from boxed vectors (O(n)).
notes :: ColumnArray
notes = fromMaybeTexts (V.fromList [Just "filled", Nothing, Just "late"])

-- Nested columns: build the children, then validate with mk*.
fills :: Either String ColumnArray
fills = mkList Nothing (VS.fromList [0, 2, 2, 3]) (primColumn PInt32 (VS.fromList [5, 7, 9]))

Read through typed views: bind asPrim, asUtf8, asBinary or asBool once per column, then index with primAt, textAt, bytesAt or boolArrayAt (listRange for list and map rows, dictKeyAt for dictionary rows).

For a whole column of Haskell values, the to*Vector conversions return an Arrow.Vector.Vector: a data family (like Data.Vector.Unboxed) used through Data.Vector.Generic with element types such as Maybe Int64 or Maybe Text, stored the way the column is. A Vector (Maybe Int64) is the column’s validity bitmap plus its storable values (8 bytes and one bit per row, no heap object per row), and fused folds over it compile to a loop that allocates nothing per row. toMaybeVector and toBoolVector are O(1) (they alias the column; a column without nulls gets a fresh all-set bitmap); toTextVector, toBytesVector and toListVector are O(rows) (offsets widened to Int), with every row a slice of the column’s data: Text rows are neither copied nor re-validated (data outside the GHC heap, such as an mmapped file, is copied once), views copy their bytes once, and dictionaries convert their values once and share them. The inverses fromMaybeVector and fromBoolVector are O(1) plus a popcount. G.convert gives a boxed Data.Vector; G.force copies the rows into fresh buffers, detaching them from the input.

{-# LANGUAGE BangPatterns #-}
module Read where

import Arrow.Column
import Data.Int (Int64)
import Data.Text (Text)
import qualified Data.Vector as V
import qualified Data.Vector.Generic as G

-- Bind the typed view once, then index; the loop does not allocate.
sumQuantities :: ColumnArray -> Maybe Int64
sumQuantities c = do
  p <- asPrim PInt64 c
  let n = primArrayLength p
      go !acc i
        | i >= n = acc
        | otherwise = go (acc + maybe 0 id (primAt p i)) (i + 1)
  pure (go 0 0)

-- The same sum over the whole column: O(1) conversion, fused fold.
sumQuantities' :: ColumnArray -> Maybe Int64
sumQuantities' c = G.foldl' (\acc m -> maybe acc (+ acc) m) 0 . toMaybeVector <$> asPrim PInt64 c

-- One copy into a Text, no UTF-8 re-validation.
noteAt :: ColumnArray -> Int -> Maybe Text
noteAt c i = asUtf8 c >>= \u -> textAt u i

-- A boxed Data.Vector when an API wants one.
quantityList :: ColumnArray -> Maybe (V.Vector (Maybe Int64))
quantityList c = G.convert . toMaybeVector <$> asPrim PInt64 c

Decoded columns alias the input bytes and keep them alive. Holding a small column from a large message pins the whole message; copyColumn copies only the bytes the column references:

module Lifetime where

import Arrow.Column (ColumnArray, copyColumn)
import Arrow.Stream (decodeArrowStream)
import Data.ByteString (ByteString)
import qualified Data.Vector as V

-- Decoded columns alias 'bytes'. Keeping one small column from a large
-- message would keep the whole message alive; copyColumn detaches it.
firstColumn :: ByteString -> Either String ColumnArray
firstColumn bytes = do
  (_, batches) <- decodeArrowStream bytes
  case batches of
    cols : _ | not (V.null cols) -> Right (copyColumn (V.head cols))
    _ -> Left "no columns"

Dictionary keys are an integer column at the wire width of the field’s index type and carry the row validity. Eq is logical: same type, same length, the same validity per row and equal values in valid rows; null slot contents, offsets, slices and dictionary layouts do not matter.

The eager encoders write the whole output into one allocation. The lazy encoders return chunks that alias the column buffers, so their cost follows the buffer count rather than the data size:

module Lazy where

import Arrow.Column (ColumnArray)
import Arrow.Stream (defaultWriteOptions, encodeArrowStreamLazy)
import Arrow.Types (Schema)
import qualified Data.ByteString.Lazy as BL
import qualified Data.Vector as V
import System.IO (Handle)

-- The chunks are the message headers and the column buffers themselves:
-- no body copy, so the cost tracks the buffer count, not the data size.
writeBatches :: Handle -> Schema -> [V.Vector ColumnArray] -> IO (Either String ())
writeBatches h schema batches =
  traverse (BL.hPut h) (encodeArrowStreamLazy defaultWriteOptions schema batches)

Migrating from the boxed constructors

The boxed constructors and their *Maybe twins (ColInt64Maybe (V.Vector (Maybe Int64)), ColUtf8 (V.Vector Text), ColListMaybe valid offsets child, …) are gone. ColInt64Maybe v becomes fromMaybes PInt64 v; ColUtf8Maybe v becomes fromMaybeTexts v; ColListMaybe valid offs child becomes mkList (validityFromBools valid) offs child; ColDictionary i keys vals becomes mkDictionary i keysColumn vals; matching on ColInt64Maybe v becomes matching ColInt64 validity v or reading through asPrim PInt64; isNullableColumn c becomes nullCount c > 0.

{-# LANGUAGE OverloadedStrings #-}
module Migrate where

import Arrow.Column
import Data.Int (Int32)
import Data.Text (Text)
import qualified Data.Vector as V
import qualified Data.Vector.Storable as VS

-- Before: ColInt64Maybe (V.fromList [Just 1, Nothing])
nullableIds :: ColumnArray
nullableIds = fromMaybes PInt64 (V.fromList [Just 1, Nothing])

-- Before: ColUtf8 (V.fromList ["a", "b"])
names :: ColumnArray
names = fromTexts (V.fromList ["a", "b"])

-- Before: ColDictionary 0 (VP.fromList [0, 1, 0]) (ColUtf8 ...)
-- Keys are an integer column at the wire width and carry the row nulls.
colours :: Either String ColumnArray
colours =
  mkDictionary 0 (fromMaybes PInt32 (V.fromList [Just 0, Nothing, Just 1]))
    (fromTexts (V.fromList ["red", "green"]))

-- Before: case c of ColInt64Maybe v -> ...; ColInt64 v -> ...
-- After: one pattern; the validity is Nothing when there are no nulls.
describe :: ColumnArray -> String
describe c = case c of
  ColInt64 Nothing v -> "dense int64 x" <> show (VS.length v)
  ColInt64 (Just m) _ -> show (validityNullCount m) <> " nulls"
  _ -> columnTag c

-- Before: ColListMaybe valid offsets child
nullableLists :: Either String ColumnArray
nullableLists =
  mkList (validityFromBools (V.fromList [True, False])) (VS.fromList [0, 1, 1])
    (primColumn PInt32 (VS.fromList [7 :: Int32]))

-- Equality is logical: layout differences that do not change the rows
-- (slices, offset bases, null slot contents) compare equal.
sameRows :: Bool
sameRows = sliceColumnArray 1 1 names == fromTexts (V.fromList ["b" :: Text])

Notable modules

Module	Purpose
Arrow.Types	Schema, field, and buffer types; schemaFingerprint
Arrow.Column	ColumnArray (Arrow-native buffers, Maybe Validity), patterns, typed accessors, from*/mk* construction, copyColumn, slice/concat/take, logical Eq
Arrow.Column.Builder	Growable pinned builders in ST/IO (re-exported by Arrow.Column)
Arrow.Column.Buffer	Bitmap, Validity, fixed-width element types, aligned allocation
Arrow.Column.Internal	Raw constructors and structural validators, for readers, writers and malformed-input tests
Arrow.Vector	Vector a data family behind the Data.Vector.Generic API in Arrow layout (storable values, bitmaps, validity plus values for Maybe a, offsets into one store for Text, ByteString and lists); what the to*Vector conversions return
Arrow.Read.Columns	Record batch to columns: buffer slicing, validation, zero-copy aliasing
Arrow.Record	Typed Table, Encoder, Decoder, projection helpers
Arrow.Record.Generic / Arrow.Record.TH	Generic and TH record derivation
Arrow.Derive	Annotation-driven deriver
Arrow.IPC	IPC message framing encode/decode
Arrow.Stream	Stream and file encode / decode (with dictionaries, including nested ones), lazy encoders, pull-based streaming reader
Arrow.File	Arrow file format readers
Arrow.FlatBufferIPC	FlatBuffer-backed IPC metadata path (re-exports .Common, .Read, .Write)
Arrow.Write	writeArrowStream / writeArrowFile, column encoders, validateColumns

Compression

Enable Zstd or LZ4 with Cabal flags (+zstd, +lz4). Compressed IPC streams follow the standard Arrow body compression layout; uncompressed IPC remains the default for maximum interoperability.

Performance

One record batch per workload through Arrow.Stream, set against arrow-rs 58 building the same batches from the same generators. Four groups per workload: encode (one allocation; arrow-rs StreamWriter into a Vec<u8>), encode lazy (chunks aliasing the column buffers; arrow-rs has no lazy writer, so it is set against StreamWriter), decode (zero copy; arrow-rs StreamReader with its default validation on), and decode + toVector (decode, convert every column with the to*Vector conversions, then G.force each result into fresh buffers so it owns its data; arrow-rs copies into owned Vec<Option<T>>, String and nested Vec values). Dictionary rows share one owned copy of the values instead of one string per row. Inputs are built outside the timed region. The ratio column is wireform-arrow time over arrow-rs time, so 2.00x means wireform-arrow takes twice as long. Run both, sequentially, with python3 scripts/run-benchmarks.py --only arrow --render; the harnesses are wireform-arrow/bench/Bench.hs and interop/arrow-rs/benches/arrow_ipc.rs.

The bench binary runs with a 64 MB nursery (-with-rtsopts=-A64m). Decoding, lazy encoding and the Arrow.Vector conversions barely allocate, but code that builds boxed values (the typed Arrow.Record rows, G.convert to a boxed vector, your own conversions) is dominated by minor collections at GHC’s default 4 MB nursery. Programs that move large batches should do the same: link with -rtsopts and run with +RTS -A64m, or bake it in with -with-rtsopts.

Encode/decode, 100k rows

Operation	arrow-rs	wireform-arrow	ratio
encode int64	39.3 µs	10.7 µs	0.27x
encode double	39.5 µs	10.3 µs	0.26x
encode nullable int64	37.8 µs	10.7 µs	0.28x
encode utf8	45.2 µs	12.9 µs	0.28x
encode nullable utf8	43.5 µs	12.5 µs	0.29x
encode mixed 6-col	413 µs	61.7 µs	0.15x
encode list	93.9 µs	26.8 µs	0.29x
encode struct<int32,double,bool>	63.0 µs	16.4 µs	0.26x
encode dictionary	31.4 µs	6.05 µs	0.19x
encode lazy int64	39.3 µs	0.63 µs	0.02x
encode lazy double	39.5 µs	0.61 µs	0.02x
encode lazy nullable int64	37.8 µs	0.65 µs	0.02x
encode lazy utf8	45.2 µs	0.69 µs	0.02x
encode lazy nullable utf8	43.5 µs	0.69 µs	0.02x
encode lazy mixed 6-col	413 µs	2.07 µs	0.01x
encode lazy list	93.9 µs	0.91 µs	0.01x
encode lazy struct<int32,double,bool>	63.0 µs	1.36 µs	0.02x
encode lazy dictionary	31.4 µs	1.12 µs	0.04x
decode int64	13.1 µs	0.48 µs	0.04x
decode double	13.1 µs	0.46 µs	0.04x
decode nullable int64	13.4 µs	0.75 µs	0.06x
decode utf8	125 µs	48.0 µs	0.38x
decode nullable utf8	123 µs	47.8 µs	0.39x
decode mixed 6-col	293 µs	97.4 µs	0.33x
decode list	110 µs	6.08 µs	0.06x
decode struct<int32,double,bool>	20.9 µs	1.04 µs	0.05x
decode dictionary	34.1 µs	6.29 µs	0.18x
decode + toVector int64	159 µs	15.0 µs	0.09x
decode + toVector double	171 µs	14.6 µs	0.08x
decode + toVector nullable int64	160 µs	14.3 µs	0.09x
decode + toVector utf8	1411 µs	241 µs	0.17x
decode + toVector nullable utf8	1458 µs	297 µs	0.20x
decode + toVector mixed 6-col	3458 µs	556 µs	0.16x
decode + toVector list	3702 µs	189 µs	0.05x
decode + toVector struct<int32,double,bool>	330 µs	25.7 µs	0.08x
decode + toVector dictionary	1338 µs	401 µs	0.30x

_{Last run 2026-10-09 11:04:54 UTC. ghc-9.8.4 on darwin-aarch64, criterion 1.6.5; arrow-rs 58.2.0, rustc 1.98.1, criterion.rs 0.7.0.}

Encode/decode, 100-row batch

Operation	arrow-rs	wireform-arrow	ratio
encode int64	0.98 µs	0.63 µs	0.64x
encode double	1.00 µs	0.62 µs	0.62x
encode nullable int64	0.98 µs	0.64 µs	0.65x
encode utf8	1.07 µs	0.66 µs	0.62x
encode nullable utf8	1.06 µs	0.66 µs	0.62x
encode mixed 6-col	2.56 µs	2.11 µs	0.82x
encode list	1.42 µs	0.93 µs	0.65x
encode struct<int32,double,bool>	1.94 µs	1.37 µs	0.71x
encode dictionary	2.24 µs	1.09 µs	0.49x
encode lazy int64	0.98 µs	0.63 µs	0.64x
encode lazy double	1.00 µs	0.61 µs	0.61x
encode lazy nullable int64	0.98 µs	0.65 µs	0.66x
encode lazy utf8	1.07 µs	0.69 µs	0.64x
encode lazy nullable utf8	1.06 µs	0.71 µs	0.67x
encode lazy mixed 6-col	2.56 µs	2.08 µs	0.81x
encode lazy list	1.42 µs	0.91 µs	0.64x
encode lazy struct<int32,double,bool>	1.94 µs	1.36 µs	0.70x
encode lazy dictionary	2.24 µs	1.12 µs	0.50x
decode int64	0.63 µs	0.48 µs	0.76x
decode double	0.62 µs	0.46 µs	0.74x
decode nullable int64	0.66 µs	0.50 µs	0.76x
decode utf8	0.74 µs	0.57 µs	0.77x
decode nullable utf8	0.76 µs	0.57 µs	0.75x
decode mixed 6-col	2.25 µs	1.54 µs	0.68x
decode list	1.08 µs	0.69 µs	0.64x
decode struct<int32,double,bool>	1.29 µs	1.04 µs	0.81x
decode dictionary	1.36 µs	0.88 µs	0.65x
decode + toVector int64	0.73 µs	0.57 µs	0.78x
decode + toVector double	0.72 µs	0.56 µs	0.78x
decode + toVector nullable int64	0.79 µs	0.58 µs	0.73x
decode + toVector utf8	1.93 µs	1.02 µs	0.53x
decode + toVector nullable utf8	1.94 µs	1.02 µs	0.53x
decode + toVector mixed 6-col	5.25 µs	2.76 µs	0.53x
decode + toVector list	4.76 µs	1.26 µs	0.26x
decode + toVector struct<int32,double,bool>	1.55 µs	1.29 µs	0.83x
decode + toVector dictionary	2.54 µs	1.56 µs	0.61x

_{Last run 2026-10-09 11:04:54 UTC. ghc-9.8.4 on darwin-aarch64, criterion 1.6.5; arrow-rs 58.2.0, rustc 1.98.1, criterion.rs 0.7.0.}

Entry points, mixed 6-column table

Operation	arrow-rs	wireform-arrow	ratio
stream encode (Arrow.Stream)	393 µs	61.5 µs	0.16x
stream encode lazy (Arrow.Stream)	393 µs	2.07 µs	0.01x
stream encode (Arrow.Write)	393 µs	61.2 µs	0.16x
stream decode (Arrow.Stream)	277 µs	96.8 µs	0.35x
file encode (Arrow.Stream)	397 µs	64.2 µs	0.16x
file encode lazy (Arrow.Stream)	397 µs	3.69 µs	0.01x
file decode (Arrow.Stream)	277 µs	97.4 µs	0.35x
file read (Arrow.File)	277 µs	97.8 µs	0.35x
typed encode (Arrow.Record)	1645 µs	1470 µs	0.89x
typed decode (Arrow.Record)	3076 µs	2944 µs	0.96x

_{Last run 2026-10-09 11:04:54 UTC. ghc-9.8.4 on darwin-aarch64, criterion 1.6.5; arrow-rs 58.2.0, rustc 1.98.1, criterion.rs 0.7.0.}

The entry-point rows map to the closest arrow-rs path: StreamWriter for every stream writer (eager, lazy, Arrow.Write), FileWriter for both file writers, FileReader for both file readers, and for the typed rows a Vec of row structs turned into one array per field (or read back into owned row structs), since arrow-rs has no record deriving. Comparisons against pyarrow and arrow-cpp are not yet measured.