Skip to content
wireform

wireform-parquet

wireform-parquet implements the Apache Parquet columnar file format. Parquet is the on-disk format behind most data warehouses and lakehouse table formats (Iceberg, Delta Lake, Hudi). Use this package when you need to read or write Parquet files directly in Haskell, with support for the encodings and compression codecs that real-world writers emit.

Key features

Section titled “Key features”
  • Full read and write via Parquet.HighLevel and lower-level page APIs
  • All major encodings: PLAIN, dictionary, DELTA_BINARY_PACKED, BYTE_STREAM_SPLIT, and hybrid RLE index pages
  • Compression codecs behind Cabal flags: Snappy, Zstd, LZ4, Gzip, and Brotli
  • Bloom filters and page indexes for sub-row-group predicate pruning
  • Page integrity: optional CRC32 checksums verified before decompression
  • Modular column encryption (AES-GCM) per the Parquet Modular Encryption spec
  • Predicate pushdown over footer statistics, page indexes, and bloom filters
  • Nested columns (lists, maps, structs) via Parquet.Nested
  • Arrow bridge for typed record batches through Parquet.Arrow
  • Template Haskell deriver via Parquet.Derive
  • Interop-tested against pyarrow

Basic usage

Section titled “Basic usage”

Most callers start with the high-level decode API for in-memory bytes, or openParquetReader for mmap-aware streaming over large files on disk:

import qualified Data.Vector        as V
import qualified Parquet.HighLevel  as PH
import qualified Parquet.Read       as PR
import qualified Parquet.Types      as PT


readParquetBytes :: ByteString -> IO ()
readParquetBytes bytes =
  case PH.decodeParquet PH.defaultReadOptions bytes of
    Left err ->
      putStrLn err
    Right pf ->
      let fm = PR.pfFooter pf
      in putStrLn $
           "rows="
             ++ show (PT.fmNumRows fm)
             ++ " rowGroups="
             ++ show (V.length (PT.fmRowGroups fm))


readParquetFile :: FilePath -> IO ()
readParquetFile path = do
  result <- PR.openParquetReader path
  case result of
    Left err ->
      putStrLn err
    Right (pf, _rowGroupIter) ->
      let fm = PR.pfFooter pf
      in putStrLn $
           "rows="
             ++ show (PT.fmNumRows fm)
             ++ " rowGroups="
             ++ show (V.length (PT.fmRowGroups fm))

For writing, pass an Arrow-shaped schema and column batches to encodeParquet:

import qualified Parquet.HighLevel as PH


writeParquet :: PH.Schema -> [V.Vector PH.ColumnData] -> ByteString
writeParquet schema rowGroups =
  PH.encodeParquet PH.defaultWriteOptions schema rowGroups

When you need projection or filter pushdown without loading every column, use the predicate and aggregate modules together with the Arrow bridge or the cross-format Wireform.Columnar facade.

Statistics-only min/max requires exact bounds; compact bounds still support filtering. Non-null row counts support nullable ancestors but return Nothing for repeated paths or insufficient metadata. fileRowCountChecked also verifies the file total against its row groups without overflow.

The schema-aware Arrow writers convert millisecond dates to INT32 days and second-resolution times/timestamps to milliseconds. Invalid dates/times, timestamp overflow, and durations without a Parquet logical type return an error. The inferred read schema uses the normalized units. Fixed-width binary, Decimal128/256, and FLOAT16 values are supported by the reader; legacy INT96 values must fit a signed 64-bit nanosecond timestamp.

For nested Arrow data, use arrowFieldToNestedSchema field together with columnArrayToNestedRows field column, then encodeParquetNested. The field controls integer annotations and temporal units, including within structs, lists, and maps. Hidden values beneath null parents or outside referenced child ranges are ignored; visible required nulls and null map keys are rejected. Both Arrow list widths use standard Parquet LIST storage. Nested booleans use bit-packed PLAIN values.

Nested writers check row/event counts and the complete V2 page payload against signed 32-bit limits. The size checks run while shredding, and all leaf level streams are planned before value buffers are materialized. For pre-shredded optional lists, buildOptionalListFileEither also validates level ranges, row boundaries, paths, and PLAIN payload sizes. The older buildOptionalListFile raises an error for invalid input. Parquet.Footer.writeFooterEither and writeRawFooterEither expose footer length/trailer errors without an exception. Typed metadata is measured before UTF-8 conversion or output allocation. Encrypted footers include crypto metadata, the nonce and authentication tag in the same checked budget.

parquetRowGroupToArrow projects and reorders nested struct children by name, widens signed Int32 to Int64 and Float to Double, and builds 64-bit offsets for requested LargeList fields. List/map wrapper names are positional. Omitted physical leaves are not decoded, and required targets reject visible nulls before reconstruction.

parquetFileArrowSchemaChecked reconstructs struct, list and map fields, including legacy list layouts. parquetRowGroupToArrow preserves null and empty containers, null elements and duplicate leaf names under different parents. It rejects inconsistent ancestor levels and malformed schema trees. Legacy timestamp annotations infer UTC; JSON columns use Arrow’s canonical arrow.json metadata, which the writers also preserve.

Performance

Section titled “Performance”

XXH64 hash: C kernel vs pure Haskell

Section titled “XXH64 hash: C kernel vs pure Haskell”
Parquet XXH64 hash: C kernel vs pure Haskell across input sizes Parquet XXH64 hash: C kernel vs pure Haskell across input sizes lower is better · ns · ghc-9.8.4 on darwin-aarch64, criterion 1.6.5 0 12500 25000 37500 50000 12.1 10.3 18.1 43.0 79.9 457 4387 28291 8 B 64 B 1 KiB 64 KiB C kernel pure Haskell Parquet XXH64 hash: C kernel vs pure Haskell across input sizes lower is better · ns · ghc-9.8.4 on darwin-aarch64, criterion 1.6.5 0 12500 25000 37500 50000 12.1 10.3 18.1 43.0 79.9 457 4387 28291 8 B 64 B 1 KiB 64 KiB C kernel pure Haskell
OperationC kernelpure Haskellratio
8 B12.1 ns10.3 ns0.85x
64 B18.1 ns43.0 ns2.37x
1 KiB79.9 ns457 ns5.71x
64 KiB4387 ns28291 ns6.45x

Last run 2026-06-27 11:45:59 UTC. ghc-9.8.4 on darwin-aarch64, criterion 1.6.5.

The C kernel pulls ahead of the pure Haskell fallback from 64 bytes up. At 8 bytes the pure path wins slightly due to call overhead. The C path is the default when the FFI is available. Page-level decode throughput depends heavily on encoding (PLAIN, DELTA_BINARY_PACKED, RLE/bitpacked) and compression codec.

The chart and table above are regenerated by wireform-stats from wireform-parquet/bench-results/summary/parquet-xxh64-c-vs-pure.json — the same source the README chart is built from.

Notable modules

Section titled “Notable modules”
ModulePurpose
Parquet.HighLevelencodeParquet, decodeParquet, WriteOptions, ReadOptions
Parquet.ReadloadParquetFilePath, openParquetReader, column chunk decoders
Parquet.WritePage encoders, row group assembly, buildParquetFile
Parquet.FooterThrift-encoded footer parse and emit
Parquet.Page / Parquet.PageIndexData page headers and per-page statistics
Parquet.BloomFilterSplit-block bloom filter decode
Parquet.EncryptionColumn-level and footer encryption (PME, AES-GCM)
Parquet.PredicateStatistics and bloom-filter predicate evaluation
Parquet.Aggregatecount(*), count(col), min, max from footer stats
Parquet.ArrowParquet columns to Arrow ColumnArray bridge
Parquet.DeriveTemplate Haskell deriver with wireform-derive annotations

Interop

Section titled “Interop”

The reader handles files produced by pyarrow, parquet-cpp, and arrow-rs, including dictionary-encoded strings, delta-packed integers, and BYTE_STREAM_SPLIT floats. Cross-language round-trip tests live in the package probe suite.