Skip to main content

Parquet line reader

Read a Parquet file row by row. The reader streams the file in record batches, so it can process large files without loading them entirely in memory.

The reader works in two modes, depending on whether you declare columns:

  • No column declared — the node exposes a single Parquet Row output and each row is emitted as one object holding every column of the file, keyed by column name.
  • Columns declared — the node exposes one output connector per declared column, labeled <column name> (<value type>), and only those columns are extracted from each row.
A Parquet file
| question | answer  | score |
|----------|---------|-------|
| Q1...    | A1...   | 0.9   |
| Q2...    | A2...   | 0.7   |

// Configuration:
// Columns: question (Text), score (Number)

// Output connectors:
// question (Text), score (Number)

Supported Parquet column types are text, integers, unsigned integers, floats, booleans, lists and structs. Null cells are emitted as empty values, and declared columns that do not exist in the file are simply skipped.

Parameters​

Columns to extract

The list of columns to read from every Parquet record. Each row of the table declares:

  • Key — the name of the column in the Parquet file.
  • Value Type — the type the extracted value is cast into (Text by default).
  • Mandatory — whether the column is required. Records missing a mandatory column fail the run; non-mandatory columns produce optional outputs.

Leave the list empty to emit each row as a single object containing all columns.

Output​

Parquet Row

When no column is declared: the current row being read, as one object keyed by column name.

One connector per declared column

When columns are declared: each declared column gets its own output connector, labeled <column name> (<value type>). The connector is optional unless the column is marked mandatory.