Skip to main content

Parquet line writer

Generate a Parquet file row by row. Each declared column becomes an input connector of the node, and every execution of the writer appends one row to the file.

Example
// Configuration:
// Columns: question (Text), score (Number), valid (Boolean)

// Input connectors:
// question (Text), score (Number), valid (Boolean)

// Output file (one row per execution):
| question | score | valid |
|----------|-------|-------|
| Q1...    | 0.9   | true  |
| Q2...    | 0.7   | false |

The Parquet schema is derived from the declared value types: Boolean columns are written as Parquet booleans, Number columns as 64-bit floats, and every other type (Text, JSON, XML, Date, Datetime, Time, Location, ...) as text. A missing or incompatible value is written as a null cell.

Rows are buffered and written to the file in batches of 1000, and the resulting file is compressed with Snappy.

Parameters​

Columns to write

The list of columns written to every Parquet record. Each row of the table declares:

  • Column — the name of the column in the produced Parquet file.
  • Value Type — the type used in the Parquet schema (Text by default).
  • Mandatory — whether the column is required. Records missing a mandatory column fail the run; non-mandatory columns accept missing values, written as nulls.

Input​

One connector per declared column

Each declared column gets its own input connector, labeled <column name> (<value type>). The connector is optional unless the column is marked mandatory.