Skip to content
GitHub

DataSet

DataSet manifests describe derived tables built using SQL or PRQL (Pipelined Relational Query Language). Datasets typically select from DataSource views and may depend on other datasets.

The smallest well-formed DataSet: the envelope, a name, a query and the datasources the query reads. dependencies is not required by the schema, but without it the dataset cache does not notice when the underlying file changes, so set it whenever the query reads a datasource.

apiVersion: bino.bi/v1alpha1
kind: DataSet
metadata:
  name: sales_by_region
spec:
  query: |
    SELECT
      region AS category,
      row_number() OVER (ORDER BY region) AS categoryIndex,
      SUM(amount) AS ac1
    FROM sales_csv
    GROUP BY region
  dependencies:
    - sales_csv

Instead of query you can write the transformation in PRQL (spec.prql) or skip it entirely with a direct source pass-through (spec.source). All attributes are listed in the Attribute Reference below.

Fields:

  • spec.query – SQL query, either inline or loaded from an external file. Required if spec.prql and spec.source are not set.
  • spec.prqlPRQL query, either inline or loaded from an external file. When set, takes precedence over spec.query. The PRQL is sent directly to the query engine which compiles it via the prql extension.
  • spec.source – direct pass-through to a DataSource without transformation. Mutually exclusive with query and prql.
  • spec.dependencies – optional list of datasource references. Can be string names or inline DataSource definitions. Inline definitions are referenced via @inline(N) syntax in queries.

For complex queries, you can store them in separate .sql or .prql files and reference them using the $file syntax. This keeps your YAML manifests clean and enables syntax highlighting in your editor.

apiVersion: bino.bi/v1alpha1
kind: DataSet
metadata:
  name: sales_summary
spec:
  query:
    $file: ./queries/sales_summary.sql
  dependencies:
    - sales_csv
apiVersion: bino.bi/v1alpha1
kind: DataSet
metadata:
  name: customer_orders
spec:
  prql:
    $file: ./queries/customer_orders.prql
  dependencies:
    - orders
    - customers
  • Paths are resolved relative to the manifest file containing the $file reference
  • Both ./relative/path.sql and relative/path.sql formats are supported
  • Absolute paths are also supported but not recommended for portability

External query files are fully integrated with bino's caching and hot-reload system:

  • Cache invalidation: When an external SQL or PRQL file changes, the dataset cache is automatically invalidated
  • Hot-reload in preview: Changes to external query files trigger automatic refresh in bino preview mode
  • Dependency graph: External files appear in bino graph output for visibility

A recommended project structure for external queries:

my-report/
├── bino.toml
├── manifests/
│   ├── datasources.yaml
│   └── datasets.yaml
├── queries/
│   ├── sales_by_region.sql
│   ├── monthly_revenue.sql
│   └── customer_analysis.prql
└── data/
    └── sales.csv
---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: sales_csv
spec:
  type: csv
  path: ./data/sales.csv
---
apiVersion: bino.bi/v1alpha1
kind: DataSet
metadata:
  name: sales_by_region
spec:
  query: |
    SELECT
      region,
      SUM(amount) AS total_amount
    FROM sales_csv
    GROUP BY region
  dependencies:
    - sales_csv
---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: sales_fact
spec:
  type: parquet
  path: ./warehouse/fact_sales/*.parquet
---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: currency_rates
spec:
  type: csv
  path: ./data/currency_rates.csv
---
apiVersion: bino.bi/v1alpha1
kind: DataSet
metadata:
  name: revenue_eur
spec:
  query: |
    SELECT
      f.region,
      f.booking_date,
      f.amount * r.rate_to_eur AS revenue_eur
    FROM sales_fact f
    JOIN currency_rates r
      ON r.currency = f.currency
     AND r.valid_on = f.booking_date;
  dependencies:
    - sales_fact
    - currency_rates

Datasets are referenced from layouts via their metadata.name using the dataset field of charts, tables, and text components.

PRQL is a modern language for transforming data that compiles to SQL. It offers a more readable, pipeline-based syntax compared to traditional SQL.

When you specify spec.prql, bino automatically loads the PRQL extension and sends your PRQL query directly to the query engine for compilation and execution.

---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: sales_csv
spec:
  type: csv
  path: ./data/sales.csv
---
apiVersion: bino.bi/v1alpha1
kind: DataSet
metadata:
  name: sales_by_region
spec:
  prql: |
    from sales_csv
    filter amount > 0
    group {region} (
      aggregate {
        total_amount = sum amount,
        order_count = count this
      }
    )
    sort {-total_amount}
  dependencies:
    - sales_csv
---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: orders
spec:
  type: csv
  path: ./data/orders.csv
---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: customers
spec:
  type: csv
  path: ./data/customers.csv
---
apiVersion: bino.bi/v1alpha1
kind: DataSet
metadata:
  name: customer_orders
spec:
  prql: |
    from orders
    join customers (==customer_id)
    derive full_name = f"{customers.first_name} {customers.last_name}"
    select {
      order_id,
      full_name,
      order_date,
      total
    }
    sort {-order_date}
    take 100
  dependencies:
    - orders
    - customers

For more PRQL syntax and examples, see the PRQL documentation.

To work correctly with visualization components like Charts and Tables, your dataset queries should return rows that conform to the standard schema. While not all fields are required for every component, following this structure ensures proper aggregation, drill-down, and filtering behavior.

The schema supports four parallel sets of scenario columns for comparative analysis (e.g., Actual vs Plan).

ColumnDescription
ac1 ... ac4Actual values (current measurements)
pp1 ... pp4Previous Period values. What a slot compares against (prior month, prior year, …) is declared per dataset with derive/assert, see Deriving previous period
fc1 ... fc4Forecast values (predictions)
pl1 ... pl4Plan/Budget values (targets)

These fields determine how data is hierarchically organized and sorted.

DimensionIndex ColumnDescription
rowGrouprowGroupIndexTop-level row grouping (e.g., "Revenue", "Costs").
categorycategoryIndexPrimary data dimension (e.g., "Product A", "Region North").
subCategorysubCategoryIndexDetail dimension for drill-down (creates "thereOf" rows).
columnGroupcolumnGroupIndexColumn-level grouping for breaking down measures.
columnSubGroupcolumnSubGroupIndexSecond column level underneath columnGroup.

Note: The two columns of a pair depend on each other in both directions. Provide the index (e.g., rowGroupIndex) whenever you provide the dimension string (e.g., rowGroup) to ensure consistent sort ordering — and likewise, an index column without its dimension string is reported by data validation.

ColumnTypeDescription
datestringRequired for TimeCharts. ISO 8601 date (e.g., 2024-01-15) or datetime (e.g., 2024-01-15T08:30:00Z). Used for time-series axes and normalization.
operationstringAggregation sign: '+' (add) or '-' (subtract). Defaults to '+'. Useful for P&L structures where costs subtract from totals.
setnamestringDataset identifier. Used by some charts to distinguish multiple query results.

Here is an example SQL query producing a compliant dataset:

SELECT
  'Revenue' as rowGroup,
  1 as rowGroupIndex,
  product_name as category,
  product_rank as categoryIndex,
  '2024-01-15' as date,
  sum(sales_amount) as ac1,
  sum(budget_amount) as pl1
FROM sales_data
GROUP BY product_name, product_rank

bino registers a few helper functions on the DuckDB engine — two scalar functions and one table macro — so they are available in every DataSet (and DataSource) query without any import or extension.

op reduces a numeric value to its sign as a string, the shape the operation control field (documented above) expects. Note that operation only accepts '+' and '-', so the empty string returned for 0 is reported by data validation — wrap the call (for example nullif(op(x), '')) when zero values can occur.

InputResult
value > 0'+'
value < 0'-'
value = 0'' (empty string)
NULLNULL

It accepts any numeric type (INTEGER, BIGINT, DECIMAL, DOUBLE, …) and returns VARCHAR.

SELECT
  account_name    AS category,
  sum(amount)     AS ac1,
  op(sum(amount)) AS operation   -- '+' for revenue, '-' for costs
FROM ledger
GROUP BY account_name

iop is the inverse of op (+ becomes - and vice versa; 0 stays ''). Use it for inverted measures, where an increase is unfavorable — costs, headcount, error rates, churn — so a rising value reads as a negative signal.

Inputopiop
value > 0'+''-'
value < 0'-''+'
value = 0''''

There is no bino-specific helper for absolute values — DuckDB's built-in abs() already does it (abs(-500)500). Combine it with op to render a signed magnitude as text:

SELECT op(variance) || abs(variance)::VARCHAR AS signed_variance
-- variance = -500  ->  '-500'
-- variance =  500  ->  '+500'

bino_shift(src, source, shift, grain) — previous period as a table macro

Section titled “bino_shift(src, source, shift, grain) — previous period as a table macro”

bino_shift is the table macro behind derive and assert. It returns every row and column of src plus one column named shifted: the value of source on the row with the same identity one shift earlier, or NULL when there is none.

ParameterValue
srcName of a view or table in the session (a DataSource or DataSet name). A macro parameter cannot be a subquery.
sourceThe slot to read from the earlier row: one of ac1ac4, pp1pp4, fc1fc4, pl1pl4
shift'<n> <unit>', e.g. '1 year', '1 month', '2 week', '3 day', '1 quarter'
grainThe period one row stands for: 'day', 'week', 'month', 'quarter' or 'year'

The identity of a row is every column except date and the sixteen slot columns; two rows match when their identities are equal and the period of the earlier one (date truncated to grain) equals the period of the later one minus shift. The extra column is always called shifted — a macro cannot name an output column from a parameter — so alias it to the slot you want:

SELECT * EXCLUDE (shifted), shifted AS pp2
FROM bino_shift('sales_view', 'ac1', '1 year', 'month')

Prefer derive on the DataSet unless you need the shifted value inside a larger query; the declaration also gives you the checks and the caption.

A DataSet can declare what a previous-period slot means instead of supplying it. derive makes bino produce the column, assert checks a column the query already returns. Both take the same shape, keyed by the slot (pp1 to pp4):

apiVersion: bino.bi/v1alpha1
kind: DataSet
metadata:
  name: sales_monthly
spec:
  query: |
    SELECT region, regionIndex, ac1, pl1, date FROM sales   -- no pp column here
  derive:
    pp1: { from: ac1, shift: 1 month, grain: month }   # previous month
    pp2: { from: ac1, shift: 1 year,  grain: month }   # same month last year
  assert:
    pp3: { from: pl1, shift: 1 year,  grain: month }   # pp3 comes from the query; bino checks it

Each declaration has three attributes, all required:

AttributeValueMeaning
fromone of ac1ac4, pp1pp4, fc1fc4, pl1pl4The slot whose value is copied from the earlier row.
shift<n> <unit>, n ≥ 1, unit day, week, month, quarter or yearHow far back the earlier row lies.
grainday, week, month, quarter or yearThe period one row of the dataset stands for. It is declared, never inferred from the dates.

Grain says what a row is, shift says which earlier row to compare against: on a monthly table, shift: 1 month is the previous month and shift: 1 year the same month last year.

The declarations work with query, prql and source alike. A slot may appear in derive or in assert, not both (dataset-derive-conflict), and a derived slot must not be returned by the query. Two declared slots are independent of each other.

For every row bino computes two keys:

  • identity — the values of every column except date and the sixteen slot columns, so all dimension columns with their index twins, operation and setname;
  • perioddate truncated to the grain (date_trunc(grain, date::DATE); a week starts on Monday).

The slot receives the from value of the row that has the same identity and whose period equals this row's period minus shift, provided that period plus shift gives this row's period again. Without such a row the slot is NULL. With from: ac1, shift: 1 month, grain: month and region as the only dimension:

regiondateac1periodperiod − shiftpp1
North2024-01-311002024-012023-12NULL — no such row
North2024-02-291102024-022024-01100
North2024-03-311202024-032024-02110
South2024-03-31802024-032024-02NULL — South has no February row
West2024-02-29702024-022024-01NULL — West is new in February
West2024-03-31NULL2024-032024-0270 — added: West has no March row, its February value is kept visible

What follows from the rule:

  • Periods are compared, not dates, so month ends line up (2024-03-31 finds 2024-02-29) and the day within the period does not matter.
  • Choose a shift that is a whole number of grains. A shift shorter than the grain (1 week on grain month) never matches. A month shift on grain day matches by day of month, and the round-trip condition leaves the 31st empty when the prior month is shorter.
  • Every identity must have at most one row per period; duplicates are an error (see below).
  • Rows before the first shifted period have no prior row: the first month of a table has no pp1, its first year no pp2. If that is every row, the build warns.
  • Index twins and control columns are part of the identity: categoryIndex, rowGroupIndex, operation and setname must be the same in both periods. An index computed per period (a rank by value) changes the identity and breaks the match; compute indexes from a stable attribute.
  • An identity that is new in this period keeps its row with an empty slot.
  • An identity that existed one shift earlier but has no row in this period gets a row: the dimension columns of the earlier row, every measure NULL, the slot filled, and the date the data uses for that period. So a region that stopped selling still shows ac1 = NULL, pp1 = 563 and the drop stays visible. Rows are only added for periods the data covers; nothing is created after the last period. assert never adds rows, it checks the rows the query supplied.

Caption. The engine captions pp1pp4 as PY by default. That is right for a year shift; for any other unit bino emits an Internationalization bundle for the artefact's language that captions the slot PP. It is placed before your own manifests, so a project override of global.pp1 still wins.

After the query runs, bino checks every row — not the validation sample — and reports through the same channel as data validation. Four of the checks are errors that fail bino build regardless of --data-validation, because a declared expectation is not a sampled type check:

CheckMessageOutcome
Derived slot suppliedslot pp2 is already in the query result; use assert: for a supplied sloterror, fails the build even under --data-validation warn
Source column missingslot ac1 (from of pp2) is missing from the query result, column "date" is missing from the query result, slot pp3 is asserted but missing from the query resulterror, fails the build even under --data-validation warn
Duplicate identity in periodduplicate rows for identity region=North in period 2024-03-01 (grain month)error, fails the build even under --data-validation warn
Assert mismatchassert pp3: 4 row(s) differ from pl1 shifted by 1 year (grain month); first at identity region=North, period 2024-03-01error, fails the build even under --data-validation warn
Derived slot emptypp2 derived from ac1 is null on every row — the query window has no prior periodwarning

The assert comparison tolerates relative noise below 1e-9; rows whose shifted value is NULL have nothing to compare and are skipped. --warn-on-query-errors downgrades the four errors like any other query failure, so preview and bino lint --execute-queries show them as warnings.

bino can validate query results against the standard schema at build time. This catches common data issues early:

CheckDescription
Type validationString fields (category, rowGroup, etc.) must be strings
Type validationNumber fields (ac1, categoryIndex, etc.) must be numbers
Enum validationoperation must be "+" or "-"
Date formatdate must be ISO 8601 (YYYY-MM-DD, or YYYY-MM-DDThh:mm:ss with an optional Z or ±hh:mm offset)
Dependent requiredDimension pairs must appear together in both directions: rowGroup requires rowGroupIndex and rowGroupIndex requires rowGroup, and likewise for category, subCategory, columnGroup and columnSubGroup

The derive/assert checks are not part of this sampled validation: they run on every row, and their four error cases fail the build in every validation mode.

Data validation is enabled by default in warn mode. Configure via CLI flags:

# Log warnings and continue (default)
bino build --data-validation=warn

# Treat validation errors as fatal
bino build --data-validation=fail

# Skip validation entirely
bino build --data-validation=off

For lint-only validation (without building), use --execute-queries:

bino lint --execute-queries

For efficiency, bino validates only the first N rows per dataset. Configure via environment variable:

BNR_DATA_VALIDATION_SAMPLE_SIZE=100 bino build

Default sample size is 10 rows.

DataSet documents support metadata.constraints to conditionally include them for specific artefacts, modes, or environments.

Include debug or diagnostic datasets only during development:

apiVersion: bino.bi/v1alpha1
kind: DataSet
metadata:
  name: debug_metrics
  constraints:
    - mode==preview
spec:
  query: |
    SELECT * FROM raw_data LIMIT 100
  dependencies:
    - raw_data

Use different datasets for different output formats:

# Detailed dataset for print (A4/Letter)
apiVersion: bino.bi/v1alpha1
kind: DataSet
metadata:
  name: sales_summary
  constraints:
    - spec.format in [a4,letter]
spec:
  query: |
    SELECT region, product, month, revenue, costs, margin
    FROM sales_fact
    GROUP BY region, product, month

---
# Summarized dataset for screen (XGA)
apiVersion: bino.bi/v1alpha1
kind: DataSet
metadata:
  name: sales_summary
  constraints:
    - field: spec.format
      operator: "=="
      value: xga
spec:
  query: |
    SELECT region, SUM(revenue) as total_revenue
    FROM sales_fact
    GROUP BY region

For the full constraint syntax and operators, see Constraints and Scoped Names.

Instead of defining DataSets as separate documents, you can define them inline directly within components. This is useful for simple, component-specific data transformations.

apiVersion: bino.bi/v1alpha1
kind: ChartStructure
metadata:
  name: sales_chart
spec:
  dataset:
    query: |
      SELECT region, SUM(amount) as total
      FROM sales_csv
      GROUP BY region
    dependencies:
      - sales_csv
  chartTitle: Sales by Region
  chartType: bar
  # ... chart configuration

You can nest inline DataSource definitions within an inline DataSet. Use the @inline(N) syntax to reference them by index:

apiVersion: bino.bi/v1alpha1
kind: ChartStructure
metadata:
  name: quick_chart
spec:
  dataset:
    dependencies:
      - type: csv
        path: ./data/sales.csv
    query: |
      SELECT region, SUM(amount) as total
      FROM @inline(0)
      GROUP BY region
  chartTitle: Sales Overview
  # ... chart configuration

The @inline(0) reference points to the first item in the dependencies array (0-indexed). When you have multiple inline DataSources:

spec:
  dataset:
    dependencies:
      - type: csv
        path: ./data/orders.csv
      - type: csv
        path: ./data/customers.csv
    query: |
      SELECT c.name, COUNT(*) as order_count
      FROM @inline(0) o
      JOIN @inline(1) c ON o.customer_id = c.id
      GROUP BY c.name

You can mix inline DataSource definitions with named references:

spec:
  dataset:
    dependencies:
      - sales_csv # Named reference to existing DataSource
      - type: csv # Inline definition
        path: ./data/rates.csv
    query: |
      SELECT s.*, r.rate
      FROM sales_csv s
      JOIN @inline(1) r ON s.currency = r.code

Note: Named references use their document name directly in SQL, while inline definitions use @inline(N).

For simple cases where you want to use a DataSource directly without transformation, use the source field:

apiVersion: bino.bi/v1alpha1
kind: DataSet
metadata:
  name: sales_passthrough
spec:
  source: sales_csv

This is equivalent to SELECT * FROM sales_csv but more explicit and efficient. The source field is mutually exclusive with query and prql.

You can also use inline DataSource definitions with source:

apiVersion: bino.bi/v1alpha1
kind: Table
metadata:
  name: raw_data_table
spec:
  dataset:
    source:
      type: csv
      path: ./data/report.csv
  # ... table configuration

When bino processes inline definitions:

  1. Materialization: Inline definitions are converted to synthetic documents during YAML loading
  2. Hash-based naming: Generated documents get unique names like _inline_datasource_a1b2c3d4
  3. Deduplication: Identical inline definitions share the same generated document
  4. Labels: Generated documents have bino.bi/generated: "true" and bino.bi/inline: "true" labels

This means inline definitions are fully compatible with caching, the dependency graph, and all other bino features.

User-defined document names must not start with _inline_ as this prefix is reserved for generated inline definitions. The lint rule inline-naming-conflict enforces this. Names starting with bino_ or _bino_ are reserved as well, for the functions and macros bino registers in every session and the views it creates for derive/assert (reserved-name-prefix).

AttributeTypeRequiredDefaultDescription
apiVersionstringyesMust be bino.bi/v1alpha1.
kindstringyesMust be DataSet.
metadata.namestringyesUnique identifier, and the table name the query is registered under. Pattern: ^(@[a-z0-9][a-z0-9_-]*/([a-z0-9][a-z0-9_-]*/)?)?[A-Za-z0-9_]([-A-Za-z0-9_]*[A-Za-z0-9_])?$ — a plain name, optionally prefixed with a registry scope such as @acme/ or @acme/kit/.
metadata.labelsobjectnoKey-value pairs for categorization and constraint matching.
metadata.annotationsobjectnoArbitrary key-value metadata, not used by the system.
metadata.descriptionstringnoFree-form description.
metadata.constraintsarraynoConditional inclusion rules. See Constraints.

At least one of query, prql, or source is required. source is mutually exclusive with query and prql — combining them is rejected by the lint rule dataset-source-exclusive. Setting query and prql together is accepted by the schema, but only prql is executed (the precedence is source before prql before query). derive and assert combine with any of the three; the same slot in both is rejected by the lint rule dataset-derive-conflict.

AttributeTypeRequiredDefaultDescriptionSample
spec.querystring or objectconditionalSQL query executed against DuckDB, inline or from an external file. DataSource and DataSet names are the table names; inline dependencies are addressed as @inline(N). Required if prql and source are not set.query: SELECT * FROM sales
spec.query.$filestringconditionalPath to an external .sql file, resolved relative to the manifest. Used instead of an inline string. See External query files.$file: ./queries/sales_summary.sql
spec.prqlstring or objectconditionalPRQL query, inline or from an external file, compiled to SQL by DuckDB's prql extension. Required if query and source are not set.prql: from sales_csv
spec.prql.$filestringconditionalPath to an external .prql file, resolved relative to the manifest. Used instead of an inline string. See External query files.$file: ./queries/customer_orders.prql
spec.sourcestring or objectconditionalDirect DataSource pass-through, equivalent to SELECT * FROM <name>. A string name or an inline DataSource definition. Required if query and prql are not set. See Direct source pass-through.source: sales_csv
spec.dependenciesarraynoList of DataSource dependencies. Feeds the dependency graph and the cache digest, so a changed input file invalidates the dataset.see below
spec.dependencies[]string or objectnoEither the metadata.name of a DataSource, or an inline DataSource definition referenced in the query as @inline(N) with N the zero-based index.- sales_csv
spec.deriveobjectnoPrevious-period slots bino produces from another slot shifted back in time, keyed by pp1pp4. The query must not return the slot. See Deriving previous period.derive: { pp2: { from: ac1, shift: 1 year, grain: month } }
spec.derive.<pp>.fromstringyesSlot the earlier row is read from: one of ac1ac4, pp1pp4, fc1fc4, pl1pl4.from: ac1
spec.derive.<pp>.shiftstringyesHow far back, <n> <unit> with the unit day, week, month, quarter or year. A year shift keeps the PY caption, any other unit is captioned PP.shift: 1 month
spec.derive.<pp>.grainstringyesThe period one row stands for: day, week, month, quarter or year. Never inferred.grain: month
spec.assertobjectnoPrevious-period slots the query supplies and bino checks, same shape as derive. A mismatch on any row with a prior period fails the build even under --data-validation warn.assert: { pp3: { from: pl1, shift: 1 year, grain: month } }

An inline DataSource — as spec.source or as an item of spec.dependencies — takes the full DataSource spec (type with the values csv, excel, parquet, inline, postgres_query, mysql_query, plus path, content, columns, connection, delimiter, sample, …). Those attributes are documented in the DataSource reference.

spec:
  query:
    $file: ./queries/sales_summary.sql
spec:
  dependencies:
    - sales_csv                    # named reference
    - type: csv                    # inline definition
      path: ./data/extra.csv
  query: |
    SELECT * FROM sales_csv
    UNION ALL
    SELECT * FROM @inline(1)