DataSource

DataSource manifests describe how bino loads raw data into the query engine. Each datasource becomes a view named after metadata.name.

Name and SQL identifier

metadata.name for DataSource must match the sqlIdentifier pattern ^([a-z_][a-z0-9_]*|@[a-z0-9][a-z0-9_-]*/[a-z0-9][a-z0-9_-]*)$, which allows two forms:

• A plain SQL identifier: lowercase letters, digits, and underscores only, starting with a letter or underscore (for example sales_daily).
• The @scope/name identity of a DataSource pulled from a bino registry (for example @acme/revenue-table).

Use the plain form directly in DataSet.spec.query. The @scope/name form has to be double-quoted there, for example FROM "@acme/revenue-table".

Minimal definition

The smallest well-formed DataSource sets the required envelope, a metadata.name that becomes the view name, the source type and the input that type needs. Every remaining attribute is a tuning knob whose default or auto-detection fits most reports.

apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: sales_daily
spec:
  type: csv
  path: ./data/sales_daily/*.csv

All attributes are listed in the Attribute Reference below.

The example above shows the csv variant. The alternatives are inline with embedded rows (see Inline datasource), parquet with a path (see Parquet files), excel with a path and an optional sheet, and postgres_query / mysql_query with connection and query (see PostgreSQL query and MySQL query).

Inline datasource

---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: kpi_inline
spec:
  type: inline
  inline:
    content:
      - { label: "Revenue", value: 123.45 }
      - { label: "EBIT", value: 12.34 }

CSV files

---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: sales_daily
spec:
  type: csv
  path: ./data/sales_daily/*.csv

CSV reader options

When the default auto-detection does not produce the right result, add CSV reader options to spec. Any option that is set switches bino from read_csv_auto to read_csv with explicit parameters.

Field	Type	DuckDB parameter	Description
delimiter	string	delim	Field delimiter character (for example ";" or "|").
header	boolean	header	Whether the first row defines column names. Default true.
skipRows	integer	skip	Number of lines to skip before reading data.
thousands	string	thousands	Thousands separator in numeric values (for example ".").
decimalSeparator	string	decimal_separator	Decimal point character (for example ",").
dateFormat	string	dateformat	Date format using DuckDB strftime specifiers (for example "%d/%m/%Y").
columns	object	columns	Map of column name to DuckDB type. Mutually exclusive with columnNames.
columnNames	string[]	names	Explicit column names. Mutually exclusive with columns.

European number format

---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: eu_sales
spec:
  type: csv
  path: ./data/eu_sales.csv
  delimiter: ";"
  thousands: "."
  decimalSeparator: ","
  dateFormat: "%d/%m/%Y"

Headerless CSV with typed columns

---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: sensor_data
spec:
  type: csv
  path: ./data/sensors.csv
  header: false
  columns:
    ts: "TIMESTAMP"
    device_id: "INTEGER"
    reading: "DECIMAL(8,3)"

Custom column names

---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: raw_export
spec:
  type: csv
  path: ./data/export.csv
  header: false
  columnNames: [date, region, amount]

Parquet files

---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: fact_sales_parquet
spec:
  type: parquet
  path: ./warehouse/fact_sales/*.parquet
  ephemeral: false # allow caching between builds

PostgreSQL query

---
apiVersion: bino.bi/v1alpha1
kind: ConnectionSecret
metadata:
  name: postgresCredentials
spec:
  type: postgres
  postgres:
    passwordFromEnv: POSTGRES_PASSWORD
---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: sales_from_postgres
spec:
  type: postgres_query
  connection:
    host: ${DB_HOST:db.example.com}
    port: 5432
    database: analytics
    schema: public
    user: reporting
    secret: postgresCredentials
  query: |
    SELECT *
    FROM fact_sales
    WHERE booking_date >= DATE '2024-01-01';

MySQL query

---
apiVersion: bino.bi/v1alpha1
kind: ConnectionSecret
metadata:
  name: mysqlCredentials
spec:
  type: mysql
  mysql:
    passwordFromEnv: MYSQL_PASSWORD
---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: sales_from_mysql
spec:
  type: mysql_query
  connection:
    host: ${DB_HOST:db.example.com}
    port: 3306
    database: analytics
    user: reporting
    secret: mysqlCredentials
  query: |
    SELECT * FROM fact_sales WHERE year = 2024;

For more on secrets and object storage, see ConnectionSecret.

Sampling

The sample property lets you load only a subset of rows from a datasource using DuckDB’s USING SAMPLE clause. This is ideal for working with large datasets during preview and development, where you need fast iteration without waiting for millions of rows to load.

sample accepts three forms:

Form	Example	DuckDB clause
integer	sample: 1000	USING SAMPLE 1000
string	sample: "10%"	USING SAMPLE 10%
object	sample: { size: 1000, method: reservoir }	USING SAMPLE 1000 (reservoir)

Sampling methods

When using the object form you can specify a method:

• bernoulli - evaluates each row independently with the given probability. Good accuracy, works well for most datasets.
• system - samples entire vector chunks. Faster than bernoulli but higher variance, not recommended for small datasets (< 10k rows).
• reservoir - returns an exact number of rows. Only method that guarantees an exact count.

When no method is specified, DuckDB uses its default (system for percentages, reservoir for row counts).

Basic usage

---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: fact_sales
spec:
  type: parquet
  path: ./warehouse/fact_sales/*.parquet
  sample: 5000 # load only 5000 rows

Percentage sampling

---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: events
spec:
  type: csv
  path: ./data/events_2024.csv
  sample: "10%" # load roughly 10% of rows

Sampling with a specific method

---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: transactions
spec:
  type: postgres_query
  connection:
    host: ${DB_HOST}
    port: 5432
    database: analytics
    user: reporting
    secret: pgCredentials
  query: |
    SELECT * FROM transactions WHERE year = 2024
  sample:
    size: 10000
    method: reservoir # guarantees exactly 10000 rows

Sampling with constraints for development only

Combine sample with constraints so the sampled datasource is used during preview and development, while the full dataset is used for production builds:

# Development: sampled data for fast iteration
---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: fact_sales
  constraints:
    - mode!=build
spec:
  type: parquet
  path: ./warehouse/fact_sales/*.parquet
  sample: "5%"


# Production: full dataset
---
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: fact_sales
  constraints:
    - mode==build
spec:
  type: parquet
  path: ./warehouse/fact_sales/*.parquet

This pattern keeps bino preview fast and responsive while ensuring bino build always produces reports from the complete dataset.

Conditional inclusion with constraints

DataSource documents support metadata.constraints to conditionally include them for specific artefacts, modes, or environments.

Environment-specific data sources

Use different data sources for development vs production:

# Mock data for development
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: sales
  constraints:
    - labels.env==dev
spec:
  type: inline
  content:
    - { region: "Test", amount: 100 }

---
# Production database
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: sales
  constraints:
    - labels.env==prod
spec:
  type: postgres_query
  connection:
    host: prod-db.example.com
    # ...

Multiple environments with in operator

Match multiple environments at once using either format:

# String format
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: staging_data
  constraints:
    - labels.env in [dev,staging,qa]
spec:
  type: postgres_query
  connection:
    host: staging-db.example.com
    port: 5432
    database: analytics
    user: reporting
  query: SELECT * FROM fact_sales

---
# Structured format (IDE-friendly)
apiVersion: bino.bi/v1alpha1
kind: DataSource
metadata:
  name: staging_data
  constraints:
    - field: labels.env
      operator: in
      value: [dev, staging, qa]
spec:
  type: postgres_query
  connection:
    host: staging-db.example.com
    port: 5432
    database: analytics
    user: reporting
  query: SELECT * FROM fact_sales

For the full constraint syntax and operators, see Constraints and Scoped Names.

Inline DataSource definitions

DataSources can also be defined inline within DataSet dependencies arrays, eliminating the need for separate documents. This is useful for simple, single-use data sources.

Example: Inline DataSource in a DataSet

apiVersion: bino.bi/v1alpha1
kind: DataSet
metadata:
  name: sales_summary
spec:
  dependencies:
    - type: csv
      path: ./data/sales.csv
  query: |
    SELECT region, SUM(amount) as total
    FROM @inline(0)
    GROUP BY region

The @inline(0) syntax references the inline DataSource by its position (0-indexed) in the dependencies array.

Supported types for inline definitions

All DataSource types can be used inline:

# CSV
- type: csv
  path: ./data/sales.csv

# Excel
- type: excel
  path: ./data/report.xlsx

# Parquet
- type: parquet
  path: ./warehouse/*.parquet

# Inline data
- type: inline
  content:
    - { region: "US", amount: 100 }
    - { region: "EU", amount: 200 }

Multiple inline DataSources

Reference multiple inline DataSources by their index:

spec:
  dependencies:
    - type: csv
      path: ./data/orders.csv
    - type: csv
      path: ./data/customers.csv
  query: |
    SELECT c.name, o.total
    FROM @inline(0) o
    JOIN @inline(1) c ON o.customer_id = c.id

Mixing inline and named references

You can combine inline definitions with references to standalone DataSource documents:

spec:
  dependencies:
    - existing_datasource # Named reference
    - type: csv # Inline definition
      path: ./data/extra.csv
  query: |
    SELECT * FROM existing_datasource
    UNION ALL
    SELECT * FROM @inline(1)

For more details on inline definitions and the @inline(N) syntax, see DataSet - Inline DataSet definitions.

Attribute Reference

Common Metadata

All document kinds share these metadata fields.

Attribute	Type	Required	Default	Description
apiVersion	string	yes	-	Must be bino.bi/v1alpha1.
kind	string	yes	-	Must be DataSource.
metadata.name	string	yes	-	Unique identifier and name of the created view. Must match the sqlIdentifier pattern: a lowercase SQL identifier such as sales_daily, or a registry identity such as @acme/revenue-table.
metadata.labels	object	no	-	Key-value pairs for categorization and constraint matching.
metadata.annotations	object	no	-	Arbitrary key-value metadata, not used by the system.
metadata.description	string	no	-	Free-form description.
metadata.constraints	array	no	-	Conditional inclusion rules. See Constraints.

Spec Attributes

Attribute	Type	Required	Default	Description	Sample
spec.type	string	yes	-	Data source type. Values: inline, excel, csv, parquet, postgres_query, mysql_query.	type: csv
spec.inline	object	conditional	-	Inline data container holding a single content property. Required when type: inline and spec.content is not set.	see below
spec.inline.content	array or string	yes	-	Inline rows as a YAML array of objects, or a JSON array encoded as a string. Required inside spec.inline.	see below
spec.content	array or string	conditional	-	Inline data as a YAML array of objects or a JSON array encoded as a string. Required when type: inline and spec.inline is not set.	see below
spec.path	string	conditional	-	File path, directory, glob pattern, or URL. Required when type is excel, csv, or parquet.	path: ./data/*.csv
spec.sheet	string	no	first sheet	Worksheet name to read for type: excel. When omitted, the first worksheet is used.	sheet: "Q1 2024"
spec.connection	object	conditional	-	Database connection details (no credentials). Required when type is postgres_query or mysql_query.	see below
spec.connection.host	string	yes	-	Database server hostname or IP address. Required inside spec.connection.	host: db.example.com
spec.connection.port	integer	yes	-	Database server port, 1-65535. Required inside spec.connection.	port: 5432
spec.connection.database	string	yes	-	Name of the database to connect to. Required inside spec.connection.	database: analytics
spec.connection.user	string	yes	-	Username for database authentication. Required inside spec.connection.	user: reporting
spec.connection.schema	string	no	public (PostgreSQL)	Database schema name.	schema: public
spec.connection.secret	string	no	-	Name of a ConnectionSecret that holds the password. Its type must match the datasource type.	secret: postgresCredentials
spec.query	string	conditional	-	SQL query executed against the remote database; its result populates the view. Required when type is postgres_query or mysql_query.	query: SELECT * FROM sales
spec.ephemeral	boolean	no	derived from the source type	Controls caching between builds. When unset, database queries and remote URLs count as changing, inline content and in-project files as stable. Set to false to cache a remote or database source.	ephemeral: false
spec.sample	integer, string, or object	no	-	Row sampling via DuckDB USING SAMPLE. An integer (≥ 1) samples rows, a string samples a percentage ("10%"), an object also picks the method.	sample: 1000
spec.sample.size	integer or string	yes	-	Number of rows (integer ≥ 1) or percentage ("10%") to sample. Required inside the object form.	size: 10000
spec.sample.method	string	no	DuckDB default (system for percentages, reservoir for row counts)	Sampling method. Values: bernoulli, reservoir, system.	method: reservoir
spec.delimiter	string	no	auto	CSV field delimiter (max 4 chars).	delimiter: ";"
spec.header	boolean	no	true	Whether the first row defines column names (CSV only).	header: false
spec.skipRows	integer	no	0	Number of lines to skip before reading data (minimum 0).	skipRows: 2
spec.thousands	string	no	-	Thousands separator in numeric values (max 1 char).	thousands: "."
spec.decimalSeparator	string	no	-	Decimal point character (max 1 char).	decimalSeparator: ","
spec.dateFormat	string	no	-	Date format using DuckDB strftime specifiers.	dateFormat: "%d/%m/%Y"
spec.columnNames	string[]	no	-	Explicit column names, at least one. For type: csv mutually exclusive with columns.	columnNames: [date, region, amount]
spec.columns	object	no	-	Column name to DuckDB type mapping. For type: csv mutually exclusive with columnNames.	see below

Inline content

spec:
  type: inline
  inline:
    content:
      - { label: "Revenue", value: 123.45 }

Connection object

spec:
  connection:
    host: db.example.com
    port: 5432
    database: analytics
    schema: public
    user: reporting
    secret: myConnectionSecret

Column types

spec:
  columns:
    ts: "TIMESTAMP"
    device_id: "INTEGER"
    reading: "DECIMAL(8,3)"

Sample object form

spec:
  sample:
    size: 10000
    method: reservoir # bernoulli | system | reservoir