Skippr Docs

Appearance

BigQuery ​

Loads data into BigQuery datasets.

Configuration ​

yaml
warehouse:
  kind: bigquery
  project: my-gcp-project
  dataset: raw_data
  location: US
FieldDefaultDescription
project(required)GCP project ID
dataset(required)BigQuery dataset
locationDataset location

CLI ​

bash
skippr connect warehouse bigquery \
  --project my-gcp-project \
  --dataset raw_data \
  --location US

Or run without flags to be prompted interactively.

FlagDescription
--projectGCP project ID
--datasetBigQuery dataset
--locationDataset location (e.g. US, EU)

Config output ​

Running connect warehouse bigquery writes the following to skippr.yaml:

yaml
warehouse:
  kind: bigquery
  project: my-gcp-project
  dataset: raw_data
  location: US

Authentication ​

Authentication uses a GCP service account key file.

VariableDescription
GOOGLE_APPLICATION_CREDENTIALSPath to a GCP service account JSON key file

Setting up a service account ​

  1. In the GCP Console, go to IAM & Admin > Service Accounts.
  2. Create a service account with the BigQuery Data Editor and BigQuery Job User roles.
  3. Create a JSON key and download it.
  4. Set the environment variable:
bash
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"

Required permissions ​

The service account needs:

  • bigquery.datasets.create (for silver/gold dataset creation)
  • bigquery.tables.create, bigquery.tables.updateData (for loading data)
  • bigquery.jobs.create (for running queries)

The BigQuery Data Editor and BigQuery Job User roles cover these.

Troubleshooting ​

SymptomFix
Could not automatically determine credentialsVerify GOOGLE_APPLICATION_CREDENTIALS points to a valid JSON key file
Access Denied: DatasetCheck the service account has the required roles on the project
Not found: DatasetThe dataset will be created automatically; ensure the service account has bigquery.datasets.create

CDC ​

BigQuery supports CDC with exactly-once MERGE DML semantics. Skippr automatically creates _skippr_order_token columns and tombstone tables.

See CDC Destinations -- BigQuery for details.