Skip to content

Bigquery

In Beta

The Syncer protocol is in beta, it has been added to CS Tools in v1.3 on a provisional basis. It may change significantly in future releases and its interface will not be concrete until v2.

Feedback from the community while it's still provisional would be extremely useful; either comment on #25 or create a new issue.

BigQuery is a serverless, cost-effective and multicloud data warehouse designed to help you turn big data into valuable business insights. It a fully-managed, serverless data warehouse that enables scalable analysis over petabytes of data.

In order to use the BigQuery syncer, you must first configure your local environment. The setup instructions below will help you create a Google Cloud Project and enable the BigQuery Storage API to allow CS Tools to interact with your BigQuery environment.

Setup instructions
  1. Head to Google Developers Console and create a new project (or select the one you already have).
  2. Go to APIs & Services > Enable APIs & services.
    • Click the button for + ENABLE APIS AND SERVICES
    • In the Search bar, find BigQuery API. Click ENABLE.
  3. Create a Service Account.
  4. Create a service account key.
    • In the Cloud Console, click the email address for the service account that you created.
    • Click Keys.
    • Click Add key, then click Create new key.
    • Click Create. A JSON key file is downloaded to your computer.
    • Click Close.

[optional]
, Move the downloaded file to ~/.config/cs_tools/bigquery/credentials.json.
Move the downloaded file to %APPDATA%\cs_tools\bigquery\credentials.json.

Want to see the source code?

If you're writing a Custom Syncer, you can check our project code for an example.

cs_tools/sync/bigquery/syncer.py

BigQuery DEFINITION.toml spec

project_id: id of your google cloud project

dataset: name of your bigquery dataset

credentials_file: optional, absolute path to your credentials JSON file
default: <cs_tools-app-directory>/bigquery/credentials.json
you can find the cs_tools app directory by running cs_tools config show

truncate_on_load: optional, either true or false, remove all data in the table prior to a new data load
default: true

How do I use the BigQuery syncer in commands?

cs_tools tools searchable bi-server bigquery:///home/user/syncers/bigquery-definition.toml --compact

cs_tools tools searchable bi-server bigquery://default --compact

cs_tools tools searchable bi-server bigquery://C:\Users\%USERNAME%\Downloads\bigquery-definition.toml --compact

cs_tools tools searchable bi-server bigquery://default --compact

Learn how to register a default for syncers in How-to: Setup a Configuration File.

Full Definition Example

definition.toml 

[configuration]
project_id = 'cs_tools'
dataset = 'thoughtspot'
credentials_file = 'C:\Users\NameyNamerson\Downloads\syncers\<project-name>.json'
truncate_on_load = true