Skip to content

Google BigQuery

BigQuery is a serverless, cost-effective and multicloud data warehouse designed to help you turn big data into valuable business insights. It a fully-managed, serverless data warehouse that enables scalable analysis over petabytes of data.

In order to use the BigQuery syncer, you must first configure your local environment. Click the setup instructions below.

Setup instructions
  1. Head to Google Developers Console and create a new project (or select the one you already have).
  2. Go to APIs & Services > Enable APIs & services.
    • Click the button for + ENABLE APIS AND SERVICES
    • In the Search bar, find BigQuery API. Click ENABLE.
  3. Create a Service Account.
  4. Create a service account key.
    • In the Cloud Console, click the email address for the service account that you created.
    • Click Keys.
    • Click Add key, then click Create new key.
    • Click Create. A JSON key file is downloaded to your computer.
    • Click Close.

[optional]
, Move the downloaded file to ~/.config/cs_tools/bigquery/credentials.json.
Move the downloaded file to %APPDATA%\cs_tools\bigquery\credentials.json.

BigQuery parameters

Required parameters are in red and Optional parameters are in blue.

  • dataset, the dataset to write new data to
    if tables do not exist in the project.dataset location already, we'll auto-create them
  • credentials_keyfile, the path to your credentials JSON
  • load_strategy, how to write new data into existing tables
    default: APPEND ( allowed: APPEND, TRUNCATE, UPSERT )
How do I use the BigQuery syncer in commands?

cs_tools tools searchable bi-server --syncer bigquery://project_id=cs_tools&dataset=cs_tools_v150&credentials_keyfile=/usr/etc/searchable-ef192fec85db.json

- or -

cs_tools tools searchable bi-server --syncer bigquery://definition.toml

Definition TOML Example

definition.toml 

[configuration]
project_id = "cs_tools"
dataset = "cs_tools_v150"
credentials_keyfile = "/usr/etc/searchable-ef192fec85db.json"
load_strategy = 'truncate'