Configure GCP for a Google BigQuery sink connector

To be able to sink data from Apache Kafka® to Google BigQuery via the dedicated Aiven connector, you need to perform the following steps in the GCP console:

Create a new Google service account and generate a JSON service key

Follow the instructions to:

  • create a new Google service account

  • create a JSON service key

The JSON service key will be used in the connector configuration

Verify that BigQuery API is enabled

The BigQuery sink connector uses the API to push the data. To enable them:

  • Navigate to the GCP API & Services dashboard and click on the BigQuery API

  • Verify the BigQuery API is already enabled or follow the steps given to enable it

Create the Google BigQuery dataset

You can either send the Apache Kafka data to an existing Google BigQuery dataset or create a new one using the GCP console by following the instructions in the dedicated page.

Tip

When creating the dataset, specify data location in a region close to where your Aiven for Apache Kafka is running, to minimize latency.

Grant dataset access to the service account

The newly created service account needs to have access to the dataset in order to write data to it. Follow the dedicated instructions to check and modify the dataset permissions. The BigQuery Data Editor is sufficient for the connector to work.