-
Notifications
You must be signed in to change notification settings - Fork 471
Add documentation for Google Pub/Sub support for ClickPipes #6198
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
3d0e72c
Add documentation for Google Pub/Sub support for ClickPipes
tpanetti 15a396a
Fix PR comments
tpanetti 61eaef2
Merge branch 'main' into tp/clickpipes-pubsub
tpanetti 295d7e7
fix formatting and capitalization
tpanetti 35b2d56
drop snapshot support in docs for pub/sub
tpanetti 79733ea
Update docs/integrations/data-ingestion/clickpipes/pubsub/02_auth.md
morsapaes File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
229 changes: 229 additions & 0 deletions
229
docs/integrations/data-ingestion/clickpipes/pubsub/01_overview.md
Large diffs are not rendered by default.
Oops, something went wrong.
141 changes: 141 additions & 0 deletions
141
docs/integrations/data-ingestion/clickpipes/pubsub/02_auth.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,141 @@ | ||
| --- | ||
| slug: /integrations/clickpipes/pubsub/auth | ||
| sidebar_label: 'Pub/Sub IAM permissions' | ||
| title: 'Pub/Sub IAM permissions' | ||
| description: 'This article describes the GCP IAM permissions ClickPipes requires to authenticate with Google Cloud Pub/Sub and consume data from your topics.' | ||
| doc_type: 'guide' | ||
| keywords: ['Google Cloud Pub/Sub', 'GCP IAM', 'service account'] | ||
| integration: | ||
| - support_level: 'core' | ||
| - category: 'clickpipes' | ||
| --- | ||
|
|
||
| :::note | ||
| You can sign up for the Private Preview waitlist [here](https://clickhouse.com/cloud/clickpipes#pubsub-private-preview). | ||
| ::: | ||
|
|
||
| This article describes the GCP IAM permissions ClickPipes requires to authenticate with Google Cloud Pub/Sub and consume data from your topics, and how to set up a service account that grants exactly those permissions. | ||
|
|
||
| ## Prerequisites {#prerequisite} | ||
|
|
||
| To follow this guide, you will need: | ||
| - An active ClickHouse Cloud service | ||
| - A GCP project containing the Pub/Sub topic you want to ingest from | ||
| - IAM permissions in that project to create service accounts and grant roles | ||
|
|
||
| ## Authentication model {#authentication-model} | ||
|
|
||
| ClickPipes for Pub/Sub authenticates with GCP using a [service account JSON key](https://cloud.google.com/iam/docs/keys-create-delete). When you create a pipe, you upload the key file; ClickPipes encrypts it at rest and uses it at runtime to: | ||
|
|
||
| - list and read topics in your project, | ||
| - create and delete the [managed subscription](/integrations/clickpipes/pubsub#managed-subscriptions) ClickPipes uses to consume messages, | ||
| - consume messages from that subscription, and | ||
| - (optionally) read native Pub/Sub schemas from the schema registry. | ||
|
|
||
| There is no workload identity or inline credential paste option — the service account JSON key is the only supported authentication method today. | ||
|
|
||
| ## Required permissions {#required-permissions} | ||
|
|
||
| ClickPipes requires the following IAM permissions on the GCP project that owns the topic. They cover the full pipe lifecycle: discovery (topic listing, validation, sampling), subscription management, steady-state ingestion, and cleanup. | ||
|
|
||
| ### Topic access (discovery and validation) {#topic-access} | ||
|
|
||
| | Permission | Purpose | | ||
| |-------------------------------------|----------------------------------------------------------------------------------| | ||
| | `pubsub.topics.list` | List available topics in the project during discovery | | ||
| | `pubsub.topics.get` | Validate topic existence and retrieve schema settings | | ||
| | `pubsub.topics.attachSubscription` | Required on the **topic** when creating a subscription against it | | ||
|
|
||
| ### Subscription lifecycle (discovery and ingestion) {#subscription-lifecycle} | ||
|
|
||
| | Permission | Purpose | | ||
| |----------------------------------|----------------------------------------------------------------------------------------------------| | ||
| | `pubsub.subscriptions.create` | Create the managed subscription (`clickpipes-{pipeID}`) and ephemeral discovery subscriptions | | ||
| | `pubsub.subscriptions.get` | Health checks (every 60s), follower polling, subscription validation | | ||
| | `pubsub.subscriptions.delete` | Clean up ephemeral discovery subscriptions and delete the managed subscription on pipe deletion | | ||
| | `pubsub.subscriptions.consume` | `Receive()`, `Ack()`, `Nack()`, and seek-to-timestamp operations | | ||
|
|
||
| ### Schema access (optional — only for native Avro/Protobuf topics) {#schema-access} | ||
|
|
||
| | Permission | Purpose | | ||
| |----------------------|------------------------------------------------------------------| | ||
| | `pubsub.schemas.get` | Retrieve native schema definitions from the Pub/Sub schema registry | | ||
|
|
||
| ## Predefined roles {#predefined-roles} | ||
|
|
||
| | Role | Sufficient? | Notes | | ||
| |-----------------------------------------------------------------------------------------------------|-------------|---------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | [`roles/pubsub.editor`](https://cloud.google.com/iam/docs/understanding-roles#pubsub.editor) | Yes | Covers all required permissions. Broadest option. | | ||
| | [`roles/pubsub.subscriber`](https://cloud.google.com/iam/docs/understanding-roles#pubsub.subscriber)| **No** | Missing `topics.list`, `topics.attachSubscription`, `subscriptions.create`, `subscriptions.delete`, and `schemas.get`. | | ||
| | [`roles/pubsub.viewer`](https://cloud.google.com/iam/docs/understanding-roles#pubsub.viewer) | **No** | Read-only — no subscription management or consumption. | | ||
| | Custom role *(recommended)* | Yes | Use the seven core permissions above (plus optional `schemas.get`) for least-privilege access. | | ||
|
|
||
| ## Setup {#setup} | ||
|
|
||
| <VerticalStepper headerLevel="h3"/> | ||
|
|
||
| ### Create a custom role (recommended) {#create-custom-role} | ||
|
|
||
| For least-privilege access, create a custom role with exactly the permissions ClickPipes needs. | ||
|
tpanetti marked this conversation as resolved.
|
||
|
|
||
| You can do this with the `gcloud` CLI: | ||
|
|
||
| ```bash | ||
| gcloud iam roles create clickpipes.pubsub.ingestion \ | ||
| --project=YOUR_PROJECT_ID \ | ||
| --title="ClickPipes Pub/Sub Ingestion" \ | ||
| --description="Permissions required by ClickHouse ClickPipes to ingest from Pub/Sub" \ | ||
| --permissions=pubsub.topics.list,pubsub.topics.get,pubsub.topics.attachSubscription,pubsub.subscriptions.create,pubsub.subscriptions.get,pubsub.subscriptions.delete,pubsub.subscriptions.consume \ | ||
| --stage=GA | ||
| ``` | ||
|
|
||
| Or, in the GCP Console, go to **IAM & Admin → Roles → Create role** and add the permissions listed in [Required permissions](#required-permissions). | ||
|
|
||
| :::note Optional permissions | ||
| Append `pubsub.schemas.get` to the `--permissions` list if you ingest from topics that use native Pub/Sub Avro or Protobuf schemas. Leave it out otherwise to keep the role minimal. | ||
| ::: | ||
|
|
||
| If you prefer to skip the custom role, you can grant `roles/pubsub.editor` instead. | ||
|
|
||
| ### Create a service account {#create-service-account} | ||
|
|
||
| Create a dedicated service account for the ClickPipe: | ||
|
|
||
| ```bash | ||
| gcloud iam service-accounts create clickpipes-pubsub \ | ||
| --project=YOUR_PROJECT_ID \ | ||
| --display-name="ClickPipes Pub/Sub Ingestion" | ||
| ``` | ||
|
|
||
| ### Grant the role to the service account {#grant-role} | ||
|
|
||
| Bind the role you created (or `roles/pubsub.editor`) to the service account at the project level: | ||
|
|
||
| ```bash | ||
| gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \ | ||
| --member="serviceAccount:clickpipes-pubsub@YOUR_PROJECT_ID.iam.gserviceaccount.com" \ | ||
| --role="projects/YOUR_PROJECT_ID/roles/clickpipes.pubsub.ingestion" | ||
| ``` | ||
|
|
||
| ### Create and download a service account key {#create-key} | ||
|
|
||
| Create a JSON key for the service account and download it locally: | ||
|
|
||
| ```bash | ||
| gcloud iam service-accounts keys create clickpipes-pubsub-key.json \ | ||
| --iam-account=clickpipes-pubsub@YOUR_PROJECT_ID.iam.gserviceaccount.com | ||
| ``` | ||
|
|
||
| You will upload this `clickpipes-pubsub-key.json` file in the ClickPipes UI when creating the pipe. | ||
|
|
||
| :::note Treat the key as a secret | ||
| Service account keys grant access to your GCP project. Store the file securely, do not commit it to source control, and rotate it periodically. ClickPipes encrypts the key at rest after upload. | ||
| ::: | ||
|
|
||
| ## Notes {#notes} | ||
|
|
||
| - `pubsub.topics.attachSubscription` is required on the **topic resource**, not the subscription. This is commonly missed when granting only subscription-level permissions. | ||
| - If your topic does not use a native Pub/Sub schema (Avro or Protobuf), the `pubsub.schemas.get` permission is not needed. | ||
| - Managed subscriptions are named `clickpipes-{pipeID}` with a 60s ack deadline, 7-day message retention, and message ordering enabled. | ||
| - Ephemeral discovery subscriptions are named `clickpipes-discovery-{uuid}` with a 10s ack deadline, 10-minute retention, and a 24-hour auto-expiry TTL. | ||
| - ClickPipes treats `PermissionDenied` and `Unauthenticated` errors as non-retryable — if a permission is missing, the pipe fails fast rather than retrying indefinitely. | ||
20 changes: 20 additions & 0 deletions
20
docs/integrations/data-ingestion/clickpipes/pubsub/index.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| --- | ||
|
|
||
|
|
||
| description: 'Landing page with table of contents for the GCP Pub/Sub ClickPipes section' | ||
| slug: /integrations/clickpipes/pubsub-index | ||
| sidebar_position: 1 | ||
| title: 'GCP Pub/Sub ClickPipes' | ||
| doc_type: 'landing-page' | ||
| keywords: ['GCP Pub/Sub ClickPipes', 'Google Cloud Pub/Sub'] | ||
| integration: | ||
| - support_level: 'core' | ||
| - category: 'clickpipes' | ||
| --- | ||
|
|
||
| <!--AUTOGENERATED_START--> | ||
| | Page | Description | | ||
| |-----|-----| | ||
| | [Integrating Google Pub/Sub with ClickHouse Cloud](/integrations/clickpipes/pubsub) | Seamlessly connect your Google Cloud Pub/Sub topics to ClickHouse Cloud. | | ||
| | [Pub/Sub IAM permissions](/integrations/clickpipes/pubsub/auth) | This article describes the GCP IAM permissions ClickPipes requires to authenticate with Google Cloud Pub/Sub and consume data from your topics. | | ||
| <!--AUTOGENERATED_END--> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file added
BIN
+233 KB
static/images/integrations/data-ingestion/clickpipes/cp_step1_pubsub.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+243 KB
static/images/integrations/data-ingestion/clickpipes/cp_step2_pubsub.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+263 KB
static/images/integrations/data-ingestion/clickpipes/cp_step3_pubsub.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.