Skip to content

Commit f3c0a5a

Browse files
authored
DOC-1105 RPCN on GCP BYOVPC (#238)
* draft process * added new page and updated config inline with review * dev review comments * update quickstart * Hyperlint reviews * updates for consistency * review comments * add missing role, reordering, update role bindings to double quotes * add text about placeholders * PM review comments * Added detail on GKE master CIDR range * Add region to addresses command * Update feature flag message * docs review comments * rewording
1 parent 1eaea1c commit f3c0a5a

File tree

7 files changed

+223
-16
lines changed

7 files changed

+223
-16
lines changed

modules/ROOT/nav.adoc

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
*** xref:get-started:cluster-types/byoc/gcp/index.adoc[GCP]
1919
**** xref:get-started:cluster-types/byoc/gcp/create-byoc-cluster-gcp.adoc[]
2020
**** xref:get-started:cluster-types/byoc/gcp/vpc-byo-gcp.adoc[]
21+
**** xref:get-started:cluster-types/byoc/gcp/enable-rpcn-byovpc-gcp.adoc[Enable Redpanda Connect on a BYOVPC Cluster on GCP]
2122
*** xref:get-started:cluster-types/byoc/remote-read-replicas.adoc[]
2223
** xref:get-started:cluster-types/create-dedicated-cloud-cluster.adoc[]
2324
** xref:get-started:create-topic.adoc[Create a Topic]

modules/develop/pages/connect/connect-quickstart.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ include::develop:partial$availability-message.adoc[]
99

1010
== Prerequisites
1111

12-
A Redpanda Cloud account for Serverless, Dedicated, or standard BYOC (not BYOVPC). If you don't already have an account, https://redpanda.com/try-redpanda/cloud-trial[sign up for a free trial^].
12+
A Redpanda Cloud account for Serverless, Dedicated, or standard BYOC. If you don't already have an account, https://redpanda.com/try-redpanda/cloud-trial[sign up for a free trial^].
1313

1414
== Before you start
1515

modules/develop/partials/availability-message.adoc

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,6 @@
22
====
33
44
* Redpanda Connect is available in limited availability (LA) for BYOC and Dedicated clusters. Features in LA are production-ready and are covered by Redpanda Support for early adopters. To unlock Redpanda Connect for your account, contact https://support.redpanda.com/hc/en-us/requests/new[Redpanda Support^].
5-
* Redpanda Connect is not available for BYOVPC clusters.
5+
* Redpanda Connect is in beta for BYOVPC clusters on GCP but not on AWS or Azure.
66
* Redpanda Connect is available in beta for Serverless clusters. Features in beta are not covered by Redpanda Support and should not be used in production environments.
7-
====
8-
9-
7+
====

modules/get-started/pages/cloud-overview.adoc

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -262,7 +262,7 @@ Features in limited availability are production-ready and are covered by Redpand
262262

263263
The following features are currently in limited availability in Redpanda Cloud:
264264

265-
* Redpanda Connect for Dedicated and BYOC (not BYOVPC)
265+
* Redpanda Connect for Dedicated and BYOC (not BYOVPC on AWS or Azure)
266266
* Serverless
267267
* Dedicated and BYOC for Azure
268268
* BYOVPC for GCP
@@ -274,7 +274,7 @@ Features in beta are available for testing and feedback. They are not covered by
274274

275275
The following features are currently in beta in Redpanda Cloud:
276276

277-
* Redpanda Connect for Serverless
277+
* Redpanda Connect for BYOVPC on GCP and Serverless
278278
* Redpanda Terraform provider
279279
* BYOVPC for AWS and Azure
280280
* Remote Read Replicas for AWS and GCP
Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
= Enable Redpanda Connect on an Existing BYOVPC Cluster on GCP
2+
:description: Add Redpanda Connect to your existing BYOVPC cluster.
3+
:page-beta: true
4+
5+
include::shared:partial$feature-flag-rpcn.adoc[]
6+
7+
To enable Redpanda Connect on an existing BYOVPC cluster, you must update your configuration. You can also create xref:get-started:cluster-types/byoc/gcp/vpc-byo-gcp.adoc[a new BYOVPC cluster] with Redpanda Connect already enabled.
8+
9+
Replace all `<placeholders>` with your own values.
10+
11+
. Create two new service accounts with the necessary permissions and roles.
12+
+
13+
.Show commands
14+
[%collapsible]
15+
====
16+
```bash
17+
# Account used to check for and read secrets, which are required to create Redpanda Connect pipelines.
18+
19+
gcloud iam service-accounts create redpanda-connect-api \
20+
--display-name="Redpanda Connect API Service Account"
21+
22+
cat << EOT > redpanda-connect-api.role
23+
{
24+
"name": "redpanda_connect_api_role",
25+
"title": "Redpanda Connect API Role",
26+
"description": "Redpanda Connect API Role",
27+
"includedPermissions": [
28+
"resourcemanager.projects.get",
29+
"secretmanager.secrets.get",
30+
"secretmanager.versions.access"
31+
],
32+
}
33+
EOT
34+
35+
gcloud iam roles create redpanda_connect_api_role --project=<service-project-id> --file redpanda-connect-api.role
36+
37+
gcloud projects add-iam-policy-binding <service-project-id> \
38+
--member="serviceAccount:redpanda-connect-api@<service-project-id>.iam.gserviceaccount.com" \
39+
--role="projects/<service-project-id>/roles/redpanda_connect_api_role"
40+
```
41+
42+
```bash
43+
# Account used to retrieve secrets and create Redpanda Connect pipelines.
44+
45+
gcloud iam service-accounts create redpanda-connect \
46+
--display-name="Redpanda Connect Service Account"
47+
48+
cat << EOT > redpanda-connect.role
49+
{
50+
"name": "redpanda_connect_role",
51+
"title": "Redpanda Connect Role",
52+
"description": "Redpanda Connect Role",
53+
"includedPermissions": [
54+
"resourcemanager.projects.get",
55+
"secretmanager.versions.access"
56+
],
57+
}
58+
EOT
59+
60+
gcloud iam roles create redpanda_connect_role --project=<service-project-id> --file redpanda-connect.role
61+
62+
gcloud projects add-iam-policy-binding <service-project-id> \
63+
--member="serviceAccount:redpanda-connect@<service-project-id>.iam.gserviceaccount.com" \
64+
--role="projects/<service-project-id>/roles/redpanda_connect_role"
65+
```
66+
====
67+
68+
. Bind the service accounts.
69+
+
70+
The account ID of the GCP service account is used to configure service account bindings. This account ID is the local part of the email address for the GCP service account. For example, if the GCP service account is `[email protected]`, then the account ID is `my-gcp-sa`.
71+
+
72+
.Show commands
73+
[%collapsible]
74+
====
75+
```
76+
gcloud iam service-accounts add-iam-policy-binding <redpanda_connect_api-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com \
77+
--role roles/iam.workloadIdentityUser \
78+
--member "serviceAccount:<service-project-id>.svc.id.goog[redpanda-connect/<redpanda_connect_api-gcp-sa-account-id>]"
79+
```
80+
```
81+
gcloud iam service-accounts add-iam-policy-binding <redpanda_connect-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com \
82+
--role roles/iam.workloadIdentityUser \
83+
--member "serviceAccount:<service-project-id>.svc.id.goog[redpanda-connect/<redpanda_connect-gcp-sa-account-id>]"
84+
```
85+
====
86+
87+
. Make a xref:api:ROOT:cloud-controlplane-api.adoc#patch-/v1/clusters/-cluster.id-[`PATCH /v1/clusters/\{cluster-id}`] request to update the cluster configuration.
88+
+
89+
.Show request
90+
[%collapsible]
91+
====
92+
```bash
93+
export CLUSTER_PATCH_BODY=`cat << EOF
94+
{
95+
"customer_managed_resources": {
96+
"gcp": {
97+
"redpanda_connect_api_service_account": {
98+
"email": "<redpanda_connect-api-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com"
99+
},
100+
"redpanda_connect_service_account": {
101+
"email": "<redpanda_connect-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com"
102+
}
103+
}
104+
}
105+
}
106+
EOF`
107+
curl -v -X PATCH \
108+
-H "Content-Type: application/json" \
109+
-H "Authorization: Bearer $AUTH_TOKEN" \
110+
-d "$CLUSTER_PATCH_BODY" $PUBLIC_API_ENDPOINT/v1/clusters/<cluster-id>
111+
```
112+
====
113+
114+
. Check Redpanda Connect is available in the Cloud UI.
115+
.. Log in to https://cloud.redpanda.com[Redpanda Cloud^].
116+
.. Go to the **Connect** page and you should see the **Redpanda Connect** tab alongside **Kafka Connect**.
117+
118+
== Next steps
119+
120+
* Choose xref:develop:connect/components/catalog.adoc[connectors for your use case].
121+
* Learn how to xref:redpanda-connect:guides:getting_started.adoc[configure, test, and run a data pipeline locally].
122+
* Try the xref:develop:connect/connect-quickstart.adoc[Redpanda Connect quickstart].
123+
* Try one of our xref:cookbooks:index.adoc[Redpanda Connect cookbooks].
124+
* Learn how to xref:develop:connect/configuration/secret-management.adoc[add secrets to your pipeline].

modules/get-started/pages/cluster-types/byoc/gcp/vpc-byo-gcp.adoc

Lines changed: 92 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ gcloud compute routers create <router-name> \
5757
--region <region> \
5858
--network <shared-vpc-name>
5959

60-
gcloud compute addresses create <address-name>
60+
gcloud compute addresses create <address-name> --region <region>
6161

6262
gcloud compute routers nats create <nat-config-name> \
6363
--project <host-project-id> \
@@ -95,6 +95,8 @@ gcloud compute firewall-rules create gke-redpanda-cluster-webhooks \
9595
--allow="tcp:9443,tcp:8443,tcp:6443"
9696
```
9797
+
98+
Replace `<gke-master-cidr-range>` with a /28 CIDR. For example: 172.16.0.32/28. For information about the master CIDR, and how to set it using `--master-ipv4-cidr`, see the **gcloud** tab in https://cloud.google.com/kubernetes-engine/docs/how-to/legacy/network-isolation#private_cp[Creating a private cluster with no client access to the public endpoint]
99+
+
98100
. Grant permission to read the VPC and related resources.
99101
+
100102
If the host project and service project are in different projects, it's helpful for the Redpanda team to have read access
@@ -171,7 +173,8 @@ cat << EOT > redpanda-agent.role
171173
"title": "Redpanda Agent Role",
172174
"description": "A role comprising general permissions allowing the agent to manage Redpanda cluster resources.",
173175
"includedPermissions": [
174-
"compute.firewalls.get",
176+
"compute.firewalls.get",
177+
"compute.disks.get",
175178
"compute.globalOperations.get",
176179
"compute.instanceGroupManagers.get",
177180
"compute.instanceGroupManagers.delete",
@@ -286,6 +289,62 @@ gcloud storage buckets add-iam-policy-binding gs://<tiered-storage-bucket-name>
286289
```
287290
====
288291

292+
* Redpanda Connect service accounts
293+
+
294+
.Show commands
295+
[%collapsible]
296+
====
297+
```bash
298+
# Account used to check for and read secrets, which are required to create Redpanda Connect pipelines.
299+
300+
gcloud iam service-accounts create redpanda-connect-api \
301+
--display-name="Redpanda Connect API Service Account"
302+
303+
cat << EOT > redpanda-connect-api.role
304+
{
305+
"name": "redpanda_connect_api_role",
306+
"title": "Redpanda Connect API Role",
307+
"description": "Redpanda Connect API Role",
308+
"includedPermissions": [
309+
"resourcemanager.projects.get",
310+
"secretmanager.secrets.get",
311+
"secretmanager.versions.access"
312+
],
313+
}
314+
EOT
315+
316+
gcloud iam roles create redpanda_connect_api_role --project=<service-project-id> --file redpanda-connect-api.role
317+
318+
gcloud projects add-iam-policy-binding <service-project-id> \
319+
--member="serviceAccount:redpanda-connect-api@<service-project-id>.iam.gserviceaccount.com" \
320+
--role="projects/<service-project-id>/roles/redpanda_connect_api_role"
321+
```
322+
```bash
323+
# Account used to retrieve secrets and create Redpanda Connect pipelines.
324+
325+
gcloud iam service-accounts create redpanda-connect \
326+
--display-name="Redpanda Connect Service Account"
327+
328+
cat << EOT > redpanda-connect.role
329+
{
330+
"name": "redpanda_connect_role",
331+
"title": "Redpanda Connect Role",
332+
"description": "Redpanda Connect Role",
333+
"includedPermissions": [
334+
"resourcemanager.projects.get",
335+
"secretmanager.versions.access"
336+
],
337+
}
338+
EOT
339+
340+
gcloud iam roles create redpanda_connect_role --project=<service-project-id> --file redpanda-connect.role
341+
342+
gcloud projects add-iam-policy-binding <service-project-id> \
343+
--member="serviceAccount:redpanda-connect@<service-project-id>.iam.gserviceaccount.com" \
344+
--role="projects/<service-project-id>/roles/redpanda_connect_role"
345+
```
346+
====
347+
289348
* Redpanda Console service account
290349
+
291350
.Show commands
@@ -320,11 +379,11 @@ gcloud iam roles create redpanda_console_secret_manager_role --project=<service-
320379
321380
gcloud projects add-iam-policy-binding <service-project-id> \
322381
--member="serviceAccount:redpanda-console@<service-project-id>.iam.gserviceaccount.com" \
323-
--role='projects/<service-project-id>/roles/redpanda_console_secret_manager_role'
382+
--role="projects/<service-project-id>/roles/redpanda_console_secret_manager_role"
324383
```
325384
====
326385

327-
* Redpanda Connectors service account
386+
* Kafka Connect connectors service account
328387
+
329388
.Show commands
330389
[%collapsible]
@@ -349,7 +408,7 @@ gcloud iam roles create redpanda_connectors_role --project=<service-project-id>
349408
350409
gcloud projects add-iam-policy-binding <service-project-id> \
351410
--member="serviceAccount:redpanda-connectors@<service-project-id>.iam.gserviceaccount.com" \
352-
--role='projects/<service-project-id>/roles/redpanda_connectors_role'
411+
--role="projects/<service-project-id>/roles/redpanda_connectors_role"
353412
```
354413
====
355414

@@ -456,7 +515,7 @@ gcloud iam roles create redpanda_gke_utility_role --project=<service-project-id>
456515
457516
gcloud projects add-iam-policy-binding <service-project-id> \
458517
--member="serviceAccount:redpanda-gke@<service-project-id>.iam.gserviceaccount.com" \
459-
--role='projects/<service-project-id>/roles/redpanda_gke_utility_role'
518+
--role="projects/<service-project-id>/roles/redpanda_gke_utility_role"
460519
```
461520
====
462521

@@ -489,7 +548,24 @@ gcloud iam service-accounts add-iam-policy-binding <console-gcp-sa-account-id>@<
489548
```
490549
====
491550

492-
* Redpanda Connectors service account
551+
* Redpanda Connect service accounts
552+
+
553+
.Show command
554+
[%collapsible]
555+
====
556+
```
557+
gcloud iam service-accounts add-iam-policy-binding <redpanda_connect_api-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com \
558+
--role roles/iam.workloadIdentityUser \
559+
--member "serviceAccount:<service-project-id>.svc.id.goog[redpanda-connect/<redpanda_connect_api-gcp-sa-account-id>]"
560+
```
561+
```
562+
gcloud iam service-accounts add-iam-policy-binding <redpanda_connect-gcp-sa-account-id>@<service-project-id>.iam.gserviceaccount.com \
563+
--role roles/iam.workloadIdentityUser \
564+
--member "serviceAccount:<service-project-id>.svc.id.goog[redpanda-connect/<redpanda_connect-gcp-sa-account-id>]"
565+
```
566+
====
567+
568+
* Kafka Connect connectors service account
493569
+
494570
.Show command
495571
[%collapsible]
@@ -532,7 +608,7 @@ gcloud iam service-accounts add-iam-policy-binding <gke-service-account-name>@<s
532608

533609
Log in to the https://cloud.redpanda.com[Redpanda Cloud UI^], and follow the steps to xref:get-started:cluster-types/byoc/gcp/create-byoc-cluster-gcp.adoc[create a BYOC cluster], with the following exceptions:
534610

535-
. On the *Network* page, select the *Customer-managed* connection type, and enter the network, service account, and storage bucket information you created.
611+
. On the *Network* page, select the *Customer-managed* connection type, and enter the network, service account, storage bucket information, and GKE master CIDR range you created.
536612

537613
. With customer-managed networks, you must grant yourself (the user deploying the cluster with `rpk`) the following permissions:
538614
+
@@ -593,6 +669,14 @@ rpk cloud byoc gcp apply --redpanda-id='<redpanda-id>' --project-id='<service-pr
593669

594670
include::get-started:partial$no-access.adoc[]
595671

672+
== Delete cluster
673+
674+
You can delete the cluster in the Cloud UI.
675+
676+
. Log in to https://cloud.redpanda.com[Redpanda Cloud^].
677+
. Select your cluster.
678+
. Go to the **Cluster settings** page and click **Delete**, then confirm your deletion.
679+
596680
== Next steps
597681

598682
xref:networking:byoc/gcp/index.adoc[Configure private networking]

modules/shared/partials/feature-flag-rpcn.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,5 +2,5 @@
22
====
33
44
* BYOVPC is an add-on feature that may require an additional purchase. To unlock this feature for your account, contact your Redpanda account team or https://www.redpanda.com/price-estimator[Redpanda Sales^].
5-
* Redpanda Connect is not available for BYOVPC clusters.
5+
* Redpanda Connect is in beta for BYOVPC clusters on GCP.
66
====

0 commit comments

Comments
 (0)