Skip to content

README.md

Latest commit

 

History

History
489 lines (358 loc) · 23.6 KB

README.md

File metadata and controls

489 lines (358 loc) · 23.6 KB

3-networks-svpc

This repo is part of a multi-part guide that shows how to configure and deploy the example.com reference architecture described in Google Cloud security foundations guide. The following table lists the parts of the guide.

0-bootstrap Bootstraps a Google Cloud organization, creating all the required resources and permissions to start using the Cloud Foundation Toolkit (CFT). This step also configures a CI/CD Pipeline for foundations code in subsequent stages.
1-org Sets up top level shared folders, networking projects, and organization-level logging, and sets baseline security settings through organizational policy.
2-environments Sets up development, nonproduction, and production environments within the Google Cloud organization that you've created.
3-networks-svpc (this file) Sets up shared VPCs with default DNS, NAT (optional), Private Service networking, VPC service controls, on-premises Dedicated Interconnect, and baseline firewall rules for each environment. It also sets up the global DNS hub.
3-networks-hub-and-spoke Sets up shared VPCs with all the default configuration found on step 3-networks-svpc, but here the architecture will be based on the Hub and Spoke network model. It also sets up the global DNS hub
4-projects Sets up a folder structure, projects, and application infrastructure pipeline for applications, which are connected as service projects to the shared VPC created in the previous stage.
5-app-infra Deploy a simple Compute Engine instance in one of the business unit projects using the infra pipeline set up in 4-projects.

For an overview of the architecture and the parts, see the terraform-example-foundation README.

Purpose

The purpose of this step is to:

  • Set up the global DNS Hub.
  • Shared VPCs with default DNS, NAT (optional), Private Service networking, VPC Service Controls (optional), on-premises Dedicated or Partner Interconnect, and baseline firewall rules for each environment.

Prerequisites

  1. 0-bootstrap executed successfully.

  2. 1-org executed successfully.

  3. 2-environments executed successfully.

  4. Obtain the value for the access_context_manager_policy_id variable. It can be obtained by running the following commands. We assume you are at the same level as directory terraform-example-foundation, If you run them from another directory, adjust your paths accordingly.

    export ORGANIZATION_ID=$(terraform -chdir="terraform-example-foundation/0-bootstrap/" output -json common_config | jq '.org_id' --raw-output)
export ACCESS_CONTEXT_MANAGER_ID=$(gcloud access-context-manager policies list --organization ${ORGANIZATION_ID} --format="value(name)")
echo "access_context_manager_policy_id = ${ACCESS_CONTEXT_MANAGER_ID}"

  5. For the manual step described in this document, you need to use the same Terraform version used on the build pipeline. Otherwise, you might experience Terraform state snapshot lock errors.

Troubleshooting

Please refer to troubleshooting if you run into issues during this step.

Usage

Note: If you are using MacOS, replace cp -RT with cp -R in the relevant commands. The -T flag is needed for Linux, but causes problems for MacOS.

Networking Architecture

This step makes use of the Dual Shared VPC architecture, and more details can be found described at the Networking section of the Google cloud security foundations guide. To see the version that makes use the Hub and Spoke mode, check the step 3-networks-hub-and-spoke.

Using Dedicated Interconnect

If you provisioned the prerequisites listed in the Dedicated Interconnect README, follow these steps to enable Dedicated Interconnect to access on-premises resources.

  1. Rename interconnect.tf.example to interconnect.tf in the shared envs folder in 3-networks-svpc/envs/shared
  2. Update the file interconnect.tf with values that are valid for your environment for the interconnects, locations, candidate subnetworks, vlan_tag8021q and peer info.
  3. Rename interconnect.tf.example to interconnect.tf in base_env folder in 3-networks-svpc/modules/base_env.
  4. Update the file interconnect.tf with values that are valid for your environment for the interconnects, locations, candidate subnetworks, vlan_tag8021q and peer info.
  5. Set variable enable_dedicated_interconnect to true
  6. The candidate subnetworks and vlan_tag8021q variables can be set to null to allow the interconnect module to auto generate these values.

Using Partner Interconnect

If you provisioned the prerequisites listed in the Partner Interconnect README follow this steps to enable Partner Interconnect to access on-premises resources.

  1. Rename partner_interconnect.tf.example to partner_interconnect.tf in the shared envs folder in 3-networks-svpc/envs/shared
  2. Rename partner_interconnect.auto.tfvars.example to partner_interconnect.auto.tfvars in the shared envs folder in 3-networks-svpc/envs/shared
  3. Update the file interconnect.tf with values that are valid for your environment for the interconnects, locations, candidate subnetworks, vlan_tag8021q and peer info.
  4. Rename partner_interconnect.tf.example to partner_interconnect.tf in the base-env folder in 3-networks-svpc/modules/base_env .
  5. Update the enable_partner_interconnect to true in each main.tf file in the environment folder in 3-networks-svpc/envs/<environment> .
  6. Update the file partner_interconnect.tf with values that are valid for your environment for the VLAN attachments, locations, and candidate subnetworks.
  7. The candidate subnetworks variable can be set to null to allow the interconnect module to auto generate this value.

OPTIONAL - Using High Availability VPN

If you are not able to use Dedicated or Partner Interconnect, you can also use an HA Cloud VPN to access on-premises resources.

  1. Rename vpn.tf.example to vpn.tf in base-env folder in 3-networks-svpc/modules/base_env.

  2. Create secret for VPN private pre-shared key and grant required roles to Networks terraform service account.

    echo '<YOUR-PRESHARED-KEY-SECRET>' | gcloud secrets create <VPN_PRIVATE_PSK_SECRET_NAME> --project <ENV_SECRETS_PROJECT> --replication-policy=automatic --data-file=-

gcloud secrets add-iam-policy-binding <VPN_PRIVATE_PSK_SECRET_NAME> --member='serviceAccount:<NETWORKS_TERRAFORM_SERVICE_ACCOUNT>' --role='roles/secretmanager.viewer' --project <ENV_SECRETS_PROJECT>
gcloud secrets add-iam-policy-binding <VPN_PRIVATE_PSK_SECRET_NAME> --member='serviceAccount:<NETWORKS_TERRAFORM_SERVICE_ACCOUNT>' --role='roles/secretmanager.secretAccessor' --project <ENV_SECRETS_PROJECT>

  3. Create secret for VPN restricted pre-shared key and grant required roles to Networks terraform service account.

    echo '<YOUR-PRESHARED-KEY-SECRET>' | gcloud secrets create <VPN_RESTRICTED_PSK_SECRET_NAME> --project <ENV_SECRETS_PROJECT> --replication-policy=automatic --data-file=-

gcloud secrets add-iam-policy-binding <VPN_RESTRICTED_PSK_SECRET_NAME> --member='serviceAccount:<NETWORKS_TERRAFORM_SERVICE_ACCOUNT>' --role='roles/secretmanager.viewer' --project <ENV_SECRETS_PROJECT>
gcloud secrets add-iam-policy-binding <VPN_RESTRICTED_PSK_SECRET_NAME> --member='serviceAccount:<NETWORKS_TERRAFORM_SERVICE_ACCOUNT>' --role='roles/secretmanager.secretAccessor' --project <ENV_SECRETS_PROJECT>

  4. In the file vpn.tf, update the values for environment, vpn_psk_secret_name, on_prem_router_ip_address1, on_prem_router_ip_address2 and bgp_peer_asn.

  5. Verify other default values are valid for your environment.

Deploying with Cloud Build

  1. Clone the gcp-networks repo based on the Terraform output from the 0-bootstrap step. Clone the repo at the same level of the terraform-example-foundation folder, the following instructions assume this layout. Run terraform output cloudbuild_project_id in the 0-bootstrap folder to get the Cloud Build Project ID.

    export CLOUD_BUILD_PROJECT_ID=$(terraform -chdir="terraform-example-foundation/0-bootstrap/" output -raw cloudbuild_project_id)
echo ${CLOUD_BUILD_PROJECT_ID}

gcloud source repos clone gcp-networks --project=${CLOUD_BUILD_PROJECT_ID}

  2. Change to the freshly cloned repo, change to the non-main branch and copy contents of foundation to new repo.

    cd gcp-networks/
git checkout -b plan

cp -RT ../terraform-example-foundation/3-networks-svpc/ .
cp ../terraform-example-foundation/build/cloudbuild-tf-* .
cp ../terraform-example-foundation/build/tf-wrapper.sh .
chmod 755 ./tf-wrapper.sh

  3. Rename common.auto.example.tfvars to common.auto.tfvars, rename production.auto.example.tfvars to production.auto.tfvars and rename access_context.auto.example.tfvars to access_context.auto.tfvars.

    mv common.auto.example.tfvars common.auto.tfvars
mv production.auto.example.tfvars production.auto.tfvars
mv access_context.auto.example.tfvars access_context.auto.tfvars

  4. Update common.auto.tfvars file with values from your environment and bootstrap. See any of the envs folder README.md files for additional information on the values in the common.auto.tfvars file. Update production.auto.tfvars file with the target_name_server_addresses. Update access_context.auto.tfvars file with the access_context_manager_policy_id. Use terraform output to get the backend bucket value from 0-bootstrap output.

    export ORGANIZATION_ID=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -json common_config | jq '.org_id' --raw-output)
export ACCESS_CONTEXT_MANAGER_ID=$(gcloud access-context-manager policies list --organization ${ORGANIZATION_ID} --format="value(name)")
echo "access_context_manager_policy_id = ${ACCESS_CONTEXT_MANAGER_ID}"

sed -i'' -e "s/ACCESS_CONTEXT_MANAGER_ID/${ACCESS_CONTEXT_MANAGER_ID}/" ./access_context.auto.tfvars

export backend_bucket=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw gcs_bucket_tfstate)
echo "remote_state_bucket = ${backend_bucket}"

sed -i'' -e "s/REMOTE_STATE_BUCKET/${backend_bucket}/" ./common.auto.tfvars

    Note: Make sure that you update the perimeter_additional_members variable with your user identity in order to be able to view/access resources in the project protected by the VPC Service Controls.

  5. Commit changes

    git add .
git commit -m 'Initialize networks repo'

  6. You must manually plan and apply the shared environment (only once) since the development, nonproduction and production environments depend on it.

  7. To use the validate option of the tf-wrapper.sh script, please follow the instructions to install the terraform-tools component.

  8. Use terraform output to get the Cloud Build project ID and the networks step Terraform Service Account from 0-bootstrap output. An environment variable GOOGLE_IMPERSONATE_SERVICE_ACCOUNT will be set using the Terraform Service Account to enable impersonation.

    export CLOUD_BUILD_PROJECT_ID=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw cloudbuild_project_id)
echo ${CLOUD_BUILD_PROJECT_ID}

export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw networks_step_terraform_service_account_email)
echo ${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT}

  9. Run init and plan and review output for environment shared.

    ./tf-wrapper.sh init shared
./tf-wrapper.sh plan shared

  10. Run validate and check for violations.

    ./tf-wrapper.sh validate shared $(pwd)/../gcp-policies ${CLOUD_BUILD_PROJECT_ID}

  11. Run apply shared.

    ./tf-wrapper.sh apply shared

  12. You must manually plan and apply the production environment since the development, nonproduction and plan environments depend on it.

    git checkout -b production

  13. Run init and plan and review output for environment production.

    ./tf-wrapper.sh init production
./tf-wrapper.sh plan production

  14. Run apply production.

    ./tf-wrapper.sh apply production
    1. Push your production branch since development and nonproduction depends it. Because this is a named environment branch, pushing to this branch triggers both terraform plan and terraform apply. Review the apply output in your Cloud Build project https://console.cloud.google.com/cloud-build/builds;region=DEFAULT_REGION?project=YOUR_CLOUD_BUILD_PROJECT_ID

Note:* The Production envrionment must be the first branch to be pushed as it includes the DNS Hub communication that will be used by other environments.

git push --set-upstream origin production

  1. Push your plan branch to trigger a plan for all environments. Because the plan branch is not a named environment branch, pushing your plan branch triggers terraform plan but not terraform apply. Review the plan output in your Cloud Build project https://console.cloud.google.com/cloud-build/builds;region=DEFAULT_REGION?project=YOUR_CLOUD_BUILD_PROJECT_ID

    git checkout plan
git push --set-upstream origin plan

  2. After plan has been applied, apply development.

  3. Merge changes to development. Because this is a named environment branch, pushing to this branch triggers both terraform plan and terraform apply. Review the apply output in your Cloud Build project https://console.cloud.google.com/cloud-build/builds;region=DEFAULT_REGION?project=YOUR_CLOUD_BUILD_PROJECT_ID

    git checkout -b development
git push origin development

  4. After development has been applied, apply nonproduction.

  5. Merge changes to nonproduction. Because this is a named environment branch, pushing to this branch triggers both terraform plan and terraform apply. Review the apply output in your Cloud Build project https://console.cloud.google.com/cloud-build/builds;region=DEFAULT_REGION?project=YOUR_CLOUD_BUILD_PROJECT_ID

    git checkout -b nonproduction
git push origin nonproduction

  6. Before executing the next steps, unset the GOOGLE_IMPERSONATE_SERVICE_ACCOUNT environment variable.

    unset GOOGLE_IMPERSONATE_SERVICE_ACCOUNT

  7. You can now move to the instructions in the 4-projects step.

Deploying with Jenkins

See 0-bootstrap README-Jenkins.md.

Deploying with GitHub Actions

See 0-bootstrap README-GitHub.md.

Run Terraform locally

  1. The next instructions assume that you are at the same level of the terraform-example-foundation folder. Create and change into gcp-network folder, copy 3-networks-svpc content, the Terraform wrapper script and ensure it can be executed. Also, initialize git so you can manage versions locally.

    mkdir gcp-network
cp -R terraform-example-foundation/3-networks-svpc/* gcp-network
cp terraform-example-foundation/build/tf-wrapper.sh gcp-network/
cp terraform-example-foundation/.gitignore gcp-network/
chmod 755 ./gcp-network/tf-wrapper.sh

  2. Navigate to gcp-network and initialize a local Git repository to manage versions locally. Then, create the environment branches.

    cd gcp-network
git init
git commit -m "initialize empty directory" --allow-empty
git checkout -b shared
git checkout -b production
git checkout -b development
git checkout -b nonproduction

  3. The next instructions assume that you are at the same level of the terraform-example-foundation folder. Change into 3-networks-svpc folder, copy the Terraform wrapper script and ensure it can be executed.

    cd terraform-example-foundation/3-networks-svpc
cp ../build/tf-wrapper.sh .
chmod 755 ./tf-wrapper.sh

  4. Rename common.auto.example.tfvars to common.auto.tfvars, rename production.auto.example.tfvars to production.auto.tfvars and rename access_context.auto.example.tfvars to access_context.auto.tfvars.

    mv common.auto.example.tfvars common.auto.tfvars
mv production.auto.example.tfvars production.auto.tfvars
mv access_context.auto.example.tfvars access_context.auto.tfvars

  5. Update common.auto.tfvars file with values from your environment and bootstrap. See any of the envs folder README.md files for additional information on the values in the common.auto.tfvars file.

  6. Update production.auto.tfvars file with the target_name_server_addresses.

  7. Update access_context.auto.tfvars file with the access_context_manager_policy_id.

  8. Use terraform output to get the backend bucket value from gcp-bootstrap output.

    export ORGANIZATION_ID=$(terraform -chdir="../gcp-bootstrap/" output -json common_config | jq '.org_id' --raw-output)
export ACCESS_CONTEXT_MANAGER_ID=$(gcloud access-context-manager policies list --organization ${ORGANIZATION_ID} --format="value(name)")
echo "access_context_manager_policy_id = ${ACCESS_CONTEXT_MANAGER_ID}"

sed -i'' -e "s/ACCESS_CONTEXT_MANAGER_ID/${ACCESS_CONTEXT_MANAGER_ID}/" ./access_context.auto.tfvars

export backend_bucket=$(terraform -chdir="../gcp-bootstrap/" output -raw gcs_bucket_tfstate)
echo "remote_state_bucket = ${backend_bucket}"

sed -i'' -e "s/REMOTE_STATE_BUCKET/${backend_bucket}/" ./common.auto.tfvars

We will now deploy each of our environments(development/production/nonproduction) using this script. When using Cloud Build or Jenkins as your CI/CD tool each environment corresponds to a branch in the repository for 3-networks-svpc step and only the corresponding environment is applied.

To use the validate option of the tf-wrapper.sh script, please follow the instructions to install the terraform-tools component.

  1. Use terraform output to get the Seed project ID and the organization step Terraform service account from 0-bootstrap output. An environment variable GOOGLE_IMPERSONATE_SERVICE_ACCOUNT will be set using the Terraform Service Account to enable impersonation.

    export SEED_PROJECT_ID=$(terraform -chdir="../gcp-bootstrap/" output -raw seed_project_id)
echo ${SEED_PROJECT_ID}

export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=$(terraform -chdir="../gcp-bootstrap/" output -raw networks_step_terraform_service_account_email)
echo ${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT}

  2. Checkout shared branch. Run init and plan and review output for environment shared.

    git checkout shared
./tf-wrapper.sh init shared
./tf-wrapper.sh plan shared

  3. Run validate and check for violations.

    ./tf-wrapper.sh validate shared $(pwd)/../gcp-policies ${SEED_PROJECT_ID}

  4. Run apply shared.

    ./tf-wrapper.sh apply shared

  5. Checkout shared production. Run init and plan and review output for environment production.

    git checkout production
git merge shared
./tf-wrapper.sh init production
./tf-wrapper.sh plan production

  6. Run validate and check for violations.

    ./tf-wrapper.sh validate production $(pwd)/../gcp-policies ${SEED_PROJECT_ID}

  7. Run apply production.

    ./tf-wrapper.sh apply production
git add .
git commit -m "Initial production commit."
cd ../

  8. Run git commit shared.

    git checkout shared
git add .
git commit -m "Initial shared commit."

  9. Checkout development branch and merge shared into it. Run init and plan and review output for environment production.

    git checkout development
git merge shared
./tf-wrapper.sh init development
./tf-wrapper.sh plan development

  10. Run validate and check for violations.

    ./tf-wrapper.sh validate development $(pwd)/../gcp-policies ${SEED_PROJECT_ID}

  11. Run apply development.

    ./tf-wrapper.sh apply development
git add .
git commit -m "Initial development commit."

  12. Checkout nonproduction and merge development into it. Run init and plan and review output for environment nonproduction.

    git checkout nonproduction
git merge development
./tf-wrapper.sh init nonproduction
./tf-wrapper.sh plan nonproduction

  13. Run validate and check for violations.

    ./tf-wrapper.sh validate nonproduction $(pwd)/../gcp-policies ${SEED_PROJECT_ID}

  14. Run apply nonproduction.

    ./tf-wrapper.sh apply nonproduction
git add .
git commit -m "Initial nonproduction commit."

If you received any errors or made any changes to the Terraform config or any .tfvars, you must re-run ./tf-wrapper.sh plan <env> before run ./tf-wrapper.sh apply <env>.

Before executing the next stages, unset the GOOGLE_IMPERSONATE_SERVICE_ACCOUNT environment variable.

unset GOOGLE_IMPERSONATE_SERVICE_ACCOUNT

(Optional) Enforce VPC Service Controls

Because enabling VPC Service Controls can be a disruptive process, this repo configures VPC Service Controls perimeters in dry run mode by default. This configuration will service traffic that crosses the security perimeter (API requests that originate from inside your perimeter communicating with external resources, or API requests from external resources communicating with resources inside your perimeter) but still allow service traffic normally.

When you are ready to enforce VPC Service Controls, we recommend that you review the guidance at Best practices for enabling VPC Service Controls. After you have added the necessary exceptions and are confident that VPC Service Controls will not disrupt your intended operations, set the variable enforce_vpcsc under the module shared_vpc to true and re-apply this stage. Then re-apply the 4-projects stage, which will inherit the new setting and include those projects inside the enforced perimeter.

When you need to make changes to an existing enforced perimeter, you can test safely by modifying the configuration of the dry run perimeter. This will log traffic denied by the dry run perimeter without impacting whether the enforced perimeter allows or denies traffic.