From Detection to Fix: Using Stackrox and Copacetic for Container Security

This repo contains instructions to complete the AWS CD Georgia workshop.

Environment requirements

To participate in the workshop you need a Kubernetes cluster — either local or remote.

• Local: k0s (https://k0sproject.io), k3s (https://k3s.io), Rancher Desktop (https://rancherdesktop.io), Minikube (https://minikube.sigs.k8s.io), etc.
• Remote: any existing cluster such as EKS, Hetzner, GKE, AKS, etc.
• kubectl and Helm must be installed
• DockerHub or any other Docker registry must be accessible
• Github account
• k9s :) (https://k9scli.io)

NB! In case of Mac with Apple Silicon, you need to run emulation of x86_64 architecture, because Collector serice of StackRox Secured Cluster Services is not available for Apple Silicon. In Rancher Desktop, you can enable emulation by going to the Preferences -> Emulation -> VZ and Enable Rosetta support for x86_64 architecture. Also, for Rancher Desktop, before deploying the collector DaemonSet, you need to ensure that mount propagation will work inside the Rancher Desktop VM. By default, Rancher Desktop mounts / and /proc as private, but the DaemonSet requires them to be shared (rshared). Run these commands inside the Rancher Desktop VM before starting the deployment:

# enter the Rancher Desktop VM
rdctl shell
# inside the VM:
sudo mount --make-rshared /
sudo mount --make-rshared /proc

This step ensures Kubernetes can use mountPropagation: HostToContainer without failing with "/proc is not a shared or slave mount" errors.

Components:

• K8s cluster
• StackRox Central Services
• StackRox Secured Cluster Services
• StackRox Relay
• Demo application

StackRox Central Services Installation

We'll install StackRox Central Services in the cluster using official Helm chart.

$ helm repo add stackrox https://raw.githubusercontent.com/stackrox/helm-charts/main/opensource/
$ helm repo update

Update storageclass in stackrox/central-services/values.yaml for the database PVC and for scannerV4 PVC, depending on the cluster provider and then install the StackRox Central Services.

helm install stackrox-central-services stackrox/stackrox-central-services -f stackrox/central-services/values.yaml -n stackrox --create-namespace --version 400.8.4

For getting access to the StackRox Central Services, you can port-forward to the StackRox Central Services and access it via the browser.

$ kubectl port-forward svc/stackrox-central-services-central 8443:443 -n stackrox

Then, you can access the StackRox Central Services via the browser at https://localhost:8443. The username and password are the same as the ones you saw in the helm output.

Stackrox Setup

Go to the StackRox Central Services and create a new init bundle. Choose name and platform for the init bundle.

Init bundle will be created and downloaded to your local machine. We'll use it later to install the StackRox Secured Cluster Services.

NB! Initial setup also requires approximately 15 minutes to complete vulnerabilities database sync.

StackRox Secured Cluster Services Installation

We'll install StackRox Secured Cluster Services in the cluster using official Helm chart.

$ helm repo add stackrox https://raw.githubusercontent.com/stackrox/helm-charts/main/opensource/
$ helm repo update
$ helm install stackrox-secured-cluster-services stackrox/stackrox-secured-cluster-services -f stackrox/secured-cluster-services/values.yaml  -f stackrox/secured-cluster-services/values-init-bundle.yaml -n stackrox --version 400.8.4

After the installation is complete we can see in the Clusters our new cluster main.

StackRox Relay Installation

We need this service to relay events from StackRox to GitHub. We'll use helm chart from original repository. Before installing the Helm chart, we need to:

1. Create a GitHub Personal Access Token with the following permissions:

• Actions: Read and write
• Workflows: Read and write
• Contents: Read and write
• Metadata: Read

Save the token to the values.yaml file.

configmap:
  GH_TOKEN: "your-token"

2. Create a random secret for StackRox webhook. It is used to authenticate the relay with StackRox.

configmap:
  ACS_WEBHOOK_SECRET: "your-secret"

3. Set GH_OWNER in the values.yaml file.

configmap:
  GH_OWNER: "your-owner"

Then, install the Helm chart:

helm repo add stackrox-relay-service https://forma22-agency.github.io/stackrox-relay-service
helm repo update
helm install stackrox-relay stackrox-relay-service/stackrox-relay-service -f stackrox/relay/values.yaml -n stackrox --version 0.0.12

4. Create a Generic Webhook in StackRox.

• name: stackrox-relay
• Endpoint: http://stackrox-relay.stackrox.svc.cluster.local/webhook (cluster internal address)
• Headers: X-ACS-TOKEN: <value of ACS_WEBHOOK_SECRET>
• Extra fields: not required — the relay adds event_type and client_payload automatically.
• Attach your policy (e.g., "No Critical CVEs") to this notifier in Enforce on Admission mode. When a deployment is blocked, StackRox will send an event to the relay.

No Critical and Important CVEs Policies

We need to create policies to block deployments with critical and important CVEs. You can find the policies in the stackrox/policies directory. Before applying them, you need to update the cluster ID (cluster id you can find in the StackRox Central Services in clusters list).

sed -i 's/3985cdcf-74dd-4929-bb48-302ecf99bf05/<your-cluster-id>/g' stackrox/policies/no-critical-cves.yaml
sed -i 's/3985cdcf-74dd-4929-bb48-302ecf99bf05/<your-cluster-id>/g' stackrox/policies/no-important-cves.yaml
kubectl apply -f stackrox/policies/no-critical-cves.yaml -n stackrox
kubectl apply -f stackrox/policies/no-important-cves.yaml -n stackrox

Demo Application

We need to deploy the demo application with vulnerabilities. Make a fork of the demo application repository and update the image repository in the values.yaml file:

git clone https://github.com/YOUR_GITHUB_USERNAME/aws-cd-georgia-demo-app.git
cd aws-cd-georgia-demo-app/helm-chart
helm install aws-cd-georgia-demo-app . -f values.yaml -n demo --create-namespace

Check that helm release was created. If our policy is working, we can see the violation in the Stackrox dashboard.

Also, relay webhook will be called with the vulnerability information and relay will dispatch event to GitHub Actions workflow. Workflow will patch the image and push it to the registry. Check that the image was patched.

Upgrade the application with patched image

We need to upgrade the application with patched image and check that the violation is resolved.

cd aws-cd-georgia-demo-app/helm-chart
helm upgrade aws-cd-georgia-demo-app . -f values.yaml -n demo

Go to Stackrox Vulnerability Management and check image vulnerability status. Now it should show that the all fixable vulnerabilities are fixed.