Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

STACKIT - A Brand By Schwarz Digits

GitHub release


Kubernetes Cluster API Provider STACKIT (CAPSTK)

Kubernetes-native declarative infrastructure for STACKIT.

⚠️ WARNING ⚠️

cluster-api-provider-stackit is still in active development.

Use at your own risk.

What is the Cluster API Provider STACKIT

The Cluster API brings declarative, Kubernetes-style APIs to cluster creation, configuration and management.

The API itself is shared across multiple cloud providers allowing for true STACKIT hybrid deployments of Kubernetes.

Cluster API Provider STACKIT is abbreviated as CAPSTK.

Documentation

Please see our book for in-depth documentation.

Use make serve-book to serve the book locally from this repository.

Install with clusterctl

Create clusterctl.yaml with the published release URL. Replace <version> with the release tag you want to install.

providers:
  - name: stackit
    url: https://github.com/stackitcloud/cluster-api-provider-stackit/releases/download/v<version>/infrastructure-components.yaml
    type: InfrastructureProvider

Install the provider:

clusterctl init --config clusterctl.yaml --infrastructure stackit:v<version>

Launching a Kubernetes cluster on STACKIT

Check out the Quick Start for launching a cluster on STACKIT.

Features

  • Native Kubernetes manifests and API
  • Doesn't use SSH for bootstrapping nodes.
  • Installs only the minimal components to bootstrap a control plane and workers.
  • Supports control planes and worker nodes on STACKIT VM instances.
  • Manages the bootstrapping of security groups and vm instances (networks are excluded for now).
  • Optional Bastion hosts for easier access of control plane or worker nodes
  • Tested Kubernetes Lifecycle (Scaling, Kubernetes Upgrades), see E2E-Tests
  • ClusterClass Topology
  • Support various Linux Distributions (tested with Ubuntu and Flatcar)
  • Release distribution via OCI Images and Manifests
  • Manage the bootstrapping of networks, security groups and vm instances.
    • Deploys Kubernetes control planes into private subnets with a separate bastion server.
  • SKE support

Compatibility with Cluster API and Kubernetes Versions

This provider's versions are compatible with the following versions of Cluster API and support all Kubernetes versions that is supported by its compatible Cluster API version:

Cluster API v1alpha4 (v0.4)Cluster API v1beta1Cluster API v1beta2
CAPSTK v1alpha1 (main)xx✓

This provider implements the v1beta2 contract, as declared in metadata.yaml, and is built against sigs.k8s.io/cluster-api v1.13.2.

(See Kubernetes support matrix of Cluster API versions).


Getting involved and contributing

Interested in contributing to cluster-api-provider-stackit? We welcome your ideas, contributions, and help. Feel free to reach out to the maintainers to learn how to get involved.

You do not need official write permissions to make an impact. We encourage active community members to take ownership, drive improvements, and help move the project forward.

We are also happy to welcome new maintainers over time. Get involved and show us what you can do!

Quick start

This guide creates a Kind management cluster and installs the published STACKIT infrastructure provider. You do not need to clone this repository or build an image.

Prerequisites

Install kind, kubectl, clusterctl, curl, and base64. You also need a STACKIT project, a service-account JSON key, an existing network, image, and machine type. See IAM permissions for the required service-account permissions.

Set the release version to install:

export CAPSTK_VERSION=v0.1.0-alpha.5

Create clusterctl.yaml for that release:

cat > clusterctl.yaml <<EOF
providers:
  - name: stackit
    url: https://github.com/stackitcloud/cluster-api-provider-stackit/releases/download/${CAPSTK_VERSION}/infrastructure-components.yaml
    type: InfrastructureProvider
EOF

Create a management cluster

Enterprise proxies like Zscaler

If your network uses a TLS-intercepting proxy such as Zscaler, the Kind node must trust the proxy's root certificate. The host's certificate store does not automatically apply inside the Kind node. Create a local kind-config.yaml that mounts the host CA bundle into the node:

cat > kind-config.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraMounts:
      - hostPath: /etc/ssl/certs/ca-certificates.crt
        containerPath: /etc/ssl/certs/ca-certificates.crt
        readOnly: true
EOF
kind create cluster --name capi-stackit --config kind-config.yaml

Without such a proxy, create the cluster without the configuration file:

kind create cluster --name capi-stackit

Select the management-cluster context and install the provider:

kubectl config use-context kind-capi-stackit

clusterctl init \
  --config clusterctl.yaml \
  --infrastructure stackit:"${CAPSTK_VERSION}"

Wait for the provider controller:

kubectl rollout status \
  --namespace cluster-api-provider-stackit-system \
  deployment/cluster-api-provider-stackit-controller-manager

Set credentials and cluster settings

Save the service-account JSON key anywhere on your machine, then set its path and the IDs for the STACKIT resources the workload cluster will use:

export STACKIT_PROJECT_ID=<project-uuid>
export STACKIT_REGION=eu01
export STACKIT_NETWORK_ID=<network-uuid>
export STACKIT_IMAGE_ID=<image-uuid>
export STACKIT_MACHINE_TYPE=c2i.4
export STACKIT_SERVICE_ACCOUNT_JSON_FILE=/path/to/service-account.json
export STACKIT_SERVICE_ACCOUNT_JSON_B64="$(base64 < "${STACKIT_SERVICE_ACCOUNT_JSON_FILE}" | tr -d '\n')"

kubectl create secret generic stackit-credentials \
  --namespace default \
  --from-literal=project-id="${STACKIT_PROJECT_ID}" \
  --from-file=serviceaccount.json="${STACKIT_SERVICE_ACCOUNT_JSON_FILE}"

You can retrieve a valid image ID using the STACKIT CLI:

stackit image list --project-id "${STACKIT_PROJECT_ID}" --output-format json --label-selector linux,prod | jq -r '.[] | select(.name == "Ubuntu 24.04") | .id'

Create a workload cluster

You can use the cluster template directly from the repository's templates directory (templates/cluster-template.yaml) if you cloned the repository, or download the template that matches the provider release:

curl --fail --location --remote-name \
  "https://github.com/stackitcloud/cluster-api-provider-stackit/releases/download/${CAPSTK_VERSION}/cluster-template.yaml"

Set the cluster values and render the template (use --from templates/cluster-template.yaml if working from a repository clone):

export CLUSTER_NAME=stackit-workload
export NAMESPACE=default
export KUBERNETES_VERSION=v1.35.8
export KUBERNETES_APT_REPOSITORY_MINOR=v1.35
export CONTROL_PLANE_MACHINE_COUNT=1
export WORKER_MACHINE_COUNT=1
export STACKIT_CREDENTIALS_SECRET_NAME=stackit-credentials
export STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGE=ghcr.io/stackitcloud/cloud-provider-stackit/cloud-controller-manager:v1.35.7

clusterctl generate cluster "${CLUSTER_NAME}" \
  --from cluster-template.yaml \
  --target-namespace "${NAMESPACE}" \
  > cluster.yaml
kubectl apply -f cluster.yaml

Watch the provider create the cluster:

kubectl get cluster,machine,stackitcluster,stackitmachine --namespace "${NAMESPACE}"

When the workload API is available, retrieve the workload kubeconfig to install a CNI:

clusterctl get kubeconfig "${CLUSTER_NAME}" \
  --namespace "${NAMESPACE}" \
  > "${CLUSTER_NAME}".kubeconfig

Workload nodes remain in NotReady status until a CNI is installed. We provide a preconfigured Cilium setup at templates/addons/cilium-values.yaml in the templates addons directory, and it is also published for each release:

curl --fail --location --remote-name \
  "https://github.com/stackitcloud/cluster-api-provider-stackit/releases/download/${CAPSTK_VERSION}/cilium-values.yaml"

Install Cilium using the Cilium CLI or Helm with these values (or use templates/addons/cilium-values.yaml directly from a local checkout):

cilium install \
  --kubeconfig "${CLUSTER_NAME}.kubeconfig" \
  --values cilium-values.yaml

If you are working from a local checkout, you can also use the development target in the Makefile backed by hack/install-workload-cni.sh:

make install-workload-cni \
  WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig"

See Workload CNI and Workload Addons for more details on CNI options and verification.

Cleanup

To fully delete the workload cluster and its cloud infrastructure:

kubectl delete -f cluster.yaml

Deletion should take at most 5 minutes. Verify that all resources are gone:

kubectl get cluster,machine,stackitcluster,stackitmachine --namespace "${NAMESPACE}"

Troubleshooting stuck deletion

If resources persist after several minutes:

  1. Check the status and conditions of persisting resources:

    kubectl describe cluster,machine,stackitcluster,stackitmachine --namespace "${NAMESPACE}"
    
  2. Inspect the controller logs for errors:

    # Core Cluster API controller
    kubectl logs -n capi-system deployment/capi-controller-manager -c manager --tail=100
    
    # STACKIT provider controller
    kubectl logs -n cluster-api-provider-stackit-system deployment/cluster-api-provider-stackit-controller-manager --tail=100
    
  3. If a StackitMachine cannot finish deletion and blocks its parent resources, you can remove its finalizer:

    kubectl patch stackitmachine <stackitmachine-name> -n "${NAMESPACE}" \
      --type=merge -p '{"metadata":{"finalizers":null}}'
    

    Removing the finalizer unblocks Kubernetes deletion for that resource and its parents. You may then have to manually delete the actual VM in the STACKIT portal or through the STACKIT API.

Use the development guide when you want to build and run the provider from a local checkout.

Usage

Use these chapters to render, create, operate, and delete STACKIT workload clusters.

Cluster Templates

CAPSTK provides clusterctl templates under templates/ for various workload cluster configurations:

  • templates/cluster-template.yaml: Default template using a pre-existing (Bring-Your-Own / BYO) STACKIT network.
  • templates/cluster-template-managed-network.yaml: Managed network template where CAPSTK automatically creates and manages a dedicated /24 STACKIT network.
  • templates/cluster-template-bastion.yaml: Enables an optional provider-managed SSH bastion host.
  • templates/cluster-template-flatcar-workers.yaml: Mixed OS cluster with Flatcar Linux worker nodes.
  • templates/cluster-template-topology.yaml: ClusterClass topology-based cluster.

Resource Manifest Breakdown

The classic templates (cluster-template.yaml and cluster-template-managed-network.yaml) render:

  • Cluster
  • StackitCluster
  • KubeadmControlPlane
  • control-plane StackitMachineTemplate
  • MachineDeployment
  • worker StackitMachineTemplate
  • worker KubeadmConfigTemplate
  • MachineHealthCheck for control-plane and worker Machines
  • ClusterResourceSet for cloud-provider-stackit

Deploying with a Bring-Your-Own (BYO) Network

In the BYO network flow, STACKIT_NETWORK_ID is known upfront. The template embeds both the ClusterResourceSet and the resource-set Secret with the pre-rendered cloud-provider-stackit.yaml.

Render a cluster:

clusterctl generate cluster "${CLUSTER_NAME}" \
  --from templates/cluster-template.yaml \
  --kubernetes-version "${KUBERNETES_VERSION}" \
  --control-plane-machine-count "${CONTROL_PLANE_MACHINE_COUNT}" \
  --worker-machine-count "${WORKER_MACHINE_COUNT}" \
  > "${CLUSTER_NAME}.yaml"

Apply the rendered manifest:

kubectl apply -f "${CLUSTER_NAME}.yaml"

Deploying with a Managed Network

When using templates/cluster-template-managed-network.yaml, STACKIT_NETWORK_ID is omitted. CAPSTK automatically provisions a dedicated /24 network during StackitCluster reconciliation and publishes its ID to StackitCluster.status.network.id.

Render and apply the cluster:

clusterctl generate cluster "${CLUSTER_NAME}" \
  --from templates/cluster-template-managed-network.yaml \
  --kubernetes-version "${KUBERNETES_VERSION}" \
  --control-plane-machine-count "${CONTROL_PLANE_MACHINE_COUNT}" \
  --worker-machine-count "${WORKER_MACHINE_COUNT}" \
  | kubectl apply -f -

Because the network ID is generated at runtime, the template creates the ClusterResourceSet but omits the Secret. Once CAPSTK provisions the network, you retrieve the network ID and supply the cloud-provider-stackit Secret or apply the addon directly. See Workload Addons for the full procedure.

External Cloud Provider Wiring

The templates configure kubeadm for an external cloud provider by setting cloud-provider=external on kubelet and controller-manager.

The management cluster must have the ClusterResourceSet feature enabled before applying templates that use it. For local validation, hack/clusterctl-local.yaml sets CLUSTER_RESOURCE_SET=true.

Set STACKIT_SERVICE_ACCOUNT_JSON_B64 to a single-line base64 encoding of the STACKIT service account JSON. Set STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGE to a cloud-provider-stackit image whose minor version matches KUBERNETES_VERSION; supported workload cluster minors are v1.33.x through v1.36.x. Use hack/validate-stackit-versions.sh before rendering to catch unsupported or mismatched versions.

The templates install or prepare the cloud controller manager, not a CNI. After the workload API is reachable, install a CNI that matches the configured pod/service CIDRs and network policy expectations before expecting Nodes to become Ready. See Workload CNI and Workload Addons.

Machine Health Checks and Remediation

The MachineHealthChecks remediate Machines whose Node does not register within 10 minutes or reports Ready=False or Ready=Unknown for 10 minutes. The MachineDeployment limits worker remediation to 2 Machines at a time via spec.remediation.maxInFlight.

Worker remediation only runs while at most 40% of the workers are unhealthy (triggerIf.unhealthyLessThanOrEqualTo, rounded down). A MachineDeployment with fewer than 3 workers is therefore never remediated; use WORKER_MACHINE_COUNT ≥ 3 for automatic replacement. A network outage longer than 10 minutes can still replace healthy workers once the first of them recovers and the unhealthy share drops below the limit.

Control-plane and worker Machines use nodeDrainTimeoutSeconds: 900, so a Node whose Pods cannot terminate does not block deletion forever. This applies to every Machine deletion, including upgrades and scale-down: Pods still protected by a PodDisruptionBudget after 15 minutes are not waited for.

ClusterClass

ClusterClass is Cluster API's reusable cluster blueprint mechanism. Instead of rendering every infrastructure, control plane, bootstrap, and worker resource directly for each workload cluster, you define a ClusterClass once and create workload clusters that reference it through Cluster.spec.topology.

For this provider, ClusterClass support is provided by:

  • templates/clusterclass.yaml
  • templates/cluster-template-topology.yaml

templates/clusterclass.yaml contains the reusable class and the STACKIT infrastructure templates. templates/cluster-template-topology.yaml contains a small topology Cluster that selects the class and passes values as topology variables.

Why use ClusterClass

Use ClusterClass when you want CAPI to own the generated cluster topology. CAPI creates and reconciles the underlying StackitCluster, KubeadmControlPlane, MachineDeployment, bootstrap templates, and StackitMachine infrastructure objects from the class.

This gives you:

  • one reusable cluster shape for multiple workload clusters
  • smaller workload cluster manifests
  • Kubernetes version, replica count, and infrastructure settings expressed as topology variables
  • standard CAPI topology behavior for scaling and upgrades
  • generated infrastructure resources with owner references, labels, and annotations managed by CAPI core

You do not need ClusterClass for a simple explicit cluster manifest. The classic template in Classic Cluster Template is still valid. ClusterClass is the preferred path when you want repeatable topology management and fleet-style cluster creation.

What the STACKIT ClusterClass configures

The current ClusterClass wires topology variables into STACKIT templates for:

  • Kubernetes version
  • control-plane replica count
  • worker replica count
  • machine type
  • image ID
  • region
  • network ID
  • credentials Secret name
  • additional STACKIT cloud labels
  • development fallback preKubeadmCommands

The ClusterClass uses these provider templates:

  • StackitClusterTemplate for the generated StackitCluster
  • one StackitMachineTemplate for control-plane machines
  • one StackitMachineTemplate for worker machines

The templates also include spec.template.metadata.labels and spec.template.metadata.annotations. CAPI topology copies this metadata onto the generated StackitCluster and StackitMachine objects. This is useful for policy, ownership, automation, and observability labels that should be present on generated infrastructure objects.

The class also defines healthCheck and deletion for the control plane and the default-worker class with the same checks, remediation limits and drain timeouts as the classic template's MachineHealthChecks (see Classic Cluster Template). Disable the health checks per cluster with spec.topology.controlPlane.healthCheck.enabled: false or the equivalent field on a MachineDeployment topology.

Prerequisites

The management cluster must run CAPI core and kubeadm-control-plane with ClusterTopology=true. Otherwise the admission webhooks reject ClusterClass, KubeadmControlPlaneTemplate, and Cluster.spec.topology.

For local clusterctl usage, hack/clusterctl-local.yaml sets CLUSTER_TOPOLOGY: "true".

The topology cluster template also includes a cloud-provider-stackit ClusterResourceSet addon and a workload-cluster Secret for the cloud controller manager. The management cluster must have the ClusterResourceSet feature enabled as well. For local validation, hack/clusterctl-local.yaml sets CLUSTER_RESOURCE_SET: "true".

Before rendering, prepare the same STACKIT variables used by the classic template:

export CLUSTER_NAME=stackit-workload
export NAMESPACE=default
export KUBERNETES_VERSION=v1.35.8
export KUBERNETES_APT_REPOSITORY_MINOR=v1.35
export CONTROL_PLANE_MACHINE_COUNT=1
export WORKER_MACHINE_COUNT=1

export STACKIT_PROJECT_ID=<project-uuid>
export STACKIT_REGION=eu01
export STACKIT_NETWORK_ID=<network-uuid>
export STACKIT_IMAGE_ID=<image-uuid>
export STACKIT_MACHINE_TYPE=c2i.2
export STACKIT_CREDENTIALS_SECRET_NAME=stackit-credentials

export STACKIT_SERVICE_ACCOUNT_JSON_FILE=./.stackit/cluster-api-provider-stackit-serviceaccount.json
export STACKIT_SERVICE_ACCOUNT_JSON_B64="$(base64 < "${STACKIT_SERVICE_ACCOUNT_JSON_FILE}" | tr -d '\n')"
export STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGE=ghcr.io/stackitcloud/cloud-provider-stackit/cloud-controller-manager:v1.35.7

STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGE must use a minor version matching KUBERNETES_VERSION.

Apply the ClusterClass

Apply the reusable class and its templates into the namespace that should own the class:

kubectl apply -n "${NAMESPACE}" -f templates/clusterclass.yaml

If the class lives in a different namespace than the workload cluster, set:

export CLUSTER_CLASS_NAMESPACE=<clusterclass-namespace>

Create a topology cluster

Render and apply a topology Cluster:

clusterctl generate cluster "${CLUSTER_NAME}" \
  --from templates/cluster-template-topology.yaml \
  --target-namespace "${NAMESPACE}" \
  --kubernetes-version "${KUBERNETES_VERSION}" \
  --control-plane-machine-count "${CONTROL_PLANE_MACHINE_COUNT}" \
  --worker-machine-count "${WORKER_MACHINE_COUNT}" \
  > "${CLUSTER_NAME}-topology.yaml"

kubectl apply -f "${CLUSTER_NAME}-topology.yaml"

CAPI topology then creates the generated infrastructure and control plane objects. Inspect them with:

kubectl get cluster,machine,stackitcluster,stackitmachine -n "${NAMESPACE}"

Check propagated template metadata:

kubectl get stackitcluster -n "${NAMESPACE}" \
  -l "cluster.x-k8s.io/cluster-name=${CLUSTER_NAME}" \
  -o go-template='{{range .items}}{{index .metadata.labels "cluster-api-provider-stackit/template-metadata"}}{{"\n"}}{{end}}'

kubectl get stackitmachine -n "${NAMESPACE}" \
  -l "cluster.x-k8s.io/cluster-name=${CLUSTER_NAME}" \
  -o go-template='{{range .items}}{{.metadata.name}} {{index .metadata.labels "cluster-api-provider-stackit/template-metadata"}}{{"\n"}}{{end}}'

Workload cluster addons

The topology cluster template installs the cloud controller manager through a ClusterResourceSet. It does not install a CNI.

After the workload API is reachable, install a CNI that matches the configured pod/service CIDRs and network policy expectations before expecting Nodes to become Ready. For a reproducible development path, use the helper documented in Workload CNI.

Delete a topology cluster

Delete the topology Cluster; CAPI deletes the generated objects and this provider deletes the STACKIT resources:

kubectl delete cluster "${CLUSTER_NAME}" -n "${NAMESPACE}"

After deletion, verify generated resources are gone:

kubectl get machine,stackitcluster,stackitmachine \
  -n "${NAMESPACE}" \
  -l "cluster.x-k8s.io/cluster-name=${CLUSTER_NAME}"

E2E validation

Run the topology workload e2e path independently only when billable STACKIT validation is intended:

env STACKIT_E2E_TOPOLOGY_WORKLOAD=true \
  KUBERNETES_VERSION=v1.35.8 \
  STACKIT_E2E_CNI=cilium \
  go test -timeout=90m -tags=e2e ./test/e2e -v -ginkgo.v \
  --ginkgo.focus='topology.*workload' --ginkgo.timeout=90m

Equivalent make target:

make test-e2e-workload-topology

The topology e2e creates a real STACKIT workload cluster, waits for Ready Machines and Nodes, verifies providerID alignment, verifies template metadata propagation, deletes the Cluster, and checks that tagged STACKIT servers and API server load balancers are cleaned up.

Workload Addons

This provider supports a tested addon flow for unmanaged STACKIT workload clusters:

  • cloud-provider-stackit (the Cloud Controller Manager or CCM) runs in the workload cluster to manage node lifecycles, provider IDs, and STACKIT Service Load Balancers.
  • A CNI (such as Cilium or Calico) is installed after the workload API server is reachable.
  • The default tested CNI path uses Cilium with preconfigured values in templates/addons/cilium-values.yaml.

This split is intentional:

  1. The cloud provider must run early so Kubernetes Nodes get STACKIT provider IDs and the external cloud-provider taint (node.cloudprovider.kubernetes.io/uninitialized) is cleared.
  2. The CNI is a cluster operator choice because routing, network policy, MTU, IPAM, and upgrade strategy are environment-specific.

Cloud Controller Manager (cloud-provider-stackit)

The STACKIT Cloud Controller Manager runs inside the workload cluster in the kube-system namespace. Its manifest is defined in templates/addons/cloud-provider-stackit.yaml and sets up:

  • ServiceAccount, RBAC roles, and ClusterRoleBindings for stackit-cloud-controller-manager
  • stackit-cloud-config ConfigMap containing cloud.yaml with the STACKIT project ID, region, and load balancer network ID
  • stackit-cloud-secret Secret containing the service account JSON key (sa_key.json)
  • Deployment running stackit-cloud-controller-manager with controllers:
    • cloud-node-controller: initializes Node objects and sets spec.providerID to stackit://<server-id>
    • cloud-node-lifecycle-controller: cleans up Node objects when servers are deleted in STACKIT
    • service-lb-controller: provisions and configures STACKIT Network Load Balancers for Kubernetes Service objects with type: LoadBalancer

Configuration Variables

The manifest templates/addons/cloud-provider-stackit.yaml expects these environment variables:

VariableDescription
STACKIT_PROJECT_IDTarget STACKIT project UUID
STACKIT_REGIONTarget STACKIT region (e.g. eu01)
STACKIT_NETWORK_IDSTACKIT network UUID for Service load balancers
STACKIT_SERVICE_ACCOUNT_JSON_B64Base64-encoded STACKIT service account JSON key
STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGECCM image matching the workload Kubernetes minor version (e.g. ghcr.io/stackitcloud/cloud-provider-stackit/cloud-controller-manager:v1.35.7)
CLUSTER_NAMEName of the workload cluster

Set your credentials and CCM image:

export STACKIT_SERVICE_ACCOUNT_JSON_FILE=./.stackit/cluster-api-provider-stackit-serviceaccount.json
export STACKIT_SERVICE_ACCOUNT_JSON_B64="$(base64 < "${STACKIT_SERVICE_ACCOUNT_JSON_FILE}" | tr -d '\n')"
export STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGE="ghcr.io/stackitcloud/cloud-provider-stackit/cloud-controller-manager:v1.35.7"

Important: The cloud-provider-stackit image minor version must match the Kubernetes minor version of the workload cluster (e.g. v1.35.x CCM for v1.35.x Kubernetes). Use hack/validate-stackit-versions.sh before deployment to verify compatibility.


Deploying CCM with BYO Networks

When using templates/cluster-template.yaml or templates/cluster-template-topology.yaml, you provide STACKIT_NETWORK_ID up front. The cluster template generates:

  1. A Cluster API ClusterResourceSet named ${CLUSTER_NAME}-cloud-provider-stackit
  2. A companion Secret containing the pre-rendered cloud-provider-stackit.yaml

As soon as the workload cluster API server is reachable, Cluster API automatically applies the CCM manifest into the workload cluster. No manual intervention is needed.


Managed Networks and cloud-provider-stackit

When deploying a cluster with a managed network using templates/cluster-template-managed-network.yaml, spec.network.id is omitted. CAPSTK creates a dedicated /24 network in STACKIT during cluster reconciliation.

Because the network ID does not exist when clusterctl generate is executed, the template creates the ClusterResourceSet but does not include the Secret. The STACKIT CCM requires this network ID in stackit-cloud-config to manage Service load balancers.

Step 1: Retrieve the Provisioned Network ID

Wait for CAPSTK to create the network and populate StackitCluster.status.network.id:

export STACKIT_NETWORK_ID=$(kubectl get stackitcluster "${CLUSTER_NAME}" \
  --namespace "${NAMESPACE}" \
  -o jsonpath='{.status.network.id}')

echo "Managed Network ID: ${STACKIT_NETWORK_ID}"

Step 2: Deploy CCM

You can provide the CCM manifest using either of two methods:

Since cluster-template-managed-network.yaml already defined a ClusterResourceSet looking for a Secret named ${CLUSTER_NAME}-cloud-provider-stackit, create that Secret in the management cluster. The ClusterResourceSet controller will automatically push it to the workload cluster:

# Render the addon manifest using envsubst and create the CRS Secret
kubectl create secret generic "${CLUSTER_NAME}-cloud-provider-stackit" \
  --namespace "${NAMESPACE}" \
  --type=addons.cluster.x-k8s.io/resource-set \
  --from-literal=cloud-provider-stackit.yaml="$(envsubst < templates/addons/cloud-provider-stackit.yaml)"

To verify the ClusterResourceSet applied the resources:

kubectl get clusterresourcesetbinding -n "${NAMESPACE}"

Method B: Direct Apply to the Workload Cluster

Alternatively, retrieve the workload cluster kubeconfig and apply the rendered addon directly:

clusterctl get kubeconfig "${CLUSTER_NAME}" \
  --namespace "${NAMESPACE}" \
  > "${CLUSTER_NAME}.kubeconfig"

envsubst < templates/addons/cloud-provider-stackit.yaml \
  | kubectl --kubeconfig "${CLUSTER_NAME}.kubeconfig" apply -f -

CNI Installation

After CCM is deployed and the workload API server is reachable, retrieve the kubeconfig if you have not already:

clusterctl get kubeconfig "${CLUSTER_NAME}" \
  --namespace "${NAMESPACE}" \
  > "${CLUSTER_NAME}.kubeconfig"

Nodes remain in NotReady status until a CNI is installed.

Installing Cilium (Default Tested Path)

Install Cilium using the checked-in values matching the cluster template's pod CIDR:

make install-workload-cni \
  WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig"

Alternatively, install with the Cilium CLI or Helm using templates/addons/cilium-values.yaml:

cilium install \
  --kubeconfig "${CLUSTER_NAME}.kubeconfig" \
  --values templates/addons/cilium-values.yaml

Installing Calico

To install Calico instead:

make install-workload-cni \
  WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig" \
  STACKIT_WORKLOAD_CNI=calico

Installing a Custom CNI

To apply an arbitrary CNI manifest:

make install-workload-cni \
  WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig" \
  CNI_MANIFEST=./my-cni.yaml

Verification

Verify that the cloud provider Deployment rolled out:

clusterctl get kubeconfig "${CLUSTER_NAME}" \
  --namespace "${NAMESPACE}" \
  > "${CLUSTER_NAME}.kubeconfig"

Install the default tested Cilium CNI:

make install-workload-cni \
  WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig"

The helper waits for the Cilium operator, Cilium DaemonSet, and Cilium Envoy DaemonSet when present.

To install Calico instead:

make install-workload-cni \
  WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig" \
  STACKIT_WORKLOAD_CNI=calico

To apply a custom CNI manifest:

make install-workload-cni \
  WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig" \
  CNI_MANIFEST=./my-cni.yaml

Custom manifests are applied as-is. The helper cannot know which workload resources prove that an arbitrary CNI is healthy, so custom CNI users must wait for their own rollout resources.

Verification

Verify that the cloud provider Deployment rolled out:

kubectl --kubeconfig "${CLUSTER_NAME}.kubeconfig" \
  -n kube-system rollout status deployment/stackit-cloud-controller-manager \
  --timeout=5m

Verify Cilium rollout for the default path:

kubectl --kubeconfig "${CLUSTER_NAME}.kubeconfig" \
  -n kube-system rollout status deployment/cilium-operator --timeout=10m
kubectl --kubeconfig "${CLUSTER_NAME}.kubeconfig" \
  -n kube-system rollout status daemonset/cilium --timeout=10m
kubectl --kubeconfig "${CLUSTER_NAME}.kubeconfig" \
  -n kube-system rollout status daemonset/cilium-envoy --timeout=10m

Verify Nodes are Ready and have STACKIT provider IDs:

kubectl --kubeconfig "${CLUSTER_NAME}.kubeconfig" get nodes
kubectl --kubeconfig "${CLUSTER_NAME}.kubeconfig" \
  get nodes \
  -o custom-columns=NAME:.metadata.name,PROVIDER_ID:.spec.providerID

Verify the external cloud-provider taint is gone:

kubectl --kubeconfig "${CLUSTER_NAME}.kubeconfig" \
  get nodes -o json \
  | jq -e '[.items[].spec.taints[]? | select(.key == "node.cloudprovider.kubernetes.io/uninitialized")] | length == 0'

Verify CAPI Machines and workload Nodes agree on STACKIT provider IDs:

kubectl -n "${NAMESPACE}" \
  get machines,stackitmachines \
  -l "cluster.x-k8s.io/cluster-name=${CLUSTER_NAME}" \
  -o custom-columns=KIND:.kind,NAME:.metadata.name,PROVIDER_ID:.spec.providerID,NODE:.status.nodeRef.name
kubectl --kubeconfig "${CLUSTER_NAME}.kubeconfig" \
  get nodes \
  -o custom-columns=NAME:.metadata.name,PROVIDER_ID:.spec.providerID

The e2e workload tests validate this flow by waiting for the embedded cloud-provider-stackit rollout, installing CNI, waiting for Ready Nodes, checking the external cloud-provider taint is removed, and verifying CAPI Machine, StackitMachine, and Node provider IDs align.

Production Addon Management

ClusterResourceSet is useful for static bootstrap addons such as this tested cloud-provider installation, but it is not a full addon lifecycle manager. For production CNI management, prefer Helm, GitOps, or a Cluster API addon provider that handles upgrades, rollback, and drift reconciliation.

Workload CNI

The default cluster template provisions STACKIT infrastructure and installs cloud-provider-stackit, but it intentionally does not install a CNI. This keeps CNI choice with the cluster operator, where network policy, routing, MTU, IPAM, and upgrade strategy belong.

Nodes will not become fully Ready until a CNI is installed. The templates ship MachineHealthChecks that replace Machines whose Node stays Ready=False for 10 minutes, so install the CNI within that window after the Nodes join.

For the complete tested workload addon flow, including the embedded cloud-provider-stackit ClusterResourceSet and verification commands, see Workload Addons.

First, retrieve the workload-cluster kubeconfig:

clusterctl get kubeconfig "${CLUSTER_NAME}" \
  --namespace "${NAMESPACE}" \
  > "${CLUSTER_NAME}.kubeconfig"

Install Cilium using the checked-in values:

make install-workload-cni \
  WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig"

The Cilium values live in templates/addons/cilium-values.yaml and match the default pod CIDR from templates/cluster-template.yaml.

To install Calico instead:

make install-workload-cni \
  WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig" \
  STACKIT_WORKLOAD_CNI=calico

To apply a fully custom CNI manifest:

make install-workload-cni \
  WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig" \
  CNI_MANIFEST=./my-cni.yaml

Custom manifests are applied as-is. The helper cannot infer which resources prove that an arbitrary CNI is healthy, so custom CNI users must wait for their own rollout resources.

For production clusters, prefer managing the CNI with Helm, GitOps, or a Cluster API addon provider. ClusterResourceSet is useful for simple static resources, but it is not a full addon lifecycle manager for upgrades, rollback, or drift reconciliation.

Cluster Upgrade

Create the workload cluster with the source version:

export KUBERNETES_VERSION=v1.33.12
clusterctl generate cluster "${CLUSTER_NAME}" \
  --from templates/cluster-template.yaml \
  > "${CLUSTER_NAME}.yaml"
kubectl apply -f "${CLUSTER_NAME}.yaml"

Upgrade the control plane first:

kubectl patch kubeadmcontrolplane "${CLUSTER_NAME}-control-plane" \
  -n "${NAMESPACE}" \
  --type merge \
  -p '{"spec":{"version":"v1.34.8"}}'

Then upgrade the workers:

kubectl patch machinedeployment "${CLUSTER_NAME}-md-0" \
  -n "${NAMESPACE}" \
  --type merge \
  -p '{"spec":{"template":{"spec":{"version":"v1.34.8"}}}}'

Watch rollout state:

kubectl get kubeadmcontrolplane,machinedeployment,machineset,machine \
  -n "${NAMESPACE}" \
  -l "cluster.x-k8s.io/cluster-name=${CLUSTER_NAME}" \
  --watch

The current tested scope verifies MachineDeployment replacement behavior and VM cleanup. Full KubeadmControlPlane and Node readiness verification depends on completing workload-cluster bootstrap and cloud-provider integration.

See the root cluster-upgrade.md tutorial for the detailed step-by-step flow.

Cleanup

Delete a workload cluster through Cluster API:

kubectl delete cluster "${CLUSTER_NAME}" -n "${NAMESPACE}"

Then check Kubernetes resources:

kubectl get machine,stackitmachine,stackitcluster \
  -n "${NAMESPACE}" \
  -l "cluster.x-k8s.io/cluster-name=${CLUSTER_NAME}"

Real e2e resources are labeled so they can be cleaned up directly through the STACKIT API if Kubernetes cleanup fails:

  • cluster-api-provider-stackit/e2e=true
  • cluster-api-provider-stackit/test-id=<unique-id>
  • cluster.x-k8s.io/cluster-name=<cluster-name>
  • cluster.x-k8s.io/cluster-namespace=<namespace>

Run direct cloud cleanup for tagged e2e resources:

make cleanup-stackit

The cleanup path must not depend on Kubernetes objects. It should find and delete resources directly by STACKIT labels.

Topics

In-depth technical guides and configuration references for Cluster API Provider STACKIT:

STACKIT Cloud Resources

The provider provisions and manages workload-cluster servers, API server load balancers, and dedicated cluster networks (when enabled), while supporting pre-existing (BYO) networks, security groups, SSH keys, and images.

Network

StackitCluster.spec.network.id is the STACKIT network ID used by all cluster machines and the provider-managed API server load balancer.

If you don't provide a network ID, CAPSTK will automatically create a dedicated /24 network for the cluster.

Planning your networks usually involves platform-level policy such as address planning, routing, peering, VPN, DNS, and shared-service access, but the default managed network is built for convenience. A newly created (managed) network will have an attached tag indicating that it is managed by CAPSTK (cluster-api-provider-stackit=true).

Only empty networks can be deleted, so when you delete your CAPSTK cluster and all cluster nodes and load balancers are removed, the managed network will be deleted too.

Note: When using a managed network, the network ID is generated dynamically during cluster reconciliation and populated on StackitCluster.status.network.id. Because templates/addons/cloud-provider-stackit.yaml requires this network ID for its Service type LoadBalancer controller, you use templates/cluster-template-managed-network.yaml and configure the Cloud Controller Manager after the network ID is provisioned. See Cloud Provider with Managed Networks for the step-by-step procedure.

Bring-Your-Own (BYO) Network

Choose or create a network in the same region as the workload cluster. The network must provide:

  • IP addresses for all control-plane and worker machines.
  • Routing between workload-cluster nodes.
  • Egress for cloud-init, package repositories when using development images, container image pulls, DNS, NTP, and STACKIT APIs.
  • Traffic required by the selected CNI.

How to create a STACKIT Network manually

Create a network with the STACKIT CLI:

export STACKIT_REGION=eu01
export STACKIT_NETWORK_NAME="${CLUSTER_NAME}-network"

stackit network create \
  --project-id "${STACKIT_PROJECT_ID}" \
  --region "${STACKIT_REGION}" \
  --name "${STACKIT_NETWORK_NAME}" \
  --ipv4-prefix "10.42.0.0/24" \
  --labels cluster-api-provider-stackit=true

Get the network ID for STACKIT_NETWORK_ID:

stackit network list \
  --project-id "${STACKIT_PROJECT_ID}" \
  --region "${STACKIT_REGION}" \
  --output-format json

Delete a test network when it is no longer used:

stackit network delete "${STACKIT_NETWORK_ID}" \
  --project-id "${STACKIT_PROJECT_ID}" \
  --region "${STACKIT_REGION}"

Security Groups

StackitMachine.spec.securityGroups is an optional list of STACKIT security group IDs attached to each created server. If omitted, STACKIT project defaults apply.

For production clusters, prefer explicit security groups over implicit project defaults. At minimum, the selected security groups must allow:

  • The Kubernetes API server on TCP 6443 from the API server load balancer and from trusted administration networks.
  • Node-to-node traffic required by kubeadm, kubelet, the control plane, and the selected CNI.
  • Egress for bootstrap, package repositories if needed, image registries, DNS, NTP, and STACKIT APIs.

Do not open SSH ingress by default. If SSH is needed for break-glass debugging, restrict it to trusted CIDRs.

Create a security group:

export STACKIT_SECURITY_GROUP_NAME="${CLUSTER_NAME}-nodes"

stackit security-group create \
  --project-id "${STACKIT_PROJECT_ID}" \
  --region "${STACKIT_REGION}" \
  --name "${STACKIT_SECURITY_GROUP_NAME}" \
  --stateful \
  --description "Cluster API STACKIT nodes" \
  --labels cluster-api-provider-stackit=true

Get the security group ID:

stackit security-group list \
  --project-id "${STACKIT_PROJECT_ID}" \
  --region "${STACKIT_REGION}" \
  --output-format json

Add a Kubernetes API ingress rule, replacing STACKIT_SECURITY_GROUP_ID and ADMIN_CIDR:

stackit security-group rule create \
  --project-id "${STACKIT_PROJECT_ID}" \
  --region "${STACKIT_REGION}" \
  --security-group-id "${STACKIT_SECURITY_GROUP_ID}" \
  --direction ingress \
  --protocol-name tcp \
  --port-range-min 6443 \
  --port-range-max 6443 \
  --ip-range "${ADMIN_CIDR}" \
  --description "Kubernetes API"

Delete a test security group when it is no longer attached to servers:

stackit security-group delete "${STACKIT_SECURITY_GROUP_ID}" \
  --project-id "${STACKIT_PROJECT_ID}" \
  --region "${STACKIT_REGION}"

SSH Keys

StackitMachine.spec.sshKeyName is optional. It references an existing STACKIT SSH key pair by name.

SSH is not used for Cluster API bootstrap. Kubeadm bootstrap data is generated by CABPK, stored in a Kubernetes Secret, and sent to STACKIT as cloud-init user data when each server is created. Leave sshKeyName and STACKIT_SSH_KEY_NAME empty when SSH access is not required.

Because the CAPSTK controller accesses STACKIT through a service account, the SSH key pairs must be attached to that service account. Key pairs are scoped per identity and cannot be shared across service accounts or users.

Authenticate the STACKIT CLI using your service account credentials:

stackit auth activate-service-account \
  --service-account-key-path "${STACKIT_SERVICE_ACCOUNT_JSON_FILE}"

Import an existing SSH public key:

export STACKIT_SSH_KEY_NAME="${CLUSTER_NAME}-break-glass"

stackit key-pair create \
  --project-id "${STACKIT_PROJECT_ID}" \
  --region "${STACKIT_REGION}" \
  --name "${STACKIT_SSH_KEY_NAME}" \
  --public-key "@${HOME}/.ssh/id_ed25519.pub" \
  --labels cluster-api-provider-stackit=true

Only create and pass a key pair when operators need SSH for debugging. Opening SSH access also requires an explicit security group rule; the provider does not need SSH to create or join Nodes.

Delete a test key pair:

stackit key-pair delete "${STACKIT_SSH_KEY_NAME}" \
  --project-id "${STACKIT_PROJECT_ID}" \
  --region "${STACKIT_REGION}"

Node Images

StackitMachine.spec.imageID is the STACKIT image ID used for the node root disk. The image must be compatible with cloud-init and kubeadm bootstrap.

For production clusters, use kubeadm-ready images that already contain the expected container runtime and Kubernetes node packages for the chosen Kubernetes minor. Runtime package installation is useful for development, but it adds dependency on external package repositories during cluster creation and slows down Machine readiness.

The current real e2e path can bootstrap generic Ubuntu images by installing containerd, kubelet, kubeadm, and kubectl in kubeadm preKubeadmCommands. Treat this as a development fallback, not the desired production path.

List candidate images:

stackit image list \
  --project-id "${STACKIT_PROJECT_ID}" \
  --region "${STACKIT_REGION}" \
  --limit 20 \
  --output-format json

Use the selected image ID as STACKIT_IMAGE_ID. Prefer non-ARM Ubuntu images or purpose-built kubeadm-ready images until this provider publishes a supported image matrix.

Verification

The command forms above were verified against STACKIT with temporary resources:

  • stackit network create, network describe, and network delete
  • stackit security-group create, security-group rule create, and security-group delete
  • stackit key-pair create and key-pair delete
  • stackit image list

OS Images

Ubuntu Nodes

CAPSTK includes templates/cluster-template.yaml as the default example for Ubuntu-based clusters:

  • control-plane machines use the Ubuntu image from STACKIT_IMAGE_ID
  • worker machines use the same Ubuntu image from STACKIT_IMAGE_ID
  • kubeadm bootstrap runs through cloud-init

Select one of the available Ubuntu images:

stackit image list \
    --project-id "${STACKIT_PROJECT_ID}" \
    --region "${STACKIT_REGION}" \
    --output-format json \
    | jq -r '.[] | select(.name | test("Ubuntu")) | "\(.id)\t\(.name)"'

The default template currently supports generic Ubuntu cloud images by installing Kubernetes node dependencies at first boot:

  • install containerd, kubelet, kubeadm, and kubectl with apt-get
  • use the Kubernetes package repository selected by KUBERNETES_APT_REPOSITORY_MINOR
  • configure containerd with SystemdCgroup = true
  • enable br_netfilter and Kubernetes bridge/IP forwarding sysctls
  • register kubelet with cloud-provider=external
  • make kubeadm use the STACKIT hostname from cloud-init metadata ({{ ds.meta_data.local_hostname }}), so cloud-provider-stackit can map the Kubernetes node to the STACKIT server

Render the default template like any other clusterctl template:

export STACKIT_IMAGE_ID=<ubuntu-image-id>
export KUBERNETES_VERSION=v1.35.8
export KUBERNETES_APT_REPOSITORY_MINOR=v1.35

clusterctl generate cluster "${CLUSTER_NAME}" \
  --from templates/cluster-template.yaml \
  --kubernetes-version "${KUBERNETES_VERSION}" \
  --control-plane-machine-count "${CONTROL_PLANE_MACHINE_COUNT}" \
  --worker-machine-count "${WORKER_MACHINE_COUNT}" \
  | kubectl apply -f -

Notes:

  • Ubuntu SSH access normally uses the ubuntu user.
  • Install a workload CNI after the API server is reachable.
  • The default template downloads Kubernetes packages during first boot. This is useful for development and e2e testing, but production clusters should use a kubeadm-ready image that already contains the expected container runtime and Kubernetes node packages for the chosen Kubernetes minor.
  • Prefer non-ARM Ubuntu images until this provider publishes a supported image matrix.

Flatcar Worker Nodes

See the official Flatcar STACKIT docs for generic Flatcar image usage on STACKIT.

CAPSTK includes templates/cluster-template-flatcar-workers.yaml as an example for mixed-image clusters:

  • control-plane machines use the regular Ubuntu image from STACKIT_IMAGE_ID
  • worker machines use the Flatcar image from STACKIT_WORKER_IMAGE_ID

Example amd64 Flatcar image:

export STACKIT_WORKER_IMAGE_ID=419c31da-39e3-4ea3-9bd8-699b44e8394f # Flatcar 4459.2.4

Flatcar requires a different bootstrap path than Ubuntu:

  • use KubeadmConfigTemplate.spec.template.spec.format: ignition
  • enable Cluster API's KubeadmBootstrapFormatIgnition feature gate
  • do not reuse Ubuntu apt-get based bootstrap commands
  • make kubeadm use the STACKIT hostname from /run/metadata/flatcar (COREOS_OPENSTACK_HOSTNAME), otherwise the node may register as localhost and cloud-provider-stackit cannot map it to the STACKIT server

Render the example template like any other clusterctl template:

clusterctl generate cluster "${CLUSTER_NAME}" \
  --from templates/cluster-template-flatcar-workers.yaml \
  --kubernetes-version "${KUBERNETES_VERSION}" \
  --control-plane-machine-count "${CONTROL_PLANE_MACHINE_COUNT}" \
  --worker-machine-count "${WORKER_MACHINE_COUNT}" \
  | kubectl apply -f -

Notes:

  • Flatcar SSH access uses the core user. When connecting through the CAPSTK bastion, use the bastion image's SSH user for the first hop, usually ubuntu, and core for the Flatcar node:

    ssh -i "${CLUSTER_SSH_KEY}" core@<flatcar-node-internal-ip> \
      -o "ProxyCommand ssh -W %h:%p -i ${CLUSTER_SSH_KEY} ubuntu@${BASTION_HOST}"
    
  • Install a workload CNI after the API server is reachable.

  • The example template downloads Kubernetes binaries during first boot. Production images should provide those binaries through an image build, package, or system extension instead.

Accessing cluster instances

By default, workload clusters created by cluster-api-provider-stackit do not expose SSH access. Cluster API bootstrap does not require SSH; CABPK provides cloud-init user data through Kubernetes bootstrap Secrets.

For break-glass access, a StackitCluster can optionally ask the provider to create one managed SSH bastion VM. The bastion is attached to the same STACKIT network as the cluster nodes, gets a provider-managed public IP, and has a provider-managed security group allowing TCP/22 only from configured CIDRs.

When bastion access is enabled, the provider also manages the node-side SSH path: it creates a separate security group for cluster nodes, allows TCP/22 from the bastion security group, and attaches that node SSH security group to control-plane and worker VMs.

Prerequisites

  • The workload cluster was generated with clusterctl generate cluster and applied to the management cluster.
  • The configured STACKIT service account role includes the bastion permissions documented in IAM Permissions Used.
  • The bastion image is an Ubuntu image with SSH enabled for the expected user, normally ubuntu.
  • The STACKIT SSH key named in spec.bastion.sshKeyName already exists for the service account used by the provider. STACKIT key pairs are not shared across service accounts, so importing a key with an inspection or admin service account does not make it usable by the controller service account.
  • To SSH from the bastion into cluster nodes, the control-plane and worker StackitMachineTemplate resources must also set spec.template.spec.sshKeyName. The bastion does not inject SSH keys into existing node VMs.

Configure SSH keys for the service account

The CAPSTK controller authenticates to STACKIT through a service account key. Because SSH key pairs are scoped to the authenticated identity in this case a service account, the SSH keys you want to use on the bastion or on cluster machines must be attached under that exact service account and not your personal user.

  1. Generate an SSH key pair locally if you do not already have one:

    ssh-keygen -t ed25519 -f ~/.ssh/id_stackit -C "capstk-access"
    
  2. Authenticate the STACKIT CLI using your service account credentials:

    stackit auth activate-service-account \
      --service-account-key-path /path/to/service-account.json
    
  3. Import the public key as a key pair under that service account:

    export STACKIT_SSH_KEY_NAME="capstk-key"
    
    stackit key-pair create \
      --project-id "${STACKIT_PROJECT_ID}" \
      --region "${STACKIT_REGION}" \
      --name "${STACKIT_SSH_KEY_NAME}" \
      --public-key "@${HOME}/.ssh/id_stackit.pub"
    
  4. Reference the key in your cluster definition:

    When using templates/cluster-template-bastion.yaml, pass the key name to the template variables:

    • STACKIT_BASTION_SSH_KEY_NAME: sets spec.bastion.sshKeyName on StackitCluster
    • STACKIT_SSH_KEY_NAME: sets spec.template.spec.sshKeyName on StackitMachineTemplate for control-plane and worker nodes

    You can use separate keys for the bastion and the nodes, or point both variables to the same key.

Enable the bastion

Patch or edit the generated StackitCluster:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: StackitCluster
metadata:
  name: <cluster-name>
  namespace: <namespace>
spec:
  bastion:
    enabled: true
    imageID: <ubuntu-image-id>
    machineType: <machine-type>
    sshKeyName: <existing-stackit-ssh-key-name>
    allowedCIDRs:
      - <your-public-ip-or-network-cidr>
    rootVolume:
      sizeGiB: 50
      performanceClass: storage_premium_perf6
      deleteOnTermination: true
    cloudInitRef:
      kind: ConfigMap
      name: <bastion-cloud-init-configmap>
      key: userData

Use a narrow allowedCIDRs value such as 203.0.113.10/32 where possible. 0.0.0.0/0 allows SSH from anywhere and should only be used deliberately.
Set rootVolume when the chosen machine type's flavor disk is too small for the chosen image.
Set cloudInitRef when the bastion host needs additional packages, users, files, or other cloud-init customization. The provider reads the referenced ConfigMap or Secret and passes the value as-is to the bastion VM. It is not applied to control-plane or worker nodes.

For node access, set the same or another existing SSH key on the machine templates:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: StackitMachineTemplate
metadata:
  name: <machine-template-name>
spec:
  template:
    spec:
      sshKeyName: <existing-stackit-ssh-key-name>

Leaving StackitMachineTemplate.spec.template.spec.sshKeyName empty is valid, but SSH to nodes through the bastion will not work.

How node SSH access is wired

Enabling the bastion creates two provider-managed SSH security groups:

  • The bastion SSH security group is attached to the bastion VM. It allows TCP/22 from spec.bastion.allowedCIDRs.
  • The node SSH security group is attached to each provider-managed control-plane and worker VM. It allows TCP/22 from the bastion security group by using the bastion security group as the remote source.

This means users connect from their workstation to the bastion public IP, and then from the bastion to node internal IPs. The provider manages the network permissions for both hops, but it does not manage SSH users or private key files. The SSH key named on the bastion and node specs must already exist in STACKIT for the service account used by the provider.

The node SSH security group is shared by all nodes in the cluster. New nodes created while bastion is enabled get the group attached during StackitMachine reconciliation.

Customize the bastion with cloud-init

spec.bastion.cloudInitRef references a complete cloud-init user-data document for the bastion VM. Use a ConfigMap for non-sensitive configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: bastion-cloud-init
data:
  userData: |
    #cloud-config
    packages:
      - jq
---
spec:
  bastion:
    enabled: true
    imageID: <ubuntu-image-id>
    machineType: <machine-type>
    sshKeyName: <existing-stackit-ssh-key-name>
    allowedCIDRs:
      - <your-public-ip-or-network-cidr>
    cloudInitRef:
      kind: ConfigMap
      name: bastion-cloud-init
      key: userData

Use a Secret instead when the cloud-init document contains sensitive values:

apiVersion: v1
kind: Secret
metadata:
  name: bastion-cloud-init
stringData:
  userData: |
    #cloud-config
    packages:
      - jq
---
spec:
  bastion:
    cloudInitRef:
      kind: Secret
      name: bastion-cloud-init
      key: userData

The referenced object must be in the same namespace as the StackitCluster. The provider does not merge the referenced data with node bootstrap data. Cloud-init user data is applied only when STACKIT creates a VM, so an existing bastion cannot be reconfigured in place. If the referenced ConfigMap or Secret content changes, the provider deletes and recreates the provider-managed bastion VM with the new user data.

Recreating the bastion temporarily interrupts SSH access, assigns a new provider-managed bastion server, and may assign a new public IP. The provider also removes the old node SSH security group path and recreates it for the new bastion so control-plane and worker nodes remain reachable through the current bastion after reconciliation completes.

Get the bastion IP

Use the management-cluster kubeconfig:

kubectl get stackitcluster <cluster-name> \
  --namespace <namespace> \
  -o jsonpath='{.status.bastion.publicIP}'

Or inspect the printcolumn:

kubectl get stackitclusters --namespace <namespace>

The Bastion IP column is empty until STACKIT has assigned and attached the public IP.

Get node internal IPs

Use the workload-cluster kubeconfig:

kubectl get nodes \
  -o custom-columns=NAME:.metadata.name,IP:'{.status.addresses[?(@.type=="InternalIP")].address}'

Alternatively, inspect the management-cluster StackitMachine addresses:

kubectl get stackitmachines --namespace <namespace> \
  -o custom-columns=NAME:.metadata.name,IP:'{.status.addresses[?(@.type=="InternalIP")].address}'

Connect through the bastion

Set local variables:

export CLUSTER_SSH_KEY="$HOME/.ssh/<private-key-file>"
export BASTION_HOST="$(kubectl get stackitcluster <cluster-name> \
  --namespace <namespace> \
  -o jsonpath='{.status.bastion.publicIP}')"

Connect to a node internal IP through the bastion:

ssh -i "${CLUSTER_SSH_KEY}" ubuntu@<node-internal-ip> \
  -o "ProxyCommand ssh -W %h:%p -i ${CLUSTER_SSH_KEY} ubuntu@${BASTION_HOST}"

Use the node image's SSH user for the final hop. For example, Ubuntu nodes usually use ubuntu, while Flatcar nodes use core.

An equivalent SSH config entry:

Host 10.*
  User ubuntu
  IdentityFile <cluster-ssh-key>
  ProxyCommand ssh -W %h:%p ubuntu@<bastion-public-ip>

Cleanup

The provider deletes the managed bastion server, public IP, security group, and security group rules when the StackitCluster is deleted. It also removes the provider-managed node SSH security group from cluster nodes and deletes that security group.

The same cleanup path runs when spec.bastion.enabled is changed from true to false: the provider removes the node SSH security group from provider-managed control-plane and worker VMs, deletes the node SSH security group and its rules, and then deletes the bastion server, public IP, and bastion security group.

Additional Notes

Using the STACKIT CLI instead of kubectl

It is also possible to use STACKIT CLI commands instead of kubectl to gather information about the cluster nodes.

For example, to use the STACKIT CLI to get the public IP address of the SSH bastion host, use this STACKIT CLI command:

set CLUSTER_NAME stackit-bastion-test

stackit server list -o json | jq -r --arg cluster "$CLUSTER_NAME" '
    .[]
    | select(.labels["cluster.x-k8s.io/cluster-name"] == $cluster)
    | select(.labels["cluster-api-provider-stackit/resource-role"] == "bastion")
    | .nics[]?.publicIp
'
188.34.94.28

You should substitute the correct cluster name for <CLUSTER_NAME> in the above command.

Similarly, to obtain the list of private IP addresses of the cluster nodes, use this STACKIT CLI command:

set CLUSTER_NAME stackit-bastion-test

stackit server list -o json | jq -r --arg cluster "$CLUSTER_NAME" '
  .[]
  | select(.labels["cluster.x-k8s.io/cluster-name"] == $cluster)
  | select(.labels["cluster-api-provider-stackit/resource-role"] != "bastion")
  | .nics[]?.ipv4
'

For names plus private IPs:

stackit server list -o json | jq -r --arg cluster "$CLUSTER_NAME" '
  .[]
  | select(.labels["cluster.x-k8s.io/cluster-name"] == $cluster)
  | select(.labels["cluster-api-provider-stackit/resource-role"] != "bastion")
  | [.name, (.nics[]?.ipv4 // empty)]
  | @tsv
'

Finally, to obtain STACKIT instance names mapped with their private IPs, you can use this STACKIT CLI command:

stackit server list -o json | jq -r --arg cluster "$CLUSTER_NAME" '
  .[]
  | select(.labels["cluster.x-k8s.io/cluster-name"] == $cluster)
  | select(.labels["cluster-api-provider-stackit/resource-role"] != "bastion")
  | [.name, (.nics[]?.ipv4 // empty)]
  | @tsv
'

Note that your STACKIT CLI must be configured with credentials that enable you to query the STACKIT Servers API.

HA, Availability Zones & Failure Domains

"Availability Zone" is a cloud provider-specific term, whereas "Failure Domain" is the broader, environment-agnostic concept used throughout Cluster API. A failure domain can represent a separation of physical switches, power feeds, rooms, racks, or distinct data centers. In the context of public cloud providers like STACKIT, failure domains and availability zones (AZs) are equivalent.

Failure Domain Discovery

The provider dynamically discovers STACKIT availability zones from the STACKIT IaaS API (/v2/regions/{region}/availability-zones) and publishes them as Cluster API failure domains on StackitCluster.status.failureDomains.

For a region such as eu01, the provider exposes the real physical zones:

eu01-1
eu01-2
eu01-3

Virtual metro zones (such as eu01-m with an -m suffix) are filtered out from published failure domains so that machines are not placed into expensive replicated metro zones by default.

Control Plane Spreading

Control plane spreading across failure domains is handled directly by Cluster API (KubeadmControlPlane). When scaling or rolling out control plane nodes, KCP inspects existing control plane machines and assigns the least-populated failure domain to Machine.spec.failureDomain (breaking ties deterministically).

For this automatic spreading to function correctly, the failure domain list in StackitCluster.status.failureDomains must be accurately populated and marked with controlPlane: true.

Machine Placement and Defaulting

When provisioning a machine, the provider determines the target availability zone using the following precedence:

  1. Machine.spec.failureDomain: Honors the placement selected by Cluster API core (e.g., KCP spreading control plane nodes or a MachineDeployment configured with a failure domain).
  2. StackitMachine.spec.availabilityZone: If explicitly configured on the StackitMachine (or via StackitMachineTemplate), this zone is used.
  3. Provider Defaulting: When failureDomain is omitted, placement depends entirely on the infrastructure provider. If neither Machine.spec.failureDomain nor StackitMachine.spec.availabilityZone is specified (such as worker machines in a default MachineDeployment), the provider sorts the cluster's published failure domains alphabetically and selects the first zone (e.g., eu01-1). This matches standard Cluster API conventions (such as CAPG) and ensures VMs are never placed into the expensive STACKIT metro zone (eu01-m) by default.

The actual zone where the VM was placed is surfaced on StackitMachine.status.availabilityZone.

Worker Node Spreading

Unlike the control plane, worker nodes do not automatically spread across failure domains within a single MachineDeployment. Spreading worker nodes across AZs is an end-user architecture decision made using the configuration fields on MachineDeployment or MachinePool types.

The standard Cluster API pattern is to define multiple MachineDeployment resources (one per availability zone), pinning each to a specific failure domain via spec.template.spec.failureDomain.

IAM Permissions Used

cluster-api-provider-stackit authenticates to STACKIT with the service account JSON stored in the StackitCluster.spec.credentialsSecretRef Secret. That service account should use a custom project role with only the permissions needed by the infrastructure provider.

Do not use broad project administrator roles for the controller. STACKIT documents custom roles as the way to bundle an explicit permission set, and role bindings as the way to assign that role to a user or service account.

Required provider permissions

The current provider implementation uses these STACKIT API operations:

Provider actionCode pathSTACKIT permission
Read the configured networkGetNetworkiaas.network.get
Create managed network (if enabled)CreateNetworkiaas.network.create
Delete managed network (if enabled)DeleteNetworkiaas.network.delete
Create VM instancesCreateServeriaas.server.create
Find existing tagged VM instancesListServersiaas.server.list
Read VM stateGetServeriaas.server.get
Read VM NIC addresses for CAPI addresses and load balancer targetsListServerNICsiaas.server.nic.list
Delete VM instancesDeleteServeriaas.server.delete
Create the optional bastion public IPCreatePublicIPiaas.public-ip.create
Find existing tagged bastion public IPsListPublicIPsiaas.public-ip.list
Read the bastion public IP after attachGetPublicIPiaas.public-ip.get
Attach the bastion public IP to the bastion serverAddPublicIpToServeriaas.server.public-ip.add
Detach the bastion public IP during cleanupRemovePublicIpFromServeriaas.server.public-ip.remove
Delete the optional bastion public IPDeletePublicIPiaas.public-ip.delete
Create the optional bastion security groupCreateSecurityGroupiaas.security-group.create
Find existing tagged bastion security groupsListSecurityGroupsiaas.security-group.list
Create bastion SSH ingress rulesCreateSecurityGroupRuleiaas.security-group.rule.create
List bastion SSH ingress rulesListSecurityGroupRulesiaas.security-group.rule.list
Delete bastion SSH ingress rules during cleanupDeleteSecurityGroupRuleiaas.security-group.rule.delete
Attach the bastion security group to the bastion serverAddSecurityGroupToServeriaas.server.security-group.add
Detach the bastion security group during cleanupRemoveSecurityGroupFromServeriaas.server.security-group.remove
Delete the optional bastion security groupDeleteSecurityGroupiaas.security-group.delete
Create the API server network load balancerCreateLoadBalancernlb.loadbalancer.create
Find existing tagged load balancersListLoadBalancersnlb.loadbalancer.list
Read the load balancer before target updatesGetLoadBalancernlb.loadbalancer.get
Delete the API server load balancerDeleteLoadBalancernlb.loadbalancer.delete
Replace the API server target poolUpdateTargetPoolnlb.targetpool.replace

The least-privilege role for the current provider is therefore:

iaas.network.create
iaas.network.delete
iaas.network.get
iaas.public-ip.create
iaas.public-ip.delete
iaas.public-ip.get
iaas.public-ip.list
iaas.server.create
iaas.server.delete
iaas.server.get
iaas.server.list
iaas.server.nic.list
iaas.server.public-ip.add
iaas.server.public-ip.remove
iaas.server.security-group.add
iaas.server.security-group.remove
iaas.security-group.create
iaas.security-group.delete
iaas.security-group.list
iaas.security-group.rule.create
iaas.security-group.rule.delete
iaas.security-group.rule.list
nlb.loadbalancer.create
nlb.loadbalancer.delete
nlb.loadbalancer.get
nlb.loadbalancer.list
nlb.targetpool.replace

This list covers StackitCluster and StackitMachine reconciliation, including the optional provider-managed bastion host. It does not include permissions for manually creating networks, SSH keys, images, or other prerequisite resources. It also does not include permissions for the in-cluster cloud-provider-stackit add-on if you configure that add-on to manage Kubernetes Service load balancers beyond the provider-managed API server load balancer.

Create a strict role and service account with OpenTofu

Use OpenTofu and the STACKIT provider to create the custom role, service account, role assignment, and service-account key as one managed setup.

The bootstrap identity used by OpenTofu needs these setup permissions:

  • iam.role.add to create the custom role
  • iam.role.get and iam.role.list to read role state
  • iam.member.add to assign the role to the service account
  • iam.member.get to read role-assignment state
  • iam.service-account.create to create the service account
  • iam.service-account.get and iam.service-account.list to read service-account state
  • iam.service-account-key.create to create the service-account key

If the same OpenTofu configuration should also destroy the setup later, the bootstrap identity also needs the corresponding remove/delete permissions: iam.role.remove, iam.service-account.delete, and iam.service-account-key.delete.

Create a STACKIT role:

resource "stackit_authorization_project_custom_role" "cluster_api_provider_stackit" {
  resource_id = var.project_id
  name        = var.role_name
  description = "Least-privilege role for cluster-api-provider-stackit VM, bastion, and API-server load-balancer reconciliation."

  permissions = [
    "iaas.network.get",
    "iaas.public-ip.create",
    "iaas.public-ip.delete",
    "iaas.public-ip.get",
    "iaas.public-ip.list",
    "iaas.server.create",
    "iaas.server.delete",
    "iaas.server.get",
    "iaas.server.list",
    "iaas.server.nic.list",
    "iaas.server.public-ip.add",
    "iaas.server.public-ip.remove",
    "iaas.server.security-group.add",
    "iaas.server.security-group.remove",
    "iaas.security-group.create",
    "iaas.security-group.delete",
    "iaas.security-group.list",
    "iaas.security-group.get",
    "iaas.security-group.rule.create",
    "iaas.security-group.rule.delete",
    "iaas.security-group.rule.list",
    "iaas.security-group.rule.get",
    "nlb.loadbalancer.create",
    "nlb.loadbalancer.delete",
    "nlb.loadbalancer.get",
    "nlb.loadbalancer.list",
    "nlb.targetpool.replace",
  ]
}

Create a STACKIT service account, assign the role and create a service account key:

resource "stackit_service_account" "cluster_api_provider_stackit" {
  project_id = var.project_id
  name       = var.service_account_name
}

resource "stackit_authorization_project_role_assignment" "cluster_api_provider_stackit" {
  resource_id = var.project_id
  role        = stackit_authorization_project_custom_role.cluster_api_provider_stackit.name
  subject     = stackit_service_account.cluster_api_provider_stackit.email
}

resource "stackit_service_account_key" "cluster_api_provider_stackit" {
  project_id            = var.project_id
  service_account_email = stackit_service_account.cluster_api_provider_stackit.email

  depends_on = [
    stackit_authorization_project_role_assignment.cluster_api_provider_stackit,
  ]
}

You will find a working example in hack/tf/iam-setup.

To apply it:

tofu init

tofu apply \
  -var "project_id=${STACKIT_PROJECT_ID}" \
  -var "bootstrap_service_account_key_path=${BOOTSTRAP_SERVICE_ACCOUNT_KEY_PATH}"

Write the generated key to a local file:

export STACKIT_SERVICE_ACCOUNT_JSON_FILE=./.stackit/cluster-api-provider-stackit-serviceaccount.json

mkdir -p "$(dirname "${STACKIT_SERVICE_ACCOUNT_JSON_FILE}")"
tofu output -raw service_account_key_json > "${STACKIT_SERVICE_ACCOUNT_JSON_FILE}"

Next, create the Kubernetes Secret used by StackitCluster:

kubectl create secret generic stackit-credentials \
  --namespace default \
  --from-literal=project-id="${STACKIT_PROJECT_ID}" \
  --from-file=serviceaccount.json="${STACKIT_SERVICE_ACCOUNT_JSON_FILE}"

For Developers

Verify the strict role with the billable e2e tests, not only by reading the permission list. The create/delete scenario exercises VM creation, VM lookup, VM deletion, load balancer creation, target-pool updates, and load balancer cleanup.

Run at least:

export STACKIT_E2E_CREATE_CLUSTER=true
export STACKIT_E2E_NODE_REF=true
export STACKIT_CREDENTIALS_SECRET_NAME=stackit-credentials
export STACKIT_CREDENTIALS_SECRET_NAMESPACE=default

make test-e2e-workload-noderef

To verify the optional bastion path, first import the SSH key pair with the same service account stored in stackit-credentials; key pairs imported with a different service account are not visible to the provider. Then set STACKIT_SSH_KEY_NAME and STACKIT_BASTION_SSH_KEY_NAME and run:

export STACKIT_E2E_CREATE_CLUSTER=true
export STACKIT_E2E_BASTION=true
export STACKIT_SSH_KEY_NAME=<provider-service-account-keypair-name>
export STACKIT_BASTION_SSH_KEY_NAME="${STACKIT_SSH_KEY_NAME}"

make test-e2e-workload-bastion

For release validation, also run the scale, worker-upgrade, control-plane upgrade, and topology e2e targets with the same strict service account:

make test-e2e-workload-scale
make test-e2e-workload-upgrade-workers
make test-e2e-workload-upgrade-control-plane
make test-e2e-workload-topology

[!CAUTION]
Some broader SDK integration tests call helper APIs that are not used by the provider at runtime. For example, TestSDKClientListNetworksIntegration calls ListNetworks and therefore needs iaas.network.list; the provider reconciler only calls GetNetwork with the configured network ID, so iaas.network.get is sufficient for runtime.

Developer Guide

Use this guide to create a local development environment from a fresh checkout. It creates a Kind management cluster, installs Cluster API, and runs the STACKIT provider image you build from this repository. That management cluster then creates workload clusters in STACKIT, so use a project intended for development and expect the workload resources to incur cost.

Before you start

Install:

  • Go
  • Docker
  • kind
  • kubectl
  • clusterctl
  • base64

You also need a STACKIT project, a service-account JSON key, an existing network, an image, and a machine type. The service account needs the permissions listed in IAM permissions.

Clone this repository and run the commands below from its root.

Create the management cluster

The management cluster runs the Cluster API controllers and the provider. It is not the Kubernetes cluster that will run your workload.

Enterprise proxies like Zscaler

If your network uses a TLS-intercepting proxy such as Zscaler, the Kind node must trust the proxy's root certificate. The host's certificate store does not automatically apply inside the Kind node. Create a local kind-config.yaml that mounts the host CA bundle into the node:

cat > kind-config.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraMounts:
      - hostPath: /etc/ssl/certs/ca-certificates.crt
        containerPath: /etc/ssl/certs/ca-certificates.crt
        readOnly: true
EOF
kind create cluster --name capi-stackit --config kind-config.yaml

Without such a proxy, create the cluster without the configuration file:

kind create cluster --name capi-stackit

Point kubectl at the new management cluster:

kubectl config use-context kind-capi-stackit

Install Cluster API

Install Cluster API core, bootstrap, and control-plane providers into the management cluster:

clusterctl init \
  --config hack/clusterctl-local.yaml \
  --core cluster-api \
  --bootstrap kubeadm \
  --control-plane kubeadm

hack/clusterctl-local.yaml enables Cluster API features needed by the repository's templates, including ClusterClass and ClusterResourceSet.

Build and deploy the local provider

Build the controller image, load it into the Kind node, and deploy the provider with that image:

export IMG=cluster-api-provider-stackit:dev
make docker-build IMG="${IMG}"
kind load docker-image "${IMG}" --name capi-stackit
make deploy IMG="${IMG}"

Wait until the locally built provider is ready:

kubectl rollout status \
  --namespace cluster-api-provider-stackit-system \
  deployment/cluster-api-provider-stackit-controller-manager

Configure STACKIT and create a workload cluster

The provider is now running from your checkout. Continue with the quick start from set credentials and cluster settings. Create the Secret in the default namespace used there, then create the workload cluster.

Because you are working from a repository checkout, use templates/cluster-template.yaml directly instead of downloading a release asset:

clusterctl generate cluster "${CLUSTER_NAME}" \
  --from templates/cluster-template.yaml \
  --target-namespace "${NAMESPACE}" \
  > cluster.yaml
kubectl apply -f cluster.yaml

When the workload API is available, retrieve the workload kubeconfig:

clusterctl get kubeconfig "${CLUSTER_NAME}" \
  --namespace "${NAMESPACE}" \
  > "${CLUSTER_NAME}".kubeconfig

Install a CNI using the repository helper (hack/install-workload-cni.sh) via make:

make install-workload-cni \
  WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig"

By default, this installs Cilium using templates/addons/cilium-values.yaml. You can also install Calico by setting STACKIT_WORKLOAD_CNI=calico, or apply a custom manifest by setting CNI_MANIFEST=path/to/manifest.yaml. See Workload CNI for more options.

Run the controller on your host

For controller debugging, stop the deployed provider first, keep kind-capi-stackit as the active kubeconfig context, and run:

make undeploy
make run

make run connects to the active management cluster. When you stop it, rebuild, load, and deploy the image again with the commands in the previous section.

For a live-reload development loop that automatically rebuilds and redeploys the controller on Go code changes, see Development with Tilt.

Cleanup

To fully delete a workload cluster and its cloud infrastructure:

kubectl delete -f cluster.yaml

Deletion should take at most 5 minutes. Verify that all resources are gone:

kubectl get cluster,machine,stackitcluster,stackitmachine --namespace "${NAMESPACE}"

Troubleshooting stuck deletion

If resources persist after several minutes:

  1. Check the status and conditions of persisting resources:

    kubectl describe cluster,machine,stackitcluster,stackitmachine --namespace "${NAMESPACE}"
    
  2. Inspect the controller logs for errors:

    # Core Cluster API controller
    kubectl logs -n capi-system deployment/capi-controller-manager -c manager --tail=100
    
    # STACKIT provider controller
    kubectl logs -n cluster-api-provider-stackit-system deployment/cluster-api-provider-stackit-controller-manager --tail=100
    
  3. If a StackitMachine cannot finish deletion and blocks its parent resources, you can remove its finalizer:

    kubectl patch stackitmachine <stackitmachine-name> -n "${NAMESPACE}" \
      --type=merge -p '{"metadata":{"finalizers":null}}'
    

    Removing the finalizer unblocks Kubernetes deletion for that resource and its parents. You may then have to manually delete the actual VM in the STACKIT portal or through the STACKIT API.

Architecture

Overview

The provider has four main areas:

  • API types and CRDs
  • Controllers
  • Cloud abstraction layer
  • clusterctl templates and release assets

Key packages:

api/v1alpha1/             Provider API types
controller/               Reconciliation logic
cloud/                    Cloud client interface and SDK implementation
cloud/fake/               In-memory fake for tests
util/                     Shared helpers
templates/                clusterctl templates
config/                   Kubebuilder manifests

The controllers do not call the STACKIT SDK directly. They use the cloud client interface, which keeps reconciliation testable and allows fake-client envtest coverage.

Cluster API Contract

The provider implements the Cluster API infrastructure contract for:

  • StackitCluster
  • StackitClusterTemplate
  • StackitMachine
  • StackitMachineTemplate

Important contract behavior:

  • StackitCluster.status.initialization.provisioned is set when cluster infrastructure is ready.
  • StackitMachine.status.initialization.provisioned is set when the VM is provisioned.
  • StackitMachine.spec.providerID is set to the provider-compatible value.
  • Conditions include observedGeneration.
  • Paused clusters or paused resources must not trigger cloud API calls.
  • Finalizers clean up provider-owned resources before Kubernetes object removal.

StackitCluster.spec.controlPlaneEndpoint is used for the Cluster API infrastructure cluster contract when the provider manages the API server load balancer endpoint.

Controllers

StackitClusterReconciler is responsible for:

  • Reading credentials
  • Looking up the configured network
  • Managing the optional API server load balancer
  • Maintaining its target pool from the control-plane machines of the cluster
  • Publishing failure domains
  • Updating readiness and contract status
  • Cleaning up provider-managed load balancers on deletion

StackitMachineReconciler is responsible for:

  • Waiting for bootstrap data
  • Creating STACKIT servers
  • Setting provider IDs and addresses
  • Deleting servers on teardown

Reconciliation must be idempotent. Re-running the same reconcile loop should be safe and should not create duplicate cloud resources.

Cloud Client

The cloud client abstraction hides STACKIT SDK details from controllers.

It covers:

  • Server create, get, list, and delete
  • Network lookup
  • Load balancer create, list, delete, and target-pool updates
  • Provider ID helpers
  • Error classification

Controllers should handle classified errors differently:

  • transient errors should be retried
  • terminal validation errors should set clear conditions
  • not-found errors during deletion should be treated as successful cleanup where appropriate

Tests use an in-memory fake cloud client for deterministic reconciliation coverage. SDK integration tests are opt-in and require real STACKIT credentials.

STACKIT ProviderID Compatibility

The providerID format is verified against the local cloud-provider-stackit repository.

Relevant reference points:

  • pkg/ccm/instances.go: Instances.makeInstanceID returns stackit://<server-id>.
  • pkg/ccm/instances.go: instanceIDFromProviderID parses stackit://<server-id> with no project or region component.
  • pkg/ccm/instances.go: getInstance resolves nodes with GetServer(projectID, region, serverID), where project and region come from the cloud-controller-manager configuration.
  • pkg/ccm/instances_test.go: the "new providerID" table entry expects stackit://hello-server.

cluster-api-provider-stackit therefore writes:

StackitMachine.spec.providerID = stackit://<server-id>
StackitMachine.status.providerID = stackit://<server-id>

Cluster API then surfaces StackitMachine.spec.providerID to Machine.spec.providerID, and cloud-provider-stackit uses the same value for Node.spec.providerID. Project ID and region are intentionally not encoded in the providerID.

Testing

Unit and Env-Tests

Testing is split into layers:

  • unit tests for helper functions and cloud logic
  • envtest controller tests with Kubernetes API machinery
  • opt-in SDK integration tests against a real STACKIT project
  • opt-in e2e tests that create billable STACKIT resources

Default test command:

make test

This runs:

  • generated manifest checks through controller-gen
  • go fmt
  • go vet
  • envtest-backed Go tests, excluding /e2e

Real cloud tests are gated by environment variables so they do not run accidentally.

E2E-Tests

The e2e suite runs against an isolated kind management cluster. The generic target runs manager startup, metrics, webhook certificate, and webhook CA injection checks without STACKIT cloud resources:

make test-e2e

The real cloud paths create billable STACKIT servers, load balancers, public IPs, and security groups. Run them only in a dedicated STACKIT project with cleanup permissions and cost controls. The main workload scenarios have dedicated make targets:

ScenarioMake targetOpt-in flag
NodeRef/providerID and Ready Nodesmake test-e2e-workload-noderefSTACKIT_E2E_NODE_REF=true
Bastion cluster create/delete with provider-managed bastion and node SSH security group cleanupmake test-e2e-workload-bastionSTACKIT_E2E_CREATE_CLUSTER=true, STACKIT_E2E_BASTION=true
Worker scale with Ready Nodesmake test-e2e-workload-scaleSTACKIT_E2E_SCALE_WORKLOAD=true
Worker upgrade with Ready replacement Nodesmake test-e2e-workload-upgrade-workersSTACKIT_E2E_UPGRADE_WORKLOAD_WORKERS=true
Control-plane upgrade with a Ready replacement Nodemake test-e2e-workload-upgrade-control-planeSTACKIT_E2E_UPGRADE_WORKLOAD_CONTROL_PLANE=true
ClusterClass topology create/ready/deletemake test-e2e-workload-topologySTACKIT_E2E_TOPOLOGY_WORKLOAD=true

The suite also contains lower-level real cloud scenarios without dedicated make targets. Run them through go test -tags=e2e ./test/e2e or make test-e2e with the corresponding opt-in flag and a focused Ginkgo expression:

ScenarioOpt-in flagSuggested focus
Single StackitMachine VM create/delete and leak checkSTACKIT_E2E_CREATE_VMS=truecreate and delete a real STACKIT VM
1 control-plane / 1 worker infrastructure lifecycle without workload Node readinessSTACKIT_E2E_CREATE_CLUSTER=truecreate and delete a 1 control-plane / 1 worker workload Cluster
Infra-only MachineDeployment worker VM scale up/downSTACKIT_E2E_SCALE_WORKERS=truescale a worker MachineDeployment
Infra-only MachineDeployment worker VM replacement during version upgradeSTACKIT_E2E_UPGRADE_WORKERS=truereplace worker VMs during a MachineDeployment version upgrade

Common required environment:

  • STACKIT_PROJECT_ID
  • STACKIT_REGION
  • STACKIT_NETWORK_ID
  • STACKIT_IMAGE_ID
  • STACKIT_AVAILABILITY_ZONE
  • STACKIT_CREDENTIALS_SECRET_NAME
  • STACKIT_CREDENTIALS_SECRET_NAMESPACE

Useful common options:

  • KUBERNETES_VERSION, for create/scale/topology paths
  • STACKIT_E2E_UPGRADE_FROM and STACKIT_E2E_UPGRADE_TO, for upgrade paths
  • STACKIT_SSH_KEY_NAME, for node SSH keys when a scenario should attach one
  • STACKIT_BASTION_SSH_KEY_NAME, or STACKIT_SSH_KEY_NAME, when STACKIT_E2E_BASTION=true
  • STACKIT_BASTION_ALLOWED_CIDRS, defaulting to 0.0.0.0/0, for bastion SSH ingress
  • STACKIT_E2E_CNI, STACKIT_E2E_CNI_MANIFEST, and CNI-specific variables
  • STACKIT_E2E_TEST_ID, for cleanup traceability

Real Cloud Node Bootstrap

The real workload-cluster e2e path is intentionally allowed to use a generic Ubuntu cloud image during development. In that mode, the e2e fixture adds kubeadm preKubeadmCommands that install and configure containerd, kubelet, kubeadm, and kubectl at runtime before kubeadm joins the node.

Treat this runtime package installation as a development fallback. It proves the provider can pass CABPK-generated cloud-init data through STACKIT user data, but it is slower and less deterministic than a kubeadm-ready image. A production setup should use an image that already contains the expected container runtime and Kubernetes node packages for the selected Kubernetes minor.

Developing Cluster API Provider STACKIT (CAPSTK) with Tilt

This guide describes a local development loop with kind, Tilt, upstream Cluster API Tilt, and this provider. Tilt runs from a local cluster-api checkout and loads cluster-api-provider-stackit through this repository's tilt-provider.yaml.

Use this workflow when you want Tilt to rebuild and redeploy the STACKIT controller while you edit Go code. Creating workload clusters still uses real STACKIT cloud resources and may incur cost.

Prerequisites

Install the development tools listed in the development guide. For this workflow you also need:

  • tilt
  • kind
  • kubectl
  • clusterctl
  • Docker or another container runtime supported by Tilt
  • a local checkout of sigs.k8s.io/cluster-api
  • a STACKIT project, service-account JSON key, network, image, and machine type

The examples below assume both repositories are checked out next to each other:

cd /path/to/workspace
git clone https://github.com/kubernetes-sigs/cluster-api.git

If your cluster-api checkout is somewhere else, adjust paths in tilt-settings.yaml.

Create the management cluster

Run Cluster API's kind-cluster target from the upstream cluster-api checkout. This creates a kind cluster with a local registry so Tilt can use local images without pushing to a remote registry.

cd /path/to/cluster-api
make kind-cluster
kubectl config current-context

The current context should be a kind context, for example kind-capi-test.

Configure Tilt

Create tilt-settings.yaml in the upstream cluster-api checkout:

kind_cluster_name: capi-test
provider_repos:
  - ../cluster-api-provider-stackit
enable_providers:
  - kubeadm-bootstrap
  - kubeadm-control-plane
  - stackit
template_dirs:
  stackit:
    - ../cluster-api-provider-stackit/templates
kustomize_substitutions:
  CLUSTER_TOPOLOGY: "true"
  CLUSTER_RESOURCE_SET: "true"
  EXP_CLUSTER_RESOURCE_SET: "true"
  NAMESPACE: default
  KUBERNETES_VERSION: v1.35.8
  CONTROL_PLANE_MACHINE_COUNT: "1"
  WORKER_MACHINE_COUNT: "1"
  STACKIT_PROJECT_ID: "<project-uuid>"
  STACKIT_REGION: eu01
  STACKIT_NETWORK_ID: "<network-uuid>"
  STACKIT_IMAGE_ID: "<image-uuid>"
  STACKIT_MACHINE_TYPE: c2i.1
  STACKIT_AVAILABILITY_ZONE: eu01-1
  STACKIT_SECURITY_GROUP_ID: "<security-group-uuid>"
  STACKIT_SSH_KEY_NAME: ""
  STACKIT_CREDENTIALS_SECRET_NAME: stackit-credentials
  STACKIT_SERVICE_ACCOUNT_JSON_B64: "<base64-service-account-json>"
  STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGE: "ghcr.io/stackitcloud/cloud-provider-stackit/cloud-controller-manager:v1.35.7"

Notes:

  • provider_repos must point to this repository. Cluster API reads tilt-provider.yaml from that path.
  • template_dirs makes the STACKIT cluster templates available as manual Tilt actions.
  • Keep STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGE on the same Kubernetes minor as KUBERNETES_VERSION.
  • Set preload_images: false if Docker Desktop or multi-architecture image preloading fails.

Create the STACKIT credentials Secret in the management cluster namespace used by the templates. The tilt-settings.yaml values are not exported into your shell, so export at least these values before running the command manually:

export STACKIT_PROJECT_ID=<project-uuid>
export STACKIT_SERVICE_ACCOUNT_JSON_FILE=/path/to/serviceaccount.json

kubectl create secret generic stackit-credentials \
  --namespace default \
  --from-literal=project-id="${STACKIT_PROJECT_ID}" \
  --from-file=serviceaccount.json="${STACKIT_SERVICE_ACCOUNT_JSON_FILE}"

Run Tilt

Start Tilt from the upstream cluster-api checkout:

cd /path/to/cluster-api
tilt up

Tilt should deploy:

  • core Cluster API
  • kubeadm bootstrap provider
  • kubeadm control plane provider
  • STACKIT infrastructure provider

Verify the provider pod:

kubectl get pods -n cluster-api-provider-stackit-system
kubectl logs -n cluster-api-provider-stackit-system \
  deployment/cluster-api-provider-stackit-controller-manager \
  -c manager

Create a workload cluster

Workload cluster creation is a manual Tilt action from the STACKIT templates. In the Tilt UI, run the default template action for templates/cluster-template.yaml, or render and apply it from the command line:

cd /path/to/cluster-api-provider-stackit

export CLUSTER_NAME=stackit-dev
export NAMESPACE=default
export KUBERNETES_VERSION=v1.35.8
export CONTROL_PLANE_MACHINE_COUNT=1
export WORKER_MACHINE_COUNT=1
export STACKIT_PROJECT_ID=<project-uuid>
export STACKIT_REGION=eu01
export STACKIT_NETWORK_ID=<network-uuid>
export STACKIT_IMAGE_ID=<image-uuid>
export STACKIT_MACHINE_TYPE=c2i.1
export STACKIT_AVAILABILITY_ZONE=eu01-1
export STACKIT_SECURITY_GROUP_ID=<security-group-uuid>
export STACKIT_SSH_KEY_NAME=""
export STACKIT_CREDENTIALS_SECRET_NAME=stackit-credentials
export STACKIT_SERVICE_ACCOUNT_JSON_B64=<base64-service-account-json>
export STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGE=ghcr.io/stackitcloud/cloud-provider-stackit/cloud-controller-manager:v1.35.7

clusterctl generate cluster "${CLUSTER_NAME:-stackit-dev}" \
  --from templates/cluster-template.yaml \
  --target-namespace default \
  | kubectl apply -f -

Watch Cluster API and STACKIT resources:

kubectl get cluster,machine,stackitcluster,stackitmachine -n default

When the control plane is ready, install a workload CNI:

clusterctl get kubeconfig "${CLUSTER_NAME:-stackit-dev}" -n default > /tmp/stackit-dev.kubeconfig
make install-workload-cni WORKLOAD_KUBECONFIG=/tmp/stackit-dev.kubeconfig

Debug the STACKIT controller

Add a debug section for the stackit provider in tilt-settings.yaml:

debug:
  stackit:
    continue: true
    port: 30000

Restart Tilt and attach a Go debugger to 127.0.0.1:30000.

Clean up

Delete workload clusters before deleting the management cluster:

kubectl delete cluster "${CLUSTER_NAME:-stackit-dev}" -n default
kind delete cluster --name capi-test

Use the STACKIT cleanup helper only for resources labelled by the e2e tests:

make cleanup-stackit STACKIT_E2E_TEST_ID=<test-id>

Troubleshooting

  • Failed to load provider. No tilt-provider.{yaml|json} file found: verify provider_repos points to this repository, not to cluster-api.
  • default_registry is required: run make kind-cluster first so the kind cluster has the local-registry ConfigMap, or configure a writable default_registry.
  • context ... is not allowed: switch to the kind management-cluster context or add the current context to allowed_contexts.
  • StackitCluster cannot read credentials: ensure the stackit-credentials Secret exists in the same namespace as the Cluster and contains project-id plus serviceaccount.json.
  • Workload Nodes stay unready: verify the selected image matches the Kubernetes version, STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGE matches the Kubernetes minor, and a CNI was installed into the workload cluster.

Release packaging

Build the GitHub Release assets locally:

make release IMG=<registry>/cluster-api-provider-stackit:<tag>

This creates a flat dist/release/ directory containing:

  • infrastructure-components.yaml
  • metadata.yaml
  • clusterclass.yaml
  • cluster-template.yaml
  • cluster-template-bastion.yaml
  • cluster-template-development.yaml
  • cluster-template-flatcar-workers.yaml
  • cluster-template-topology.yaml
  • cilium-values.yaml
  • cloud-provider-stackit.yaml
  • install.yaml
  • checksums.txt

infrastructure-components.yaml is the Cluster API provider manifest used by clusterctl. install.yaml is the standalone installer for kubectl apply.

For local clusterctl repository development, use:

make clusterctl-release IMG=<registry>/cluster-api-provider-stackit:<tag>

Release And Distribution Implementation Plan

This plan turns the current local-only release packaging into a production-ready release flow for Cluster API Provider STACKIT (CAPSTK). The repo already has make clusterctl-release, make build-installer, metadata.yaml, clusterctl templates under templates/, and release docs that explicitly call this work unfinished.

Current State

  • Makefile can generate local clusterctl assets under dist/clusterctl/infrastructure-stackit/<version>/.
  • metadata.yaml maps release series 0.1 to the Cluster API v1beta2 contract.
  • hack/clusterctl-local.yaml points clusterctl at a local provider repository, but currently references latest while the Makefile writes a concrete version directory.
  • CI has lint, unit/envtest, and kind e2e workflows only.
  • go.mod currently uses Cluster API v1.13.2 and Kubernetes libraries v0.35.4.
  • Production docs say the release path is unfinished and needs versioned images, published clusterctl assets, upgrade guidance, support matrix, image signing, and SBOMs.

Upstream Requirements And Guidance

  • Cluster API providers must publish a provider repository with metadata.yaml, a components YAML, and optional cluster templates. For an infrastructure provider, the recommended components asset name is infrastructure-components.yaml.
  • Clusterctl local repositories use infrastructure-<provider>/<semver>/...; each version folder must contain the components, metadata, and templates for that release.
  • metadata.yaml documents release series to Cluster API contract mappings and is strictly validated by newer clusterctl versions.
  • clusterctl upgrade upgrades provider components and CRDs/controllers, but it does not upgrade workload Cluster API objects or workload Kubernetes versions.
  • Sigstore Cosign supports keyless container signing through OIDC in GitHub Actions. Sign images by digest, not by tag.
  • GitHub artifact attestations and/or Cosign attestations can publish provenance and SBOM attestations.

Useful references:

  • https://cluster-api.sigs.k8s.io/developer/providers/contracts/clusterctl
  • https://cluster-api.sigs.k8s.io/clusterctl/commands/upgrade
  • https://cluster-api.sigs.k8s.io/reference/versions
  • https://docs.sigstore.dev/cosign/signing/signing_with_containers/
  • https://github.com/actions/attest

Implementation Steps

1. Choose The Distribution Contract

Use GitHub Releases as the clusterctl provider repository and GHCR as the controller image registry.

Primary distribution:

  • clusterctl init --infrastructure stackit:<version> using release assets.
  • Controller image published to ghcr.io/<owner>/cluster-api-provider-stackit.

Secondary distribution:

  • Attach install.yaml to the same release for users who do not use clusterctl.

Keep Helm out of scope unless maintainers explicitly choose it later.

Update:

  • docs/src/development/release-packaging.md
  • docs/src/quick-start.md
  • README.md

2. Harden Clusterctl Packaging

Update make clusterctl-release so CLUSTERCTL_RELEASE_VERSION is derived from the release tag in CI instead of defaulting permanently to v0.1.0.

Required generated assets:

  • metadata.yaml
  • infrastructure-components.yaml
  • cluster-template.yaml
  • cluster-template-bastion.yaml
  • cluster-template-development.yaml
  • cluster-template-topology.yaml
  • clusterclass.yaml
  • addons/*.yaml

Fix the local repository mismatch:

  • Either generate a latest symlink/copy under dist/clusterctl/infrastructure-stackit/latest.
  • Or update hack/clusterctl-local.yaml to point at the explicit CLUSTERCTL_RELEASE_VERSION.

Add make verify-clusterctl-release:

  • Generate assets.
  • Create or reuse an isolated kind cluster.
  • Run clusterctl init against hack/clusterctl-local.yaml.
  • Verify the provider Deployment, CRDs, and metadata are accepted.

3. Publish Versioned Images

Add .github/workflows/release.yml, triggered by v* tags.

Build and publish multi-architecture images:

  • Required: linux/amd64, linux/arm64.
  • Optional: linux/s390x, linux/ppc64le only after explicit test coverage.

Publish immutable tags:

  • ghcr.io/<owner>/cluster-api-provider-stackit:<tag>
  • Optional stable aliases: <major>.<minor> and latest for stable releases only.

Render release manifests with the exact version tag or digest. Prefer digest pinning if clusterctl/template UX remains acceptable.

4. Publish Release Assets

Release workflow sequence:

  1. Checkout full history.

  2. Setup Go from go.mod.

  3. Run make test.

  4. Build and push the multi-arch image.

  5. Run:

    make clusterctl-release \
      IMG=ghcr.io/<owner>/cluster-api-provider-stackit:<tag> \
      CLUSTERCTL_RELEASE_VERSION=<tag>
    
  6. Run:

    make build-installer IMG=ghcr.io/<owner>/cluster-api-provider-stackit:<tag>
    
  7. Generate checksums for all release assets.

  8. Upload assets to the GitHub Release.

Add a PR dry-run workflow that verifies image build and asset generation without publishing anything.

5. Add Signing, SBOMs, And Provenance

Use GitHub Actions OIDC with keyless Sigstore/Cosign.

Workflow permissions needed:

permissions:
  contents: write
  packages: write
  id-token: write
  attestations: write
  artifact-metadata: write

Image signing:

  • Install Cosign.
  • Sign the pushed image by digest.
  • Publish verification instructions in docs.

SBOM:

  • Generate SPDX JSON and CycloneDX JSON using Syft or Docker BuildKit SBOM support.
  • Upload SBOM files as release assets.
  • Attach SBOM attestations to the image.

Provenance:

  • Publish build provenance through GitHub artifact attestations and/or Cosign attestations.

Docs must include examples:

cosign verify ghcr.io/<owner>/cluster-api-provider-stackit@sha256:<digest>
cosign verify-attestation ghcr.io/<owner>/cluster-api-provider-stackit@sha256:<digest>
sha256sum -c checksums.txt

6. Publish Provider Upgrade Guidance

Create docs/src/usage/provider-upgrade.md.

Cover management-cluster provider upgrades:

clusterctl upgrade plan
clusterctl upgrade apply --infrastructure stackit:<version>

State explicitly:

  • Clusterctl upgrades provider components.
  • Clusterctl does not upgrade workload Cluster, MachineDeployment, Machine, or Kubernetes versions.
  • Workload Kubernetes upgrades are handled separately through CAPI object changes.

Move or rewrite cluster-upgrade.md into the docs tree as workload Kubernetes upgrade guidance.

Initial support policy:

  • Support adjacent provider minor upgrades only.
  • Do not support skip-minor upgrades until validated by e2e tests.
  • Pre-release versions require explicit version selection for all providers.

7. Create The Support Matrix

Fill docs/src/topics/reference/versions.md.

Matrix columns:

  • CAPSTK version.
  • Cluster API version.
  • Cluster API contract.
  • Management Kubernetes versions.
  • Workload Kubernetes versions.
  • Cloud provider STACKIT image version requirement.
  • Tested CNI versions.
  • Tested STACKIT regions.
  • Tested architectures.
  • Upgrade paths validated by e2e.

Seed current values:

  • CAPSTK v0.1.x
  • Cluster API v1.13.x
  • Contract v1beta2
  • Kubernetes libraries v0.35.x
  • Workload cluster minors already documented as v1.33.x through v1.36.x until validated/updated.

Make the support matrix release-owned: every release must update or explicitly confirm it.

8. Add Release Validation Gates

Create a release checklist and encode as much as possible in CI.

Required gates:

  • make lint
  • make test
  • generated files clean after make manifests generate
  • make clusterctl-release
  • make build-installer
  • local clusterctl init from generated assets
  • release asset checksums generated
  • image pushed by immutable tag
  • image signed by digest
  • SBOM generated and attached

Billable e2e gates for release candidates:

  • make test-e2e
  • make test-e2e-workload-noderef
  • make test-e2e-workload-scale
  • make test-e2e-workload-upgrade-workers
  • make test-e2e-workload-upgrade-control-plane
  • make test-e2e-workload-topology

Run billable e2e only in a dedicated STACKIT project with cleanup permissions and cost controls.

9. Fix Contract And Documentation Mismatches Before First Production Release

Verify CRD contract metadata:

  • Docs claim CRDs carry cluster.x-k8s.io/v1beta2: v1alpha1.
  • Generated CRDs should be checked and fixed if that label is missing.
  • metadata.yaml contract must match the actual CRD contract metadata.

Add CAPI RBAC aggregation if missing:

  • Ship roles with cluster.x-k8s.io/aggregate-to-manager: "true" where required so CAPI core controllers can manage provider resources correctly.

Validate ClusterClass behavior:

  • Existing topology e2e covers a small topology cluster.
  • Add release validation for template metadata and HA topology variants before claiming broader production ClusterClass support.

Acceptance Criteria

The release story is complete when an AI agent or maintainer can:

  1. Tag a release.
  2. CI builds and publishes a versioned multi-arch image.
  3. CI signs the image by digest.
  4. CI generates SBOMs and provenance.
  5. CI publishes clusterctl assets and install.yaml to a GitHub Release.
  6. A fresh kind management cluster can run clusterctl init from the published release assets.
  7. Docs explain install, verify, upgrade, support matrix, and workload Kubernetes upgrade behavior.
  8. Release validation covers unit/envtest, local clusterctl init, and the selected billable STACKIT e2e paths.

Make Targets

Common targets:

TargetPurpose
make testRun generated checks, formatting, vet, and non-e2e tests
make manifestsRegenerate CRDs and RBAC
make generateRegenerate DeepCopy code
make installInstall CRDs into the current cluster
make uninstallRemove CRDs from the current cluster
make docker-buildBuild the controller image
make deployDeploy the controller into the current cluster
make undeployRemove the controller deployment
make build-installerGenerate dist/install.yaml
make clusterctl-releaseGenerate local clusterctl release assets
make cleanup-stackitDelete tagged e2e resources through STACKIT APIs
make install-workload-cniInstall Cilium, Calico, or a custom CNI manifest into a workload cluster
make test-e2e-workload-noderefRun billable workload NodeRef/providerID and Ready Node e2e
make test-e2e-workload-scaleRun billable workload worker scale and Ready Node e2e
make test-e2e-workload-upgrade-workersRun billable workload worker upgrade and Ready replacement Node e2e
make test-e2e-workload-upgrade-control-planeRun billable workload control-plane upgrade and Ready replacement Node e2e
make test-e2e-workload-topologyRun billable ClusterClass topology create/ready/delete e2e

Documentation targets inside docs/:

TargetPurpose
make buildBuild the mdBook
make serveServe the mdBook locally
make cleanRemove generated book output

Troubleshooting

ClusterClass is rejected

If the API server rejects ClusterClass, KubeadmControlPlaneTemplate, or Cluster.spec.topology, check the CAPI manager feature gates:

kubectl -n capi-system get deployment capi-controller-manager \
  -o jsonpath='{.spec.template.spec.containers[0].args}'

kubectl -n capi-kubeadm-control-plane-system \
  get deployment capi-kubeadm-control-plane-controller-manager \
  -o jsonpath='{.spec.template.spec.containers[0].args}'

Both must include ClusterTopology=true.

Machine waits for bootstrap data

Check the owning Machine and bootstrap Secret:

kubectl describe machine "${MACHINE_NAME}" -n "${NAMESPACE}"
kubectl get secret -n "${NAMESPACE}"

StackitMachine intentionally does not contain user data directly. Bootstrap data comes from the CAPI bootstrap Secret.

STACKIT CLI might also be valueable to check server logs during boot: stackit server log <server-id>

Cloud resources remain after a failed e2e run

Run direct cleanup by e2e tags:

make cleanup-stackit

Then verify no tagged STACKIT VMs or load balancers remain.

CRD Reference

tbd