Kubernetes Cluster API Provider STACKIT (CAPSTK)
Kubernetes-native declarative infrastructure for STACKIT.
⚠️ WARNING ⚠️
cluster-api-provider-stackitis still in active development.Use at your own risk.
What is the Cluster API Provider STACKIT
The Cluster API brings declarative, Kubernetes-style APIs to cluster creation, configuration and management.
The API itself is shared across multiple cloud providers allowing for true STACKIT hybrid deployments of Kubernetes.
Cluster API Provider STACKIT is abbreviated as CAPSTK.
Documentation
Please see our book for in-depth documentation.
Use make serve-book to serve the book locally from this repository.
Install with clusterctl
Create clusterctl.yaml with the published release URL. Replace <version>
with the release tag you want to install.
providers:
- name: stackit
url: https://github.com/stackitcloud/cluster-api-provider-stackit/releases/download/v<version>/infrastructure-components.yaml
type: InfrastructureProvider
Install the provider:
clusterctl init --config clusterctl.yaml --infrastructure stackit:v<version>
Launching a Kubernetes cluster on STACKIT
Check out the Quick Start for launching a cluster on STACKIT.
Features
- Native Kubernetes manifests and API
- Doesn't use SSH for bootstrapping nodes.
- Installs only the minimal components to bootstrap a control plane and workers.
- Supports control planes and worker nodes on STACKIT VM instances.
- Manages the bootstrapping of security groups and vm instances (networks are excluded for now).
- Optional Bastion hosts for easier access of control plane or worker nodes
- Tested Kubernetes Lifecycle (Scaling, Kubernetes Upgrades), see E2E-Tests
- ClusterClass Topology
- Support various Linux Distributions (tested with Ubuntu and Flatcar)
- Release distribution via OCI Images and Manifests
-
Manage the bootstrapping of networks, security groups and vm instances.
- Deploys Kubernetes control planes into private subnets with a separate bastion server.
- SKE support
Compatibility with Cluster API and Kubernetes Versions
This provider's versions are compatible with the following versions of Cluster API and support all Kubernetes versions that is supported by its compatible Cluster API version:
| Cluster API v1alpha4 (v0.4) | Cluster API v1beta1 | Cluster API v1beta2 | |
|---|---|---|---|
CAPSTK v1alpha1 (main) | x | x | ✓ |
This provider implements the v1beta2 contract, as declared in
metadata.yaml, and is built against sigs.k8s.io/cluster-api
v1.13.2.
(See Kubernetes support matrix of Cluster API versions).
Getting involved and contributing
Interested in contributing to cluster-api-provider-stackit? We welcome your ideas, contributions, and help. Feel free to reach out to the maintainers to learn how to get involved.
You do not need official write permissions to make an impact. We encourage active community members to take ownership, drive improvements, and help move the project forward.
We are also happy to welcome new maintainers over time. Get involved and show us what you can do!
Quick start
This guide creates a Kind management cluster and installs the published STACKIT infrastructure provider. You do not need to clone this repository or build an image.
Prerequisites
Install kind, kubectl, clusterctl, curl, and base64. You also need a
STACKIT project, a service-account JSON key, an existing network, image, and
machine type. See IAM permissions for the
required service-account permissions.
Set the release version to install:
export CAPSTK_VERSION=v0.1.0-alpha.5
Create clusterctl.yaml for that release:
cat > clusterctl.yaml <<EOF
providers:
- name: stackit
url: https://github.com/stackitcloud/cluster-api-provider-stackit/releases/download/${CAPSTK_VERSION}/infrastructure-components.yaml
type: InfrastructureProvider
EOF
Create a management cluster
Enterprise proxies like Zscaler
If your network uses a TLS-intercepting proxy such as Zscaler, the Kind node
must trust the proxy's root certificate. The host's certificate store does not
automatically apply inside the Kind node. Create a local kind-config.yaml
that mounts the host CA bundle into the node:
cat > kind-config.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraMounts:
- hostPath: /etc/ssl/certs/ca-certificates.crt
containerPath: /etc/ssl/certs/ca-certificates.crt
readOnly: true
EOF
kind create cluster --name capi-stackit --config kind-config.yaml
Without such a proxy, create the cluster without the configuration file:
kind create cluster --name capi-stackit
Select the management-cluster context and install the provider:
kubectl config use-context kind-capi-stackit
clusterctl init \
--config clusterctl.yaml \
--infrastructure stackit:"${CAPSTK_VERSION}"
Wait for the provider controller:
kubectl rollout status \
--namespace cluster-api-provider-stackit-system \
deployment/cluster-api-provider-stackit-controller-manager
Set credentials and cluster settings
Save the service-account JSON key anywhere on your machine, then set its path and the IDs for the STACKIT resources the workload cluster will use:
export STACKIT_PROJECT_ID=<project-uuid>
export STACKIT_REGION=eu01
export STACKIT_NETWORK_ID=<network-uuid>
export STACKIT_IMAGE_ID=<image-uuid>
export STACKIT_MACHINE_TYPE=c2i.4
export STACKIT_SERVICE_ACCOUNT_JSON_FILE=/path/to/service-account.json
export STACKIT_SERVICE_ACCOUNT_JSON_B64="$(base64 < "${STACKIT_SERVICE_ACCOUNT_JSON_FILE}" | tr -d '\n')"
kubectl create secret generic stackit-credentials \
--namespace default \
--from-literal=project-id="${STACKIT_PROJECT_ID}" \
--from-file=serviceaccount.json="${STACKIT_SERVICE_ACCOUNT_JSON_FILE}"
You can retrieve a valid image ID using the STACKIT CLI:
stackit image list --project-id "${STACKIT_PROJECT_ID}" --output-format json --label-selector linux,prod | jq -r '.[] | select(.name == "Ubuntu 24.04") | .id'
Create a workload cluster
You can use the cluster template directly from the repository's templates
directory (templates/cluster-template.yaml) if you cloned the repository, or
download the template that matches the provider release:
curl --fail --location --remote-name \
"https://github.com/stackitcloud/cluster-api-provider-stackit/releases/download/${CAPSTK_VERSION}/cluster-template.yaml"
Set the cluster values and render the template (use
--from templates/cluster-template.yaml if working from a repository clone):
export CLUSTER_NAME=stackit-workload
export NAMESPACE=default
export KUBERNETES_VERSION=v1.35.8
export KUBERNETES_APT_REPOSITORY_MINOR=v1.35
export CONTROL_PLANE_MACHINE_COUNT=1
export WORKER_MACHINE_COUNT=1
export STACKIT_CREDENTIALS_SECRET_NAME=stackit-credentials
export STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGE=ghcr.io/stackitcloud/cloud-provider-stackit/cloud-controller-manager:v1.35.7
clusterctl generate cluster "${CLUSTER_NAME}" \
--from cluster-template.yaml \
--target-namespace "${NAMESPACE}" \
> cluster.yaml
kubectl apply -f cluster.yaml
Watch the provider create the cluster:
kubectl get cluster,machine,stackitcluster,stackitmachine --namespace "${NAMESPACE}"
When the workload API is available, retrieve the workload kubeconfig to install a CNI:
clusterctl get kubeconfig "${CLUSTER_NAME}" \
--namespace "${NAMESPACE}" \
> "${CLUSTER_NAME}".kubeconfig
Workload nodes remain in NotReady status until a CNI is installed. We provide
a preconfigured Cilium setup at templates/addons/cilium-values.yaml in the
templates addons directory, and it is also published for each release:
curl --fail --location --remote-name \
"https://github.com/stackitcloud/cluster-api-provider-stackit/releases/download/${CAPSTK_VERSION}/cilium-values.yaml"
Install Cilium using the Cilium CLI or Helm with these values (or use
templates/addons/cilium-values.yaml directly from a local checkout):
cilium install \
--kubeconfig "${CLUSTER_NAME}.kubeconfig" \
--values cilium-values.yaml
If you are working from a local checkout, you can also use the development
target in the Makefile backed by hack/install-workload-cni.sh:
make install-workload-cni \
WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig"
See Workload CNI and Workload Addons for more details on CNI options and verification.
Cleanup
To fully delete the workload cluster and its cloud infrastructure:
kubectl delete -f cluster.yaml
Deletion should take at most 5 minutes. Verify that all resources are gone:
kubectl get cluster,machine,stackitcluster,stackitmachine --namespace "${NAMESPACE}"
Troubleshooting stuck deletion
If resources persist after several minutes:
-
Check the status and conditions of persisting resources:
kubectl describe cluster,machine,stackitcluster,stackitmachine --namespace "${NAMESPACE}" -
Inspect the controller logs for errors:
# Core Cluster API controller kubectl logs -n capi-system deployment/capi-controller-manager -c manager --tail=100 # STACKIT provider controller kubectl logs -n cluster-api-provider-stackit-system deployment/cluster-api-provider-stackit-controller-manager --tail=100 -
If a
StackitMachinecannot finish deletion and blocks its parent resources, you can remove its finalizer:kubectl patch stackitmachine <stackitmachine-name> -n "${NAMESPACE}" \ --type=merge -p '{"metadata":{"finalizers":null}}'Removing the finalizer unblocks Kubernetes deletion for that resource and its parents. You may then have to manually delete the actual VM in the STACKIT portal or through the STACKIT API.
Use the development guide when you want to build and run the provider from a local checkout.
Usage
Use these chapters to render, create, operate, and delete STACKIT workload clusters.
- Classic Cluster Template creates explicit CAPI resources for each workload cluster.
- ClusterClass creates workload clusters from a reusable CAPI topology blueprint.
- Workload Addons explains the tested cloud-provider-stackit and CNI installation flow.
- Workload CNI covers installing a CNI after the workload API is reachable.
- Upgrade and Cleanup cover common lifecycle operations.
Cluster Templates
CAPSTK provides clusterctl templates under templates/ for various workload cluster configurations:
templates/cluster-template.yaml: Default template using a pre-existing (Bring-Your-Own / BYO) STACKIT network.templates/cluster-template-managed-network.yaml: Managed network template where CAPSTK automatically creates and manages a dedicated/24STACKIT network.templates/cluster-template-bastion.yaml: Enables an optional provider-managed SSH bastion host.templates/cluster-template-flatcar-workers.yaml: Mixed OS cluster with Flatcar Linux worker nodes.templates/cluster-template-topology.yaml: ClusterClass topology-based cluster.
Resource Manifest Breakdown
The classic templates (cluster-template.yaml and cluster-template-managed-network.yaml) render:
ClusterStackitClusterKubeadmControlPlane- control-plane
StackitMachineTemplate MachineDeployment- worker
StackitMachineTemplate - worker
KubeadmConfigTemplate MachineHealthCheckfor control-plane and worker MachinesClusterResourceSetforcloud-provider-stackit
Deploying with a Bring-Your-Own (BYO) Network
In the BYO network flow, STACKIT_NETWORK_ID is known upfront. The template embeds both the ClusterResourceSet and the resource-set Secret with the pre-rendered cloud-provider-stackit.yaml.
Render a cluster:
clusterctl generate cluster "${CLUSTER_NAME}" \
--from templates/cluster-template.yaml \
--kubernetes-version "${KUBERNETES_VERSION}" \
--control-plane-machine-count "${CONTROL_PLANE_MACHINE_COUNT}" \
--worker-machine-count "${WORKER_MACHINE_COUNT}" \
> "${CLUSTER_NAME}.yaml"
Apply the rendered manifest:
kubectl apply -f "${CLUSTER_NAME}.yaml"
Deploying with a Managed Network
When using templates/cluster-template-managed-network.yaml, STACKIT_NETWORK_ID is omitted. CAPSTK automatically provisions a dedicated /24 network during StackitCluster reconciliation and publishes its ID to StackitCluster.status.network.id.
Render and apply the cluster:
clusterctl generate cluster "${CLUSTER_NAME}" \
--from templates/cluster-template-managed-network.yaml \
--kubernetes-version "${KUBERNETES_VERSION}" \
--control-plane-machine-count "${CONTROL_PLANE_MACHINE_COUNT}" \
--worker-machine-count "${WORKER_MACHINE_COUNT}" \
| kubectl apply -f -
Because the network ID is generated at runtime, the template creates the ClusterResourceSet but omits the Secret. Once CAPSTK provisions the network, you retrieve the network ID and supply the cloud-provider-stackit Secret or apply the addon directly. See Workload Addons for the full procedure.
External Cloud Provider Wiring
The templates configure kubeadm for an external cloud provider by setting
cloud-provider=external on kubelet and controller-manager.
The management cluster must have the ClusterResourceSet feature enabled before applying templates that use it. For local validation, hack/clusterctl-local.yaml sets CLUSTER_RESOURCE_SET=true.
Set STACKIT_SERVICE_ACCOUNT_JSON_B64 to a single-line base64 encoding of the
STACKIT service account JSON. Set STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGE to a
cloud-provider-stackit image whose minor version matches
KUBERNETES_VERSION; supported workload cluster minors are v1.33.x through
v1.36.x. Use hack/validate-stackit-versions.sh before rendering to catch
unsupported or mismatched versions.
The templates install or prepare the cloud controller manager, not a CNI. After the workload API is reachable, install a CNI that matches the configured pod/service CIDRs and network policy expectations before expecting Nodes to become Ready. See Workload CNI and Workload Addons.
Machine Health Checks and Remediation
The MachineHealthChecks remediate Machines whose Node does not register within
10 minutes or reports Ready=False or Ready=Unknown for 10 minutes. The
MachineDeployment limits worker remediation to 2 Machines at a time via
spec.remediation.maxInFlight.
Worker remediation only runs while at most 40% of the workers are unhealthy
(triggerIf.unhealthyLessThanOrEqualTo, rounded down). A MachineDeployment
with fewer than 3 workers is therefore never remediated; use
WORKER_MACHINE_COUNT ≥ 3 for automatic replacement. A network outage longer
than 10 minutes can still replace healthy workers once the first of them
recovers and the unhealthy share drops below the limit.
Control-plane and worker Machines use nodeDrainTimeoutSeconds: 900, so a
Node whose Pods cannot terminate does not block deletion forever. This applies
to every Machine deletion, including upgrades and scale-down: Pods still
protected by a PodDisruptionBudget after 15 minutes are not waited for.
ClusterClass
ClusterClass is Cluster API's reusable cluster blueprint mechanism. Instead of
rendering every infrastructure, control plane, bootstrap, and worker resource
directly for each workload cluster, you define a ClusterClass once and create
workload clusters that reference it through Cluster.spec.topology.
For this provider, ClusterClass support is provided by:
templates/clusterclass.yamltemplates/cluster-template-topology.yaml
templates/clusterclass.yaml contains the reusable class and the STACKIT
infrastructure templates. templates/cluster-template-topology.yaml contains a
small topology Cluster that selects the class and passes values as topology
variables.
Why use ClusterClass
Use ClusterClass when you want CAPI to own the generated cluster topology. CAPI
creates and reconciles the underlying StackitCluster, KubeadmControlPlane,
MachineDeployment, bootstrap templates, and StackitMachine infrastructure
objects from the class.
This gives you:
- one reusable cluster shape for multiple workload clusters
- smaller workload cluster manifests
- Kubernetes version, replica count, and infrastructure settings expressed as topology variables
- standard CAPI topology behavior for scaling and upgrades
- generated infrastructure resources with owner references, labels, and annotations managed by CAPI core
You do not need ClusterClass for a simple explicit cluster manifest. The classic template in Classic Cluster Template is still valid. ClusterClass is the preferred path when you want repeatable topology management and fleet-style cluster creation.
What the STACKIT ClusterClass configures
The current ClusterClass wires topology variables into STACKIT templates for:
- Kubernetes version
- control-plane replica count
- worker replica count
- machine type
- image ID
- region
- network ID
- credentials Secret name
- additional STACKIT cloud labels
- development fallback
preKubeadmCommands
The ClusterClass uses these provider templates:
StackitClusterTemplatefor the generatedStackitCluster- one
StackitMachineTemplatefor control-plane machines - one
StackitMachineTemplatefor worker machines
The templates also include spec.template.metadata.labels and
spec.template.metadata.annotations. CAPI topology copies this metadata onto
the generated StackitCluster and StackitMachine objects. This is useful for
policy, ownership, automation, and observability labels that should be present
on generated infrastructure objects.
The class also defines healthCheck and deletion for the control plane and
the default-worker class with the same checks, remediation limits and drain
timeouts as the classic template's MachineHealthChecks (see
Classic Cluster Template). Disable the health checks per
cluster with spec.topology.controlPlane.healthCheck.enabled: false or the
equivalent field on a MachineDeployment topology.
Prerequisites
The management cluster must run CAPI core and kubeadm-control-plane with
ClusterTopology=true. Otherwise the admission webhooks reject ClusterClass,
KubeadmControlPlaneTemplate, and Cluster.spec.topology.
For local clusterctl usage, hack/clusterctl-local.yaml sets
CLUSTER_TOPOLOGY: "true".
The topology cluster template also includes a cloud-provider-stackit
ClusterResourceSet addon and a workload-cluster Secret for the cloud
controller manager. The management cluster must have the ClusterResourceSet
feature enabled as well. For local validation, hack/clusterctl-local.yaml sets
CLUSTER_RESOURCE_SET: "true".
Before rendering, prepare the same STACKIT variables used by the classic template:
export CLUSTER_NAME=stackit-workload
export NAMESPACE=default
export KUBERNETES_VERSION=v1.35.8
export KUBERNETES_APT_REPOSITORY_MINOR=v1.35
export CONTROL_PLANE_MACHINE_COUNT=1
export WORKER_MACHINE_COUNT=1
export STACKIT_PROJECT_ID=<project-uuid>
export STACKIT_REGION=eu01
export STACKIT_NETWORK_ID=<network-uuid>
export STACKIT_IMAGE_ID=<image-uuid>
export STACKIT_MACHINE_TYPE=c2i.2
export STACKIT_CREDENTIALS_SECRET_NAME=stackit-credentials
export STACKIT_SERVICE_ACCOUNT_JSON_FILE=./.stackit/cluster-api-provider-stackit-serviceaccount.json
export STACKIT_SERVICE_ACCOUNT_JSON_B64="$(base64 < "${STACKIT_SERVICE_ACCOUNT_JSON_FILE}" | tr -d '\n')"
export STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGE=ghcr.io/stackitcloud/cloud-provider-stackit/cloud-controller-manager:v1.35.7
STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGE must use a minor version matching
KUBERNETES_VERSION.
Apply the ClusterClass
Apply the reusable class and its templates into the namespace that should own the class:
kubectl apply -n "${NAMESPACE}" -f templates/clusterclass.yaml
If the class lives in a different namespace than the workload cluster, set:
export CLUSTER_CLASS_NAMESPACE=<clusterclass-namespace>
Create a topology cluster
Render and apply a topology Cluster:
clusterctl generate cluster "${CLUSTER_NAME}" \
--from templates/cluster-template-topology.yaml \
--target-namespace "${NAMESPACE}" \
--kubernetes-version "${KUBERNETES_VERSION}" \
--control-plane-machine-count "${CONTROL_PLANE_MACHINE_COUNT}" \
--worker-machine-count "${WORKER_MACHINE_COUNT}" \
> "${CLUSTER_NAME}-topology.yaml"
kubectl apply -f "${CLUSTER_NAME}-topology.yaml"
CAPI topology then creates the generated infrastructure and control plane objects. Inspect them with:
kubectl get cluster,machine,stackitcluster,stackitmachine -n "${NAMESPACE}"
Check propagated template metadata:
kubectl get stackitcluster -n "${NAMESPACE}" \
-l "cluster.x-k8s.io/cluster-name=${CLUSTER_NAME}" \
-o go-template='{{range .items}}{{index .metadata.labels "cluster-api-provider-stackit/template-metadata"}}{{"\n"}}{{end}}'
kubectl get stackitmachine -n "${NAMESPACE}" \
-l "cluster.x-k8s.io/cluster-name=${CLUSTER_NAME}" \
-o go-template='{{range .items}}{{.metadata.name}} {{index .metadata.labels "cluster-api-provider-stackit/template-metadata"}}{{"\n"}}{{end}}'
Workload cluster addons
The topology cluster template installs the cloud controller manager through a
ClusterResourceSet. It does not install a CNI.
After the workload API is reachable, install a CNI that matches the configured pod/service CIDRs and network policy expectations before expecting Nodes to become Ready. For a reproducible development path, use the helper documented in Workload CNI.
Delete a topology cluster
Delete the topology Cluster; CAPI deletes the generated objects and this
provider deletes the STACKIT resources:
kubectl delete cluster "${CLUSTER_NAME}" -n "${NAMESPACE}"
After deletion, verify generated resources are gone:
kubectl get machine,stackitcluster,stackitmachine \
-n "${NAMESPACE}" \
-l "cluster.x-k8s.io/cluster-name=${CLUSTER_NAME}"
E2E validation
Run the topology workload e2e path independently only when billable STACKIT validation is intended:
env STACKIT_E2E_TOPOLOGY_WORKLOAD=true \
KUBERNETES_VERSION=v1.35.8 \
STACKIT_E2E_CNI=cilium \
go test -timeout=90m -tags=e2e ./test/e2e -v -ginkgo.v \
--ginkgo.focus='topology.*workload' --ginkgo.timeout=90m
Equivalent make target:
make test-e2e-workload-topology
The topology e2e creates a real STACKIT workload cluster, waits for Ready Machines and Nodes, verifies providerID alignment, verifies template metadata propagation, deletes the Cluster, and checks that tagged STACKIT servers and API server load balancers are cleaned up.
Workload Addons
This provider supports a tested addon flow for unmanaged STACKIT workload clusters:
cloud-provider-stackit(the Cloud Controller Manager or CCM) runs in the workload cluster to manage node lifecycles, provider IDs, and STACKIT Service Load Balancers.- A CNI (such as Cilium or Calico) is installed after the workload API server is reachable.
- The default tested CNI path uses Cilium with preconfigured values in
templates/addons/cilium-values.yaml.
This split is intentional:
- The cloud provider must run early so Kubernetes Nodes get STACKIT provider IDs
and the external cloud-provider taint (
node.cloudprovider.kubernetes.io/uninitialized) is cleared. - The CNI is a cluster operator choice because routing, network policy, MTU, IPAM, and upgrade strategy are environment-specific.
Cloud Controller Manager (cloud-provider-stackit)
The STACKIT Cloud Controller Manager runs inside the workload cluster in the kube-system namespace.
Its manifest is defined in templates/addons/cloud-provider-stackit.yaml and sets up:
- ServiceAccount, RBAC roles, and ClusterRoleBindings for
stackit-cloud-controller-manager stackit-cloud-configConfigMap containingcloud.yamlwith the STACKIT project ID, region, and load balancer network IDstackit-cloud-secretSecret containing the service account JSON key (sa_key.json)- Deployment running
stackit-cloud-controller-managerwith controllers:cloud-node-controller: initializes Node objects and setsspec.providerIDtostackit://<server-id>cloud-node-lifecycle-controller: cleans up Node objects when servers are deleted in STACKITservice-lb-controller: provisions and configures STACKIT Network Load Balancers for KubernetesServiceobjects withtype: LoadBalancer
Configuration Variables
The manifest templates/addons/cloud-provider-stackit.yaml expects these environment variables:
| Variable | Description |
|---|---|
STACKIT_PROJECT_ID | Target STACKIT project UUID |
STACKIT_REGION | Target STACKIT region (e.g. eu01) |
STACKIT_NETWORK_ID | STACKIT network UUID for Service load balancers |
STACKIT_SERVICE_ACCOUNT_JSON_B64 | Base64-encoded STACKIT service account JSON key |
STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGE | CCM image matching the workload Kubernetes minor version (e.g. ghcr.io/stackitcloud/cloud-provider-stackit/cloud-controller-manager:v1.35.7) |
CLUSTER_NAME | Name of the workload cluster |
Set your credentials and CCM image:
export STACKIT_SERVICE_ACCOUNT_JSON_FILE=./.stackit/cluster-api-provider-stackit-serviceaccount.json
export STACKIT_SERVICE_ACCOUNT_JSON_B64="$(base64 < "${STACKIT_SERVICE_ACCOUNT_JSON_FILE}" | tr -d '\n')"
export STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGE="ghcr.io/stackitcloud/cloud-provider-stackit/cloud-controller-manager:v1.35.7"
Important: The
cloud-provider-stackitimage minor version must match the Kubernetes minor version of the workload cluster (e.g. v1.35.x CCM for v1.35.x Kubernetes). Usehack/validate-stackit-versions.shbefore deployment to verify compatibility.
Deploying CCM with BYO Networks
When using templates/cluster-template.yaml or templates/cluster-template-topology.yaml, you provide STACKIT_NETWORK_ID up front. The cluster template generates:
- A Cluster API
ClusterResourceSetnamed${CLUSTER_NAME}-cloud-provider-stackit - A companion
Secretcontaining the pre-renderedcloud-provider-stackit.yaml
As soon as the workload cluster API server is reachable, Cluster API automatically applies the CCM manifest into the workload cluster. No manual intervention is needed.
Managed Networks and cloud-provider-stackit
When deploying a cluster with a managed network using templates/cluster-template-managed-network.yaml, spec.network.id is omitted. CAPSTK creates a dedicated /24 network in STACKIT during cluster reconciliation.
Because the network ID does not exist when clusterctl generate is executed, the template creates the ClusterResourceSet but does not include the Secret. The STACKIT CCM requires this network ID in stackit-cloud-config to manage Service load balancers.
Step 1: Retrieve the Provisioned Network ID
Wait for CAPSTK to create the network and populate StackitCluster.status.network.id:
export STACKIT_NETWORK_ID=$(kubectl get stackitcluster "${CLUSTER_NAME}" \
--namespace "${NAMESPACE}" \
-o jsonpath='{.status.network.id}')
echo "Managed Network ID: ${STACKIT_NETWORK_ID}"
Step 2: Deploy CCM
You can provide the CCM manifest using either of two methods:
Method A: Via ClusterResourceSet Secret (Recommended)
Since cluster-template-managed-network.yaml already defined a ClusterResourceSet looking for a Secret named ${CLUSTER_NAME}-cloud-provider-stackit, create that Secret in the management cluster. The ClusterResourceSet controller will automatically push it to the workload cluster:
# Render the addon manifest using envsubst and create the CRS Secret
kubectl create secret generic "${CLUSTER_NAME}-cloud-provider-stackit" \
--namespace "${NAMESPACE}" \
--type=addons.cluster.x-k8s.io/resource-set \
--from-literal=cloud-provider-stackit.yaml="$(envsubst < templates/addons/cloud-provider-stackit.yaml)"
To verify the ClusterResourceSet applied the resources:
kubectl get clusterresourcesetbinding -n "${NAMESPACE}"
Method B: Direct Apply to the Workload Cluster
Alternatively, retrieve the workload cluster kubeconfig and apply the rendered addon directly:
clusterctl get kubeconfig "${CLUSTER_NAME}" \
--namespace "${NAMESPACE}" \
> "${CLUSTER_NAME}.kubeconfig"
envsubst < templates/addons/cloud-provider-stackit.yaml \
| kubectl --kubeconfig "${CLUSTER_NAME}.kubeconfig" apply -f -
CNI Installation
After CCM is deployed and the workload API server is reachable, retrieve the kubeconfig if you have not already:
clusterctl get kubeconfig "${CLUSTER_NAME}" \
--namespace "${NAMESPACE}" \
> "${CLUSTER_NAME}.kubeconfig"
Nodes remain in NotReady status until a CNI is installed.
Installing Cilium (Default Tested Path)
Install Cilium using the checked-in values matching the cluster template's pod CIDR:
make install-workload-cni \
WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig"
Alternatively, install with the Cilium CLI or Helm using templates/addons/cilium-values.yaml:
cilium install \
--kubeconfig "${CLUSTER_NAME}.kubeconfig" \
--values templates/addons/cilium-values.yaml
Installing Calico
To install Calico instead:
make install-workload-cni \
WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig" \
STACKIT_WORKLOAD_CNI=calico
Installing a Custom CNI
To apply an arbitrary CNI manifest:
make install-workload-cni \
WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig" \
CNI_MANIFEST=./my-cni.yaml
Verification
Verify that the cloud provider Deployment rolled out:
clusterctl get kubeconfig "${CLUSTER_NAME}" \
--namespace "${NAMESPACE}" \
> "${CLUSTER_NAME}.kubeconfig"
Install the default tested Cilium CNI:
make install-workload-cni \
WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig"
The helper waits for the Cilium operator, Cilium DaemonSet, and Cilium Envoy DaemonSet when present.
To install Calico instead:
make install-workload-cni \
WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig" \
STACKIT_WORKLOAD_CNI=calico
To apply a custom CNI manifest:
make install-workload-cni \
WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig" \
CNI_MANIFEST=./my-cni.yaml
Custom manifests are applied as-is. The helper cannot know which workload resources prove that an arbitrary CNI is healthy, so custom CNI users must wait for their own rollout resources.
Verification
Verify that the cloud provider Deployment rolled out:
kubectl --kubeconfig "${CLUSTER_NAME}.kubeconfig" \
-n kube-system rollout status deployment/stackit-cloud-controller-manager \
--timeout=5m
Verify Cilium rollout for the default path:
kubectl --kubeconfig "${CLUSTER_NAME}.kubeconfig" \
-n kube-system rollout status deployment/cilium-operator --timeout=10m
kubectl --kubeconfig "${CLUSTER_NAME}.kubeconfig" \
-n kube-system rollout status daemonset/cilium --timeout=10m
kubectl --kubeconfig "${CLUSTER_NAME}.kubeconfig" \
-n kube-system rollout status daemonset/cilium-envoy --timeout=10m
Verify Nodes are Ready and have STACKIT provider IDs:
kubectl --kubeconfig "${CLUSTER_NAME}.kubeconfig" get nodes
kubectl --kubeconfig "${CLUSTER_NAME}.kubeconfig" \
get nodes \
-o custom-columns=NAME:.metadata.name,PROVIDER_ID:.spec.providerID
Verify the external cloud-provider taint is gone:
kubectl --kubeconfig "${CLUSTER_NAME}.kubeconfig" \
get nodes -o json \
| jq -e '[.items[].spec.taints[]? | select(.key == "node.cloudprovider.kubernetes.io/uninitialized")] | length == 0'
Verify CAPI Machines and workload Nodes agree on STACKIT provider IDs:
kubectl -n "${NAMESPACE}" \
get machines,stackitmachines \
-l "cluster.x-k8s.io/cluster-name=${CLUSTER_NAME}" \
-o custom-columns=KIND:.kind,NAME:.metadata.name,PROVIDER_ID:.spec.providerID,NODE:.status.nodeRef.name
kubectl --kubeconfig "${CLUSTER_NAME}.kubeconfig" \
get nodes \
-o custom-columns=NAME:.metadata.name,PROVIDER_ID:.spec.providerID
The e2e workload tests validate this flow by waiting for the embedded
cloud-provider-stackit rollout, installing CNI, waiting for Ready Nodes,
checking the external cloud-provider taint is removed, and verifying CAPI
Machine, StackitMachine, and Node provider IDs align.
Production Addon Management
ClusterResourceSet is useful for static bootstrap addons such as this tested
cloud-provider installation, but it is not a full addon lifecycle manager. For
production CNI management, prefer Helm, GitOps, or a Cluster API addon provider
that handles upgrades, rollback, and drift reconciliation.
Workload CNI
The default cluster template provisions STACKIT infrastructure and installs
cloud-provider-stackit, but it intentionally does not install a CNI. This
keeps CNI choice with the cluster operator, where network policy, routing, MTU,
IPAM, and upgrade strategy belong.
Nodes will not become fully Ready until a CNI is installed. The templates ship
MachineHealthChecks that replace Machines whose Node stays Ready=False for 10
minutes, so install the CNI within that window after the Nodes join.
For the complete tested workload addon flow, including the embedded
cloud-provider-stackit ClusterResourceSet and verification commands, see
Workload Addons.
First, retrieve the workload-cluster kubeconfig:
clusterctl get kubeconfig "${CLUSTER_NAME}" \
--namespace "${NAMESPACE}" \
> "${CLUSTER_NAME}.kubeconfig"
Install Cilium using the checked-in values:
make install-workload-cni \
WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig"
The Cilium values live in templates/addons/cilium-values.yaml and match the
default pod CIDR from templates/cluster-template.yaml.
To install Calico instead:
make install-workload-cni \
WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig" \
STACKIT_WORKLOAD_CNI=calico
To apply a fully custom CNI manifest:
make install-workload-cni \
WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig" \
CNI_MANIFEST=./my-cni.yaml
Custom manifests are applied as-is. The helper cannot infer which resources prove that an arbitrary CNI is healthy, so custom CNI users must wait for their own rollout resources.
For production clusters, prefer managing the CNI with Helm, GitOps, or a
Cluster API addon provider. ClusterResourceSet is useful for simple static
resources, but it is not a full addon lifecycle manager for upgrades, rollback,
or drift reconciliation.
Cluster Upgrade
Create the workload cluster with the source version:
export KUBERNETES_VERSION=v1.33.12
clusterctl generate cluster "${CLUSTER_NAME}" \
--from templates/cluster-template.yaml \
> "${CLUSTER_NAME}.yaml"
kubectl apply -f "${CLUSTER_NAME}.yaml"
Upgrade the control plane first:
kubectl patch kubeadmcontrolplane "${CLUSTER_NAME}-control-plane" \
-n "${NAMESPACE}" \
--type merge \
-p '{"spec":{"version":"v1.34.8"}}'
Then upgrade the workers:
kubectl patch machinedeployment "${CLUSTER_NAME}-md-0" \
-n "${NAMESPACE}" \
--type merge \
-p '{"spec":{"template":{"spec":{"version":"v1.34.8"}}}}'
Watch rollout state:
kubectl get kubeadmcontrolplane,machinedeployment,machineset,machine \
-n "${NAMESPACE}" \
-l "cluster.x-k8s.io/cluster-name=${CLUSTER_NAME}" \
--watch
The current tested scope verifies MachineDeployment replacement behavior and VM cleanup. Full KubeadmControlPlane and Node readiness verification depends on completing workload-cluster bootstrap and cloud-provider integration.
See the root cluster-upgrade.md tutorial for the detailed step-by-step flow.
Cleanup
Delete a workload cluster through Cluster API:
kubectl delete cluster "${CLUSTER_NAME}" -n "${NAMESPACE}"
Then check Kubernetes resources:
kubectl get machine,stackitmachine,stackitcluster \
-n "${NAMESPACE}" \
-l "cluster.x-k8s.io/cluster-name=${CLUSTER_NAME}"
Real e2e resources are labeled so they can be cleaned up directly through the STACKIT API if Kubernetes cleanup fails:
cluster-api-provider-stackit/e2e=truecluster-api-provider-stackit/test-id=<unique-id>cluster.x-k8s.io/cluster-name=<cluster-name>cluster.x-k8s.io/cluster-namespace=<namespace>
Run direct cloud cleanup for tagged e2e resources:
make cleanup-stackit
The cleanup path must not depend on Kubernetes objects. It should find and delete resources directly by STACKIT labels.
Topics
In-depth technical guides and configuration references for Cluster API Provider STACKIT:
- STACKIT Cloud Resources: Networks (Managed and BYO), Security Groups, SSH keys, and node images.
- OS Images: Configuring Ubuntu and Flatcar Linux machine images.
- Accessing VM Instances: Provider-managed SSH bastion host and node SSH access.
- HA, Availability Zones & Failure Domains: Dynamic availability zone discovery and machine placement.
- IAM Permissions Used: Least-privilege service account roles and OpenTofu setup.
STACKIT Cloud Resources
The provider provisions and manages workload-cluster servers, API server load balancers, and dedicated cluster networks (when enabled), while supporting pre-existing (BYO) networks, security groups, SSH keys, and images.
Network
StackitCluster.spec.network.id is the STACKIT network ID used by all cluster
machines and the provider-managed API server load balancer.
If you don't provide a network ID, CAPSTK will automatically create a dedicated /24 network for the cluster.
Planning your networks usually involves platform-level policy such as address
planning, routing, peering, VPN, DNS, and shared-service access, but the default
managed network is built for convenience. A newly created (managed) network
will have an attached tag indicating that it is managed by CAPSTK (cluster-api-provider-stackit=true).
Only empty networks can be deleted, so when you delete your CAPSTK cluster and all cluster nodes and load balancers are removed, the managed network will be deleted too.
Note: When using a managed network, the network ID is generated dynamically during cluster reconciliation and populated on
StackitCluster.status.network.id. Becausetemplates/addons/cloud-provider-stackit.yamlrequires this network ID for itsServicetypeLoadBalancercontroller, you usetemplates/cluster-template-managed-network.yamland configure the Cloud Controller Manager after the network ID is provisioned. See Cloud Provider with Managed Networks for the step-by-step procedure.
Bring-Your-Own (BYO) Network
Choose or create a network in the same region as the workload cluster. The network must provide:
- IP addresses for all control-plane and worker machines.
- Routing between workload-cluster nodes.
- Egress for cloud-init, package repositories when using development images, container image pulls, DNS, NTP, and STACKIT APIs.
- Traffic required by the selected CNI.
How to create a STACKIT Network manually
Create a network with the STACKIT CLI:
export STACKIT_REGION=eu01
export STACKIT_NETWORK_NAME="${CLUSTER_NAME}-network"
stackit network create \
--project-id "${STACKIT_PROJECT_ID}" \
--region "${STACKIT_REGION}" \
--name "${STACKIT_NETWORK_NAME}" \
--ipv4-prefix "10.42.0.0/24" \
--labels cluster-api-provider-stackit=true
Get the network ID for STACKIT_NETWORK_ID:
stackit network list \
--project-id "${STACKIT_PROJECT_ID}" \
--region "${STACKIT_REGION}" \
--output-format json
Delete a test network when it is no longer used:
stackit network delete "${STACKIT_NETWORK_ID}" \
--project-id "${STACKIT_PROJECT_ID}" \
--region "${STACKIT_REGION}"
Security Groups
StackitMachine.spec.securityGroups is an optional list of STACKIT security
group IDs attached to each created server. If omitted, STACKIT project defaults
apply.
For production clusters, prefer explicit security groups over implicit project defaults. At minimum, the selected security groups must allow:
- The Kubernetes API server on TCP
6443from the API server load balancer and from trusted administration networks. - Node-to-node traffic required by kubeadm, kubelet, the control plane, and the selected CNI.
- Egress for bootstrap, package repositories if needed, image registries, DNS, NTP, and STACKIT APIs.
Do not open SSH ingress by default. If SSH is needed for break-glass debugging, restrict it to trusted CIDRs.
Create a security group:
export STACKIT_SECURITY_GROUP_NAME="${CLUSTER_NAME}-nodes"
stackit security-group create \
--project-id "${STACKIT_PROJECT_ID}" \
--region "${STACKIT_REGION}" \
--name "${STACKIT_SECURITY_GROUP_NAME}" \
--stateful \
--description "Cluster API STACKIT nodes" \
--labels cluster-api-provider-stackit=true
Get the security group ID:
stackit security-group list \
--project-id "${STACKIT_PROJECT_ID}" \
--region "${STACKIT_REGION}" \
--output-format json
Add a Kubernetes API ingress rule, replacing STACKIT_SECURITY_GROUP_ID and
ADMIN_CIDR:
stackit security-group rule create \
--project-id "${STACKIT_PROJECT_ID}" \
--region "${STACKIT_REGION}" \
--security-group-id "${STACKIT_SECURITY_GROUP_ID}" \
--direction ingress \
--protocol-name tcp \
--port-range-min 6443 \
--port-range-max 6443 \
--ip-range "${ADMIN_CIDR}" \
--description "Kubernetes API"
Delete a test security group when it is no longer attached to servers:
stackit security-group delete "${STACKIT_SECURITY_GROUP_ID}" \
--project-id "${STACKIT_PROJECT_ID}" \
--region "${STACKIT_REGION}"
SSH Keys
StackitMachine.spec.sshKeyName is optional. It references an existing STACKIT
SSH key pair by name.
SSH is not used for Cluster API bootstrap. Kubeadm bootstrap data is generated
by CABPK, stored in a Kubernetes Secret, and sent to STACKIT as cloud-init user
data when each server is created. Leave sshKeyName and
STACKIT_SSH_KEY_NAME empty when SSH access is not required.
Because the CAPSTK controller accesses STACKIT through a service account, the SSH key pairs must be attached to that service account. Key pairs are scoped per identity and cannot be shared across service accounts or users.
Authenticate the STACKIT CLI using your service account credentials:
stackit auth activate-service-account \
--service-account-key-path "${STACKIT_SERVICE_ACCOUNT_JSON_FILE}"
Import an existing SSH public key:
export STACKIT_SSH_KEY_NAME="${CLUSTER_NAME}-break-glass"
stackit key-pair create \
--project-id "${STACKIT_PROJECT_ID}" \
--region "${STACKIT_REGION}" \
--name "${STACKIT_SSH_KEY_NAME}" \
--public-key "@${HOME}/.ssh/id_ed25519.pub" \
--labels cluster-api-provider-stackit=true
Only create and pass a key pair when operators need SSH for debugging. Opening SSH access also requires an explicit security group rule; the provider does not need SSH to create or join Nodes.
Delete a test key pair:
stackit key-pair delete "${STACKIT_SSH_KEY_NAME}" \
--project-id "${STACKIT_PROJECT_ID}" \
--region "${STACKIT_REGION}"
Node Images
StackitMachine.spec.imageID is the STACKIT image ID used for the node root
disk. The image must be compatible with cloud-init and kubeadm bootstrap.
For production clusters, use kubeadm-ready images that already contain the expected container runtime and Kubernetes node packages for the chosen Kubernetes minor. Runtime package installation is useful for development, but it adds dependency on external package repositories during cluster creation and slows down Machine readiness.
The current real e2e path can bootstrap generic Ubuntu images by installing
containerd, kubelet, kubeadm, and kubectl in kubeadm
preKubeadmCommands. Treat this as a development fallback, not the desired
production path.
List candidate images:
stackit image list \
--project-id "${STACKIT_PROJECT_ID}" \
--region "${STACKIT_REGION}" \
--limit 20 \
--output-format json
Use the selected image ID as STACKIT_IMAGE_ID. Prefer non-ARM Ubuntu images
or purpose-built kubeadm-ready images until this provider publishes a supported
image matrix.
Verification
The command forms above were verified against STACKIT with temporary resources:
stackit network create,network describe, andnetwork deletestackit security-group create,security-group rule create, andsecurity-group deletestackit key-pair createandkey-pair deletestackit image list
OS Images
Ubuntu Nodes
CAPSTK includes templates/cluster-template.yaml as the default example for
Ubuntu-based clusters:
- control-plane machines use the Ubuntu image from
STACKIT_IMAGE_ID - worker machines use the same Ubuntu image from
STACKIT_IMAGE_ID - kubeadm bootstrap runs through cloud-init
Select one of the available Ubuntu images:
stackit image list \
--project-id "${STACKIT_PROJECT_ID}" \
--region "${STACKIT_REGION}" \
--output-format json \
| jq -r '.[] | select(.name | test("Ubuntu")) | "\(.id)\t\(.name)"'
The default template currently supports generic Ubuntu cloud images by installing Kubernetes node dependencies at first boot:
- install
containerd,kubelet,kubeadm, andkubectlwithapt-get - use the Kubernetes package repository selected by
KUBERNETES_APT_REPOSITORY_MINOR - configure containerd with
SystemdCgroup = true - enable
br_netfilterand Kubernetes bridge/IP forwarding sysctls - register kubelet with
cloud-provider=external - make kubeadm use the STACKIT hostname from cloud-init metadata
(
{{ ds.meta_data.local_hostname }}), so cloud-provider-stackit can map the Kubernetes node to the STACKIT server
Render the default template like any other clusterctl template:
export STACKIT_IMAGE_ID=<ubuntu-image-id>
export KUBERNETES_VERSION=v1.35.8
export KUBERNETES_APT_REPOSITORY_MINOR=v1.35
clusterctl generate cluster "${CLUSTER_NAME}" \
--from templates/cluster-template.yaml \
--kubernetes-version "${KUBERNETES_VERSION}" \
--control-plane-machine-count "${CONTROL_PLANE_MACHINE_COUNT}" \
--worker-machine-count "${WORKER_MACHINE_COUNT}" \
| kubectl apply -f -
Notes:
- Ubuntu SSH access normally uses the
ubuntuuser. - Install a workload CNI after the API server is reachable.
- The default template downloads Kubernetes packages during first boot. This is useful for development and e2e testing, but production clusters should use a kubeadm-ready image that already contains the expected container runtime and Kubernetes node packages for the chosen Kubernetes minor.
- Prefer non-ARM Ubuntu images until this provider publishes a supported image matrix.
Flatcar Worker Nodes
See the official Flatcar STACKIT docs for generic Flatcar image usage on STACKIT.
CAPSTK includes templates/cluster-template-flatcar-workers.yaml as an example
for mixed-image clusters:
- control-plane machines use the regular Ubuntu image from
STACKIT_IMAGE_ID - worker machines use the Flatcar image from
STACKIT_WORKER_IMAGE_ID
Example amd64 Flatcar image:
export STACKIT_WORKER_IMAGE_ID=419c31da-39e3-4ea3-9bd8-699b44e8394f # Flatcar 4459.2.4
Flatcar requires a different bootstrap path than Ubuntu:
- use
KubeadmConfigTemplate.spec.template.spec.format: ignition - enable Cluster API's
KubeadmBootstrapFormatIgnitionfeature gate - do not reuse Ubuntu
apt-getbased bootstrap commands - make kubeadm use the STACKIT hostname from
/run/metadata/flatcar(COREOS_OPENSTACK_HOSTNAME), otherwise the node may register aslocalhostand cloud-provider-stackit cannot map it to the STACKIT server
Render the example template like any other clusterctl template:
clusterctl generate cluster "${CLUSTER_NAME}" \
--from templates/cluster-template-flatcar-workers.yaml \
--kubernetes-version "${KUBERNETES_VERSION}" \
--control-plane-machine-count "${CONTROL_PLANE_MACHINE_COUNT}" \
--worker-machine-count "${WORKER_MACHINE_COUNT}" \
| kubectl apply -f -
Notes:
-
Flatcar SSH access uses the
coreuser. When connecting through the CAPSTK bastion, use the bastion image's SSH user for the first hop, usuallyubuntu, andcorefor the Flatcar node:ssh -i "${CLUSTER_SSH_KEY}" core@<flatcar-node-internal-ip> \ -o "ProxyCommand ssh -W %h:%p -i ${CLUSTER_SSH_KEY} ubuntu@${BASTION_HOST}" -
Install a workload CNI after the API server is reachable.
-
The example template downloads Kubernetes binaries during first boot. Production images should provide those binaries through an image build, package, or system extension instead.
Accessing cluster instances
By default, workload clusters created by cluster-api-provider-stackit do not
expose SSH access. Cluster API bootstrap does not require SSH; CABPK provides
cloud-init user data through Kubernetes bootstrap Secrets.
For break-glass access, a StackitCluster can optionally ask the provider to
create one managed SSH bastion VM. The bastion is attached to the same STACKIT
network as the cluster nodes, gets a provider-managed public IP, and has a
provider-managed security group allowing TCP/22 only from configured CIDRs.
When bastion access is enabled, the provider also manages the node-side SSH path: it creates a separate security group for cluster nodes, allows TCP/22 from the bastion security group, and attaches that node SSH security group to control-plane and worker VMs.
Prerequisites
- The workload cluster was generated with
clusterctl generate clusterand applied to the management cluster. - The configured STACKIT service account role includes the bastion permissions documented in IAM Permissions Used.
- The bastion image is an Ubuntu image with SSH enabled for the expected user,
normally
ubuntu. - The STACKIT SSH key named in
spec.bastion.sshKeyNamealready exists for the service account used by the provider. STACKIT key pairs are not shared across service accounts, so importing a key with an inspection or admin service account does not make it usable by the controller service account. - To SSH from the bastion into cluster nodes, the control-plane and worker
StackitMachineTemplateresources must also setspec.template.spec.sshKeyName. The bastion does not inject SSH keys into existing node VMs.
Configure SSH keys for the service account
The CAPSTK controller authenticates to STACKIT through a service account key. Because SSH key pairs are scoped to the authenticated identity in this case a service account, the SSH keys you want to use on the bastion or on cluster machines must be attached under that exact service account and not your personal user.
-
Generate an SSH key pair locally if you do not already have one:
ssh-keygen -t ed25519 -f ~/.ssh/id_stackit -C "capstk-access" -
Authenticate the STACKIT CLI using your service account credentials:
stackit auth activate-service-account \ --service-account-key-path /path/to/service-account.json -
Import the public key as a key pair under that service account:
export STACKIT_SSH_KEY_NAME="capstk-key" stackit key-pair create \ --project-id "${STACKIT_PROJECT_ID}" \ --region "${STACKIT_REGION}" \ --name "${STACKIT_SSH_KEY_NAME}" \ --public-key "@${HOME}/.ssh/id_stackit.pub" -
Reference the key in your cluster definition:
When using
templates/cluster-template-bastion.yaml, pass the key name to the template variables:STACKIT_BASTION_SSH_KEY_NAME: setsspec.bastion.sshKeyNameonStackitClusterSTACKIT_SSH_KEY_NAME: setsspec.template.spec.sshKeyNameonStackitMachineTemplatefor control-plane and worker nodes
You can use separate keys for the bastion and the nodes, or point both variables to the same key.
Enable the bastion
Patch or edit the generated StackitCluster:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: StackitCluster
metadata:
name: <cluster-name>
namespace: <namespace>
spec:
bastion:
enabled: true
imageID: <ubuntu-image-id>
machineType: <machine-type>
sshKeyName: <existing-stackit-ssh-key-name>
allowedCIDRs:
- <your-public-ip-or-network-cidr>
rootVolume:
sizeGiB: 50
performanceClass: storage_premium_perf6
deleteOnTermination: true
cloudInitRef:
kind: ConfigMap
name: <bastion-cloud-init-configmap>
key: userData
Use a narrow allowedCIDRs value such as 203.0.113.10/32 where possible.
0.0.0.0/0 allows SSH from anywhere and should only be used deliberately.
Set rootVolume when the chosen machine type's flavor disk is too small for
the chosen image.
Set cloudInitRef when the bastion host needs additional
packages, users, files, or other cloud-init customization. The provider reads
the referenced ConfigMap or Secret and passes the value as-is to the bastion
VM. It is not applied to control-plane or worker nodes.
For node access, set the same or another existing SSH key on the machine templates:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: StackitMachineTemplate
metadata:
name: <machine-template-name>
spec:
template:
spec:
sshKeyName: <existing-stackit-ssh-key-name>
Leaving StackitMachineTemplate.spec.template.spec.sshKeyName empty is valid,
but SSH to nodes through the bastion will not work.
How node SSH access is wired
Enabling the bastion creates two provider-managed SSH security groups:
- The bastion SSH security group is attached to the bastion VM. It allows
TCP/22 from
spec.bastion.allowedCIDRs. - The node SSH security group is attached to each provider-managed control-plane and worker VM. It allows TCP/22 from the bastion security group by using the bastion security group as the remote source.
This means users connect from their workstation to the bastion public IP, and then from the bastion to node internal IPs. The provider manages the network permissions for both hops, but it does not manage SSH users or private key files. The SSH key named on the bastion and node specs must already exist in STACKIT for the service account used by the provider.
The node SSH security group is shared by all nodes in the cluster. New nodes
created while bastion is enabled get the group attached during
StackitMachine reconciliation.
Customize the bastion with cloud-init
spec.bastion.cloudInitRef references a complete cloud-init user-data document
for the bastion VM. Use a ConfigMap for non-sensitive configuration:
apiVersion: v1
kind: ConfigMap
metadata:
name: bastion-cloud-init
data:
userData: |
#cloud-config
packages:
- jq
---
spec:
bastion:
enabled: true
imageID: <ubuntu-image-id>
machineType: <machine-type>
sshKeyName: <existing-stackit-ssh-key-name>
allowedCIDRs:
- <your-public-ip-or-network-cidr>
cloudInitRef:
kind: ConfigMap
name: bastion-cloud-init
key: userData
Use a Secret instead when the cloud-init document contains sensitive values:
apiVersion: v1
kind: Secret
metadata:
name: bastion-cloud-init
stringData:
userData: |
#cloud-config
packages:
- jq
---
spec:
bastion:
cloudInitRef:
kind: Secret
name: bastion-cloud-init
key: userData
The referenced object must be in the same namespace as the StackitCluster.
The provider does not merge the referenced data with node bootstrap data.
Cloud-init user data is applied only when STACKIT creates a VM, so an existing
bastion cannot be reconfigured in place. If the referenced ConfigMap or Secret
content changes, the provider deletes and recreates the provider-managed
bastion VM with the new user data.
Recreating the bastion temporarily interrupts SSH access, assigns a new provider-managed bastion server, and may assign a new public IP. The provider also removes the old node SSH security group path and recreates it for the new bastion so control-plane and worker nodes remain reachable through the current bastion after reconciliation completes.
Get the bastion IP
Use the management-cluster kubeconfig:
kubectl get stackitcluster <cluster-name> \
--namespace <namespace> \
-o jsonpath='{.status.bastion.publicIP}'
Or inspect the printcolumn:
kubectl get stackitclusters --namespace <namespace>
The Bastion IP column is empty until STACKIT has assigned and attached the
public IP.
Get node internal IPs
Use the workload-cluster kubeconfig:
kubectl get nodes \
-o custom-columns=NAME:.metadata.name,IP:'{.status.addresses[?(@.type=="InternalIP")].address}'
Alternatively, inspect the management-cluster StackitMachine addresses:
kubectl get stackitmachines --namespace <namespace> \
-o custom-columns=NAME:.metadata.name,IP:'{.status.addresses[?(@.type=="InternalIP")].address}'
Connect through the bastion
Set local variables:
export CLUSTER_SSH_KEY="$HOME/.ssh/<private-key-file>"
export BASTION_HOST="$(kubectl get stackitcluster <cluster-name> \
--namespace <namespace> \
-o jsonpath='{.status.bastion.publicIP}')"
Connect to a node internal IP through the bastion:
ssh -i "${CLUSTER_SSH_KEY}" ubuntu@<node-internal-ip> \
-o "ProxyCommand ssh -W %h:%p -i ${CLUSTER_SSH_KEY} ubuntu@${BASTION_HOST}"
Use the node image's SSH user for the final hop. For example, Ubuntu nodes
usually use ubuntu, while Flatcar nodes use core.
An equivalent SSH config entry:
Host 10.*
User ubuntu
IdentityFile <cluster-ssh-key>
ProxyCommand ssh -W %h:%p ubuntu@<bastion-public-ip>
Cleanup
The provider deletes the managed bastion server, public IP, security group, and
security group rules when the StackitCluster is deleted. It also removes the
provider-managed node SSH security group from cluster nodes and deletes that
security group.
The same cleanup path runs when spec.bastion.enabled is changed from true
to false: the provider removes the node SSH security group from
provider-managed control-plane and worker VMs, deletes the node SSH security
group and its rules, and then deletes the bastion server, public IP, and
bastion security group.
Additional Notes
Using the STACKIT CLI instead of kubectl
It is also possible to use STACKIT CLI commands instead of kubectl to gather information about the cluster nodes.
For example, to use the STACKIT CLI to get the public IP address of the SSH bastion host, use this STACKIT CLI command:
set CLUSTER_NAME stackit-bastion-test
stackit server list -o json | jq -r --arg cluster "$CLUSTER_NAME" '
.[]
| select(.labels["cluster.x-k8s.io/cluster-name"] == $cluster)
| select(.labels["cluster-api-provider-stackit/resource-role"] == "bastion")
| .nics[]?.publicIp
'
188.34.94.28
You should substitute the correct cluster name for <CLUSTER_NAME> in the above command.
Similarly, to obtain the list of private IP addresses of the cluster nodes, use this STACKIT CLI command:
set CLUSTER_NAME stackit-bastion-test
stackit server list -o json | jq -r --arg cluster "$CLUSTER_NAME" '
.[]
| select(.labels["cluster.x-k8s.io/cluster-name"] == $cluster)
| select(.labels["cluster-api-provider-stackit/resource-role"] != "bastion")
| .nics[]?.ipv4
'
For names plus private IPs:
stackit server list -o json | jq -r --arg cluster "$CLUSTER_NAME" '
.[]
| select(.labels["cluster.x-k8s.io/cluster-name"] == $cluster)
| select(.labels["cluster-api-provider-stackit/resource-role"] != "bastion")
| [.name, (.nics[]?.ipv4 // empty)]
| @tsv
'
Finally, to obtain STACKIT instance names mapped with their private IPs, you can use this STACKIT CLI command:
stackit server list -o json | jq -r --arg cluster "$CLUSTER_NAME" '
.[]
| select(.labels["cluster.x-k8s.io/cluster-name"] == $cluster)
| select(.labels["cluster-api-provider-stackit/resource-role"] != "bastion")
| [.name, (.nics[]?.ipv4 // empty)]
| @tsv
'
Note that your STACKIT CLI must be configured with credentials that enable you to query the STACKIT Servers API.
HA, Availability Zones & Failure Domains
"Availability Zone" is a cloud provider-specific term, whereas "Failure Domain" is the broader, environment-agnostic concept used throughout Cluster API. A failure domain can represent a separation of physical switches, power feeds, rooms, racks, or distinct data centers. In the context of public cloud providers like STACKIT, failure domains and availability zones (AZs) are equivalent.
Failure Domain Discovery
The provider dynamically discovers STACKIT availability zones from the STACKIT
IaaS API (/v2/regions/{region}/availability-zones) and publishes them as Cluster
API failure domains on StackitCluster.status.failureDomains.
For a region such as eu01, the provider exposes the real physical zones:
eu01-1
eu01-2
eu01-3
Virtual metro zones (such as eu01-m with an -m suffix) are filtered out from
published failure domains so that machines are not placed into expensive
replicated metro zones by default.
Control Plane Spreading
Control plane spreading across failure domains is handled directly by Cluster API
(KubeadmControlPlane). When scaling or rolling out control plane nodes, KCP inspects
existing control plane machines and assigns the least-populated failure domain to
Machine.spec.failureDomain (breaking ties deterministically).
For this automatic spreading to function correctly, the failure domain list in
StackitCluster.status.failureDomains must be accurately populated and marked with
controlPlane: true.
Machine Placement and Defaulting
When provisioning a machine, the provider determines the target availability zone using the following precedence:
Machine.spec.failureDomain: Honors the placement selected by Cluster API core (e.g., KCP spreading control plane nodes or a MachineDeployment configured with a failure domain).StackitMachine.spec.availabilityZone: If explicitly configured on theStackitMachine(or viaStackitMachineTemplate), this zone is used.- Provider Defaulting: When
failureDomainis omitted, placement depends entirely on the infrastructure provider. If neitherMachine.spec.failureDomainnorStackitMachine.spec.availabilityZoneis specified (such as worker machines in a defaultMachineDeployment), the provider sorts the cluster's published failure domains alphabetically and selects the first zone (e.g.,eu01-1). This matches standard Cluster API conventions (such as CAPG) and ensures VMs are never placed into the expensive STACKIT metro zone (eu01-m) by default.
The actual zone where the VM was placed is surfaced on StackitMachine.status.availabilityZone.
Worker Node Spreading
Unlike the control plane, worker nodes do not automatically spread across failure domains
within a single MachineDeployment. Spreading worker nodes across AZs is an end-user
architecture decision made using the configuration fields on MachineDeployment or
MachinePool types.
The standard Cluster API pattern is to define multiple MachineDeployment resources (one
per availability zone), pinning each to a specific failure domain via
spec.template.spec.failureDomain.
IAM Permissions Used
cluster-api-provider-stackit authenticates to STACKIT with the service
account JSON stored in the StackitCluster.spec.credentialsSecretRef Secret.
That service account should use a custom project role with only the permissions
needed by the infrastructure provider.
Do not use broad project administrator roles for the controller. STACKIT documents custom roles as the way to bundle an explicit permission set, and role bindings as the way to assign that role to a user or service account.
Required provider permissions
The current provider implementation uses these STACKIT API operations:
| Provider action | Code path | STACKIT permission |
|---|---|---|
| Read the configured network | GetNetwork | iaas.network.get |
| Create managed network (if enabled) | CreateNetwork | iaas.network.create |
| Delete managed network (if enabled) | DeleteNetwork | iaas.network.delete |
| Create VM instances | CreateServer | iaas.server.create |
| Find existing tagged VM instances | ListServers | iaas.server.list |
| Read VM state | GetServer | iaas.server.get |
| Read VM NIC addresses for CAPI addresses and load balancer targets | ListServerNICs | iaas.server.nic.list |
| Delete VM instances | DeleteServer | iaas.server.delete |
| Create the optional bastion public IP | CreatePublicIP | iaas.public-ip.create |
| Find existing tagged bastion public IPs | ListPublicIPs | iaas.public-ip.list |
| Read the bastion public IP after attach | GetPublicIP | iaas.public-ip.get |
| Attach the bastion public IP to the bastion server | AddPublicIpToServer | iaas.server.public-ip.add |
| Detach the bastion public IP during cleanup | RemovePublicIpFromServer | iaas.server.public-ip.remove |
| Delete the optional bastion public IP | DeletePublicIP | iaas.public-ip.delete |
| Create the optional bastion security group | CreateSecurityGroup | iaas.security-group.create |
| Find existing tagged bastion security groups | ListSecurityGroups | iaas.security-group.list |
| Create bastion SSH ingress rules | CreateSecurityGroupRule | iaas.security-group.rule.create |
| List bastion SSH ingress rules | ListSecurityGroupRules | iaas.security-group.rule.list |
| Delete bastion SSH ingress rules during cleanup | DeleteSecurityGroupRule | iaas.security-group.rule.delete |
| Attach the bastion security group to the bastion server | AddSecurityGroupToServer | iaas.server.security-group.add |
| Detach the bastion security group during cleanup | RemoveSecurityGroupFromServer | iaas.server.security-group.remove |
| Delete the optional bastion security group | DeleteSecurityGroup | iaas.security-group.delete |
| Create the API server network load balancer | CreateLoadBalancer | nlb.loadbalancer.create |
| Find existing tagged load balancers | ListLoadBalancers | nlb.loadbalancer.list |
| Read the load balancer before target updates | GetLoadBalancer | nlb.loadbalancer.get |
| Delete the API server load balancer | DeleteLoadBalancer | nlb.loadbalancer.delete |
| Replace the API server target pool | UpdateTargetPool | nlb.targetpool.replace |
The least-privilege role for the current provider is therefore:
iaas.network.create
iaas.network.delete
iaas.network.get
iaas.public-ip.create
iaas.public-ip.delete
iaas.public-ip.get
iaas.public-ip.list
iaas.server.create
iaas.server.delete
iaas.server.get
iaas.server.list
iaas.server.nic.list
iaas.server.public-ip.add
iaas.server.public-ip.remove
iaas.server.security-group.add
iaas.server.security-group.remove
iaas.security-group.create
iaas.security-group.delete
iaas.security-group.list
iaas.security-group.rule.create
iaas.security-group.rule.delete
iaas.security-group.rule.list
nlb.loadbalancer.create
nlb.loadbalancer.delete
nlb.loadbalancer.get
nlb.loadbalancer.list
nlb.targetpool.replace
This list covers StackitCluster and StackitMachine reconciliation,
including the optional provider-managed bastion host. It does not include
permissions for manually creating networks, SSH keys, images, or other
prerequisite resources. It also does not include permissions for the in-cluster
cloud-provider-stackit add-on if you configure that add-on to manage
Kubernetes Service load balancers beyond the provider-managed API server load
balancer.
Create a strict role and service account with OpenTofu
Use OpenTofu and the STACKIT provider to create the custom role, service account, role assignment, and service-account key as one managed setup.
The bootstrap identity used by OpenTofu needs these setup permissions:
iam.role.addto create the custom roleiam.role.getandiam.role.listto read role stateiam.member.addto assign the role to the service accountiam.member.getto read role-assignment stateiam.service-account.createto create the service accountiam.service-account.getandiam.service-account.listto read service-account stateiam.service-account-key.createto create the service-account key
If the same OpenTofu configuration should also destroy the setup later, the
bootstrap identity also needs the corresponding remove/delete permissions:
iam.role.remove, iam.service-account.delete, and
iam.service-account-key.delete.
Create a STACKIT role:
resource "stackit_authorization_project_custom_role" "cluster_api_provider_stackit" {
resource_id = var.project_id
name = var.role_name
description = "Least-privilege role for cluster-api-provider-stackit VM, bastion, and API-server load-balancer reconciliation."
permissions = [
"iaas.network.get",
"iaas.public-ip.create",
"iaas.public-ip.delete",
"iaas.public-ip.get",
"iaas.public-ip.list",
"iaas.server.create",
"iaas.server.delete",
"iaas.server.get",
"iaas.server.list",
"iaas.server.nic.list",
"iaas.server.public-ip.add",
"iaas.server.public-ip.remove",
"iaas.server.security-group.add",
"iaas.server.security-group.remove",
"iaas.security-group.create",
"iaas.security-group.delete",
"iaas.security-group.list",
"iaas.security-group.get",
"iaas.security-group.rule.create",
"iaas.security-group.rule.delete",
"iaas.security-group.rule.list",
"iaas.security-group.rule.get",
"nlb.loadbalancer.create",
"nlb.loadbalancer.delete",
"nlb.loadbalancer.get",
"nlb.loadbalancer.list",
"nlb.targetpool.replace",
]
}
Create a STACKIT service account, assign the role and create a service account key:
resource "stackit_service_account" "cluster_api_provider_stackit" {
project_id = var.project_id
name = var.service_account_name
}
resource "stackit_authorization_project_role_assignment" "cluster_api_provider_stackit" {
resource_id = var.project_id
role = stackit_authorization_project_custom_role.cluster_api_provider_stackit.name
subject = stackit_service_account.cluster_api_provider_stackit.email
}
resource "stackit_service_account_key" "cluster_api_provider_stackit" {
project_id = var.project_id
service_account_email = stackit_service_account.cluster_api_provider_stackit.email
depends_on = [
stackit_authorization_project_role_assignment.cluster_api_provider_stackit,
]
}
You will find a working example in hack/tf/iam-setup.
To apply it:
tofu init
tofu apply \
-var "project_id=${STACKIT_PROJECT_ID}" \
-var "bootstrap_service_account_key_path=${BOOTSTRAP_SERVICE_ACCOUNT_KEY_PATH}"
Write the generated key to a local file:
export STACKIT_SERVICE_ACCOUNT_JSON_FILE=./.stackit/cluster-api-provider-stackit-serviceaccount.json
mkdir -p "$(dirname "${STACKIT_SERVICE_ACCOUNT_JSON_FILE}")"
tofu output -raw service_account_key_json > "${STACKIT_SERVICE_ACCOUNT_JSON_FILE}"
Next, create the Kubernetes Secret used by StackitCluster:
kubectl create secret generic stackit-credentials \
--namespace default \
--from-literal=project-id="${STACKIT_PROJECT_ID}" \
--from-file=serviceaccount.json="${STACKIT_SERVICE_ACCOUNT_JSON_FILE}"
For Developers
Verify the strict role with the billable e2e tests, not only by reading the permission list. The create/delete scenario exercises VM creation, VM lookup, VM deletion, load balancer creation, target-pool updates, and load balancer cleanup.
Run at least:
export STACKIT_E2E_CREATE_CLUSTER=true
export STACKIT_E2E_NODE_REF=true
export STACKIT_CREDENTIALS_SECRET_NAME=stackit-credentials
export STACKIT_CREDENTIALS_SECRET_NAMESPACE=default
make test-e2e-workload-noderef
To verify the optional bastion path, first import the SSH key pair with the
same service account stored in stackit-credentials; key pairs imported with a
different service account are not visible to the provider. Then set
STACKIT_SSH_KEY_NAME and STACKIT_BASTION_SSH_KEY_NAME and run:
export STACKIT_E2E_CREATE_CLUSTER=true
export STACKIT_E2E_BASTION=true
export STACKIT_SSH_KEY_NAME=<provider-service-account-keypair-name>
export STACKIT_BASTION_SSH_KEY_NAME="${STACKIT_SSH_KEY_NAME}"
make test-e2e-workload-bastion
For release validation, also run the scale, worker-upgrade, control-plane upgrade, and topology e2e targets with the same strict service account:
make test-e2e-workload-scale
make test-e2e-workload-upgrade-workers
make test-e2e-workload-upgrade-control-plane
make test-e2e-workload-topology
[!CAUTION]
Some broader SDK integration tests call helper APIs that are not used by the provider at runtime. For example,TestSDKClientListNetworksIntegrationcallsListNetworksand therefore needsiaas.network.list; the provider reconciler only callsGetNetworkwith the configured network ID, soiaas.network.getis sufficient for runtime.
Developer Guide
Use this guide to create a local development environment from a fresh checkout. It creates a Kind management cluster, installs Cluster API, and runs the STACKIT provider image you build from this repository. That management cluster then creates workload clusters in STACKIT, so use a project intended for development and expect the workload resources to incur cost.
Before you start
Install:
- Go
- Docker
kindkubectlclusterctlbase64
You also need a STACKIT project, a service-account JSON key, an existing network, an image, and a machine type. The service account needs the permissions listed in IAM permissions.
Clone this repository and run the commands below from its root.
Create the management cluster
The management cluster runs the Cluster API controllers and the provider. It is not the Kubernetes cluster that will run your workload.
Enterprise proxies like Zscaler
If your network uses a TLS-intercepting proxy such as Zscaler, the Kind node
must trust the proxy's root certificate. The host's certificate store does not
automatically apply inside the Kind node. Create a local kind-config.yaml
that mounts the host CA bundle into the node:
cat > kind-config.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraMounts:
- hostPath: /etc/ssl/certs/ca-certificates.crt
containerPath: /etc/ssl/certs/ca-certificates.crt
readOnly: true
EOF
kind create cluster --name capi-stackit --config kind-config.yaml
Without such a proxy, create the cluster without the configuration file:
kind create cluster --name capi-stackit
Point kubectl at the new management cluster:
kubectl config use-context kind-capi-stackit
Install Cluster API
Install Cluster API core, bootstrap, and control-plane providers into the management cluster:
clusterctl init \
--config hack/clusterctl-local.yaml \
--core cluster-api \
--bootstrap kubeadm \
--control-plane kubeadm
hack/clusterctl-local.yaml enables Cluster API features needed by the
repository's templates, including ClusterClass and ClusterResourceSet.
Build and deploy the local provider
Build the controller image, load it into the Kind node, and deploy the provider with that image:
export IMG=cluster-api-provider-stackit:dev
make docker-build IMG="${IMG}"
kind load docker-image "${IMG}" --name capi-stackit
make deploy IMG="${IMG}"
Wait until the locally built provider is ready:
kubectl rollout status \
--namespace cluster-api-provider-stackit-system \
deployment/cluster-api-provider-stackit-controller-manager
Configure STACKIT and create a workload cluster
The provider is now running from your checkout. Continue with the quick start
from set credentials and cluster settings.
Create the Secret in the default namespace used there, then create the workload
cluster.
Because you are working from a repository checkout, use
templates/cluster-template.yaml directly instead of downloading a release
asset:
clusterctl generate cluster "${CLUSTER_NAME}" \
--from templates/cluster-template.yaml \
--target-namespace "${NAMESPACE}" \
> cluster.yaml
kubectl apply -f cluster.yaml
When the workload API is available, retrieve the workload kubeconfig:
clusterctl get kubeconfig "${CLUSTER_NAME}" \
--namespace "${NAMESPACE}" \
> "${CLUSTER_NAME}".kubeconfig
Install a CNI using the repository helper (hack/install-workload-cni.sh) via
make:
make install-workload-cni \
WORKLOAD_KUBECONFIG="${CLUSTER_NAME}.kubeconfig"
By default, this installs Cilium using templates/addons/cilium-values.yaml. You
can also install Calico by setting STACKIT_WORKLOAD_CNI=calico, or apply a
custom manifest by setting CNI_MANIFEST=path/to/manifest.yaml. See
Workload CNI for more options.
Run the controller on your host
For controller debugging, stop the deployed provider first, keep
kind-capi-stackit as the active kubeconfig context, and run:
make undeploy
make run
make run connects to the active management cluster. When you stop it, rebuild,
load, and deploy the image again with the commands in the previous section.
For a live-reload development loop that automatically rebuilds and redeploys the controller on Go code changes, see Development with Tilt.
Cleanup
To fully delete a workload cluster and its cloud infrastructure:
kubectl delete -f cluster.yaml
Deletion should take at most 5 minutes. Verify that all resources are gone:
kubectl get cluster,machine,stackitcluster,stackitmachine --namespace "${NAMESPACE}"
Troubleshooting stuck deletion
If resources persist after several minutes:
-
Check the status and conditions of persisting resources:
kubectl describe cluster,machine,stackitcluster,stackitmachine --namespace "${NAMESPACE}" -
Inspect the controller logs for errors:
# Core Cluster API controller kubectl logs -n capi-system deployment/capi-controller-manager -c manager --tail=100 # STACKIT provider controller kubectl logs -n cluster-api-provider-stackit-system deployment/cluster-api-provider-stackit-controller-manager --tail=100 -
If a
StackitMachinecannot finish deletion and blocks its parent resources, you can remove its finalizer:kubectl patch stackitmachine <stackitmachine-name> -n "${NAMESPACE}" \ --type=merge -p '{"metadata":{"finalizers":null}}'Removing the finalizer unblocks Kubernetes deletion for that resource and its parents. You may then have to manually delete the actual VM in the STACKIT portal or through the STACKIT API.
Architecture
Overview
The provider has four main areas:
- API types and CRDs
- Controllers
- Cloud abstraction layer
- clusterctl templates and release assets
Key packages:
api/v1alpha1/ Provider API types
controller/ Reconciliation logic
cloud/ Cloud client interface and SDK implementation
cloud/fake/ In-memory fake for tests
util/ Shared helpers
templates/ clusterctl templates
config/ Kubebuilder manifests
The controllers do not call the STACKIT SDK directly. They use the cloud client interface, which keeps reconciliation testable and allows fake-client envtest coverage.
Cluster API Contract
The provider implements the Cluster API infrastructure contract for:
StackitClusterStackitClusterTemplateStackitMachineStackitMachineTemplate
Important contract behavior:
StackitCluster.status.initialization.provisionedis set when cluster infrastructure is ready.StackitMachine.status.initialization.provisionedis set when the VM is provisioned.StackitMachine.spec.providerIDis set to the provider-compatible value.- Conditions include
observedGeneration. - Paused clusters or paused resources must not trigger cloud API calls.
- Finalizers clean up provider-owned resources before Kubernetes object removal.
StackitCluster.spec.controlPlaneEndpoint is used for the Cluster API
infrastructure cluster contract when the provider manages the API server load
balancer endpoint.
Controllers
StackitClusterReconciler is responsible for:
- Reading credentials
- Looking up the configured network
- Managing the optional API server load balancer
- Maintaining its target pool from the control-plane machines of the cluster
- Publishing failure domains
- Updating readiness and contract status
- Cleaning up provider-managed load balancers on deletion
StackitMachineReconciler is responsible for:
- Waiting for bootstrap data
- Creating STACKIT servers
- Setting provider IDs and addresses
- Deleting servers on teardown
Reconciliation must be idempotent. Re-running the same reconcile loop should be safe and should not create duplicate cloud resources.
Cloud Client
The cloud client abstraction hides STACKIT SDK details from controllers.
It covers:
- Server create, get, list, and delete
- Network lookup
- Load balancer create, list, delete, and target-pool updates
- Provider ID helpers
- Error classification
Controllers should handle classified errors differently:
- transient errors should be retried
- terminal validation errors should set clear conditions
- not-found errors during deletion should be treated as successful cleanup where appropriate
Tests use an in-memory fake cloud client for deterministic reconciliation coverage. SDK integration tests are opt-in and require real STACKIT credentials.
STACKIT ProviderID Compatibility
The providerID format is verified against the local cloud-provider-stackit
repository.
Relevant reference points:
pkg/ccm/instances.go:Instances.makeInstanceIDreturnsstackit://<server-id>.pkg/ccm/instances.go:instanceIDFromProviderIDparsesstackit://<server-id>with no project or region component.pkg/ccm/instances.go:getInstanceresolves nodes withGetServer(projectID, region, serverID), where project and region come from the cloud-controller-manager configuration.pkg/ccm/instances_test.go: the "new providerID" table entry expectsstackit://hello-server.
cluster-api-provider-stackit therefore writes:
StackitMachine.spec.providerID = stackit://<server-id>
StackitMachine.status.providerID = stackit://<server-id>
Cluster API then surfaces StackitMachine.spec.providerID to
Machine.spec.providerID, and cloud-provider-stackit uses the same value for
Node.spec.providerID. Project ID and region are intentionally not encoded in
the providerID.
Testing
Unit and Env-Tests
Testing is split into layers:
- unit tests for helper functions and cloud logic
- envtest controller tests with Kubernetes API machinery
- opt-in SDK integration tests against a real STACKIT project
- opt-in e2e tests that create billable STACKIT resources
Default test command:
make test
This runs:
- generated manifest checks through controller-gen
go fmtgo vet- envtest-backed Go tests, excluding
/e2e
Real cloud tests are gated by environment variables so they do not run accidentally.
E2E-Tests
The e2e suite runs against an isolated kind management cluster. The generic target runs manager startup, metrics, webhook certificate, and webhook CA injection checks without STACKIT cloud resources:
make test-e2e
The real cloud paths create billable STACKIT servers, load balancers, public IPs, and security groups. Run them only in a dedicated STACKIT project with cleanup permissions and cost controls. The main workload scenarios have dedicated make targets:
| Scenario | Make target | Opt-in flag |
|---|---|---|
| NodeRef/providerID and Ready Nodes | make test-e2e-workload-noderef | STACKIT_E2E_NODE_REF=true |
| Bastion cluster create/delete with provider-managed bastion and node SSH security group cleanup | make test-e2e-workload-bastion | STACKIT_E2E_CREATE_CLUSTER=true, STACKIT_E2E_BASTION=true |
| Worker scale with Ready Nodes | make test-e2e-workload-scale | STACKIT_E2E_SCALE_WORKLOAD=true |
| Worker upgrade with Ready replacement Nodes | make test-e2e-workload-upgrade-workers | STACKIT_E2E_UPGRADE_WORKLOAD_WORKERS=true |
| Control-plane upgrade with a Ready replacement Node | make test-e2e-workload-upgrade-control-plane | STACKIT_E2E_UPGRADE_WORKLOAD_CONTROL_PLANE=true |
| ClusterClass topology create/ready/delete | make test-e2e-workload-topology | STACKIT_E2E_TOPOLOGY_WORKLOAD=true |
The suite also contains lower-level real cloud scenarios without dedicated make
targets. Run them through go test -tags=e2e ./test/e2e or make test-e2e
with the corresponding opt-in flag and a focused Ginkgo expression:
| Scenario | Opt-in flag | Suggested focus |
|---|---|---|
Single StackitMachine VM create/delete and leak check | STACKIT_E2E_CREATE_VMS=true | create and delete a real STACKIT VM |
| 1 control-plane / 1 worker infrastructure lifecycle without workload Node readiness | STACKIT_E2E_CREATE_CLUSTER=true | create and delete a 1 control-plane / 1 worker workload Cluster |
Infra-only MachineDeployment worker VM scale up/down | STACKIT_E2E_SCALE_WORKERS=true | scale a worker MachineDeployment |
Infra-only MachineDeployment worker VM replacement during version upgrade | STACKIT_E2E_UPGRADE_WORKERS=true | replace worker VMs during a MachineDeployment version upgrade |
Common required environment:
STACKIT_PROJECT_IDSTACKIT_REGIONSTACKIT_NETWORK_IDSTACKIT_IMAGE_IDSTACKIT_AVAILABILITY_ZONESTACKIT_CREDENTIALS_SECRET_NAMESTACKIT_CREDENTIALS_SECRET_NAMESPACE
Useful common options:
KUBERNETES_VERSION, for create/scale/topology pathsSTACKIT_E2E_UPGRADE_FROMandSTACKIT_E2E_UPGRADE_TO, for upgrade pathsSTACKIT_SSH_KEY_NAME, for node SSH keys when a scenario should attach oneSTACKIT_BASTION_SSH_KEY_NAME, orSTACKIT_SSH_KEY_NAME, whenSTACKIT_E2E_BASTION=trueSTACKIT_BASTION_ALLOWED_CIDRS, defaulting to0.0.0.0/0, for bastion SSH ingressSTACKIT_E2E_CNI,STACKIT_E2E_CNI_MANIFEST, and CNI-specific variablesSTACKIT_E2E_TEST_ID, for cleanup traceability
Real Cloud Node Bootstrap
The real workload-cluster e2e path is intentionally allowed to use a generic
Ubuntu cloud image during development. In that mode, the e2e fixture adds
kubeadm preKubeadmCommands that install and configure containerd,
kubelet, kubeadm, and kubectl at runtime before kubeadm joins the node.
Treat this runtime package installation as a development fallback. It proves the provider can pass CABPK-generated cloud-init data through STACKIT user data, but it is slower and less deterministic than a kubeadm-ready image. A production setup should use an image that already contains the expected container runtime and Kubernetes node packages for the selected Kubernetes minor.
Developing Cluster API Provider STACKIT (CAPSTK) with Tilt
This guide describes a local development loop with kind, Tilt,
upstream Cluster API Tilt, and this provider. Tilt runs from
a local cluster-api checkout and loads cluster-api-provider-stackit through
this repository's tilt-provider.yaml.
Use this workflow when you want Tilt to rebuild and redeploy the STACKIT controller while you edit Go code. Creating workload clusters still uses real STACKIT cloud resources and may incur cost.
Prerequisites
Install the development tools listed in the development guide. For this workflow you also need:
tiltkindkubectlclusterctl- Docker or another container runtime supported by Tilt
- a local checkout of
sigs.k8s.io/cluster-api - a STACKIT project, service-account JSON key, network, image, and machine type
The examples below assume both repositories are checked out next to each other:
cd /path/to/workspace
git clone https://github.com/kubernetes-sigs/cluster-api.git
If your cluster-api checkout is somewhere else, adjust paths in
tilt-settings.yaml.
Create the management cluster
Run Cluster API's kind-cluster target from the upstream cluster-api
checkout. This creates a kind cluster with a local registry so Tilt can use
local images without pushing to a remote registry.
cd /path/to/cluster-api
make kind-cluster
kubectl config current-context
The current context should be a kind context, for example kind-capi-test.
Configure Tilt
Create tilt-settings.yaml in the upstream cluster-api checkout:
kind_cluster_name: capi-test
provider_repos:
- ../cluster-api-provider-stackit
enable_providers:
- kubeadm-bootstrap
- kubeadm-control-plane
- stackit
template_dirs:
stackit:
- ../cluster-api-provider-stackit/templates
kustomize_substitutions:
CLUSTER_TOPOLOGY: "true"
CLUSTER_RESOURCE_SET: "true"
EXP_CLUSTER_RESOURCE_SET: "true"
NAMESPACE: default
KUBERNETES_VERSION: v1.35.8
CONTROL_PLANE_MACHINE_COUNT: "1"
WORKER_MACHINE_COUNT: "1"
STACKIT_PROJECT_ID: "<project-uuid>"
STACKIT_REGION: eu01
STACKIT_NETWORK_ID: "<network-uuid>"
STACKIT_IMAGE_ID: "<image-uuid>"
STACKIT_MACHINE_TYPE: c2i.1
STACKIT_AVAILABILITY_ZONE: eu01-1
STACKIT_SECURITY_GROUP_ID: "<security-group-uuid>"
STACKIT_SSH_KEY_NAME: ""
STACKIT_CREDENTIALS_SECRET_NAME: stackit-credentials
STACKIT_SERVICE_ACCOUNT_JSON_B64: "<base64-service-account-json>"
STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGE: "ghcr.io/stackitcloud/cloud-provider-stackit/cloud-controller-manager:v1.35.7"
Notes:
provider_reposmust point to this repository. Cluster API readstilt-provider.yamlfrom that path.template_dirsmakes the STACKIT cluster templates available as manual Tilt actions.- Keep
STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGEon the same Kubernetes minor asKUBERNETES_VERSION. - Set
preload_images: falseif Docker Desktop or multi-architecture image preloading fails.
Create the STACKIT credentials Secret in the management cluster namespace used by
the templates. The tilt-settings.yaml values are not exported into your shell,
so export at least these values before running the command manually:
export STACKIT_PROJECT_ID=<project-uuid>
export STACKIT_SERVICE_ACCOUNT_JSON_FILE=/path/to/serviceaccount.json
kubectl create secret generic stackit-credentials \
--namespace default \
--from-literal=project-id="${STACKIT_PROJECT_ID}" \
--from-file=serviceaccount.json="${STACKIT_SERVICE_ACCOUNT_JSON_FILE}"
Run Tilt
Start Tilt from the upstream cluster-api checkout:
cd /path/to/cluster-api
tilt up
Tilt should deploy:
- core Cluster API
- kubeadm bootstrap provider
- kubeadm control plane provider
- STACKIT infrastructure provider
Verify the provider pod:
kubectl get pods -n cluster-api-provider-stackit-system
kubectl logs -n cluster-api-provider-stackit-system \
deployment/cluster-api-provider-stackit-controller-manager \
-c manager
Create a workload cluster
Workload cluster creation is a manual Tilt action from the STACKIT templates. In
the Tilt UI, run the default template action for
templates/cluster-template.yaml, or render and apply it from the command line:
cd /path/to/cluster-api-provider-stackit
export CLUSTER_NAME=stackit-dev
export NAMESPACE=default
export KUBERNETES_VERSION=v1.35.8
export CONTROL_PLANE_MACHINE_COUNT=1
export WORKER_MACHINE_COUNT=1
export STACKIT_PROJECT_ID=<project-uuid>
export STACKIT_REGION=eu01
export STACKIT_NETWORK_ID=<network-uuid>
export STACKIT_IMAGE_ID=<image-uuid>
export STACKIT_MACHINE_TYPE=c2i.1
export STACKIT_AVAILABILITY_ZONE=eu01-1
export STACKIT_SECURITY_GROUP_ID=<security-group-uuid>
export STACKIT_SSH_KEY_NAME=""
export STACKIT_CREDENTIALS_SECRET_NAME=stackit-credentials
export STACKIT_SERVICE_ACCOUNT_JSON_B64=<base64-service-account-json>
export STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGE=ghcr.io/stackitcloud/cloud-provider-stackit/cloud-controller-manager:v1.35.7
clusterctl generate cluster "${CLUSTER_NAME:-stackit-dev}" \
--from templates/cluster-template.yaml \
--target-namespace default \
| kubectl apply -f -
Watch Cluster API and STACKIT resources:
kubectl get cluster,machine,stackitcluster,stackitmachine -n default
When the control plane is ready, install a workload CNI:
clusterctl get kubeconfig "${CLUSTER_NAME:-stackit-dev}" -n default > /tmp/stackit-dev.kubeconfig
make install-workload-cni WORKLOAD_KUBECONFIG=/tmp/stackit-dev.kubeconfig
Debug the STACKIT controller
Add a debug section for the stackit provider in tilt-settings.yaml:
debug:
stackit:
continue: true
port: 30000
Restart Tilt and attach a Go debugger to 127.0.0.1:30000.
Clean up
Delete workload clusters before deleting the management cluster:
kubectl delete cluster "${CLUSTER_NAME:-stackit-dev}" -n default
kind delete cluster --name capi-test
Use the STACKIT cleanup helper only for resources labelled by the e2e tests:
make cleanup-stackit STACKIT_E2E_TEST_ID=<test-id>
Troubleshooting
Failed to load provider. No tilt-provider.{yaml|json} file found: verifyprovider_repospoints to this repository, not tocluster-api.default_registry is required: runmake kind-clusterfirst so the kind cluster has the local-registry ConfigMap, or configure a writabledefault_registry.context ... is not allowed: switch to the kind management-cluster context or add the current context toallowed_contexts.StackitClustercannot read credentials: ensure thestackit-credentialsSecret exists in the same namespace as the Cluster and containsproject-idplusserviceaccount.json.- Workload Nodes stay unready: verify the selected image matches the Kubernetes
version,
STACKIT_CLOUD_CONTROLLER_MANAGER_IMAGEmatches the Kubernetes minor, and a CNI was installed into the workload cluster.
Release packaging
Build the GitHub Release assets locally:
make release IMG=<registry>/cluster-api-provider-stackit:<tag>
This creates a flat dist/release/ directory containing:
infrastructure-components.yamlmetadata.yamlclusterclass.yamlcluster-template.yamlcluster-template-bastion.yamlcluster-template-development.yamlcluster-template-flatcar-workers.yamlcluster-template-topology.yamlcilium-values.yamlcloud-provider-stackit.yamlinstall.yamlchecksums.txt
infrastructure-components.yaml is the Cluster API provider manifest used by
clusterctl. install.yaml is the standalone installer for kubectl apply.
For local clusterctl repository development, use:
make clusterctl-release IMG=<registry>/cluster-api-provider-stackit:<tag>
Release And Distribution Implementation Plan
This plan turns the current local-only release packaging into a production-ready
release flow for Cluster API Provider STACKIT (CAPSTK). The repo already has
make clusterctl-release, make build-installer, metadata.yaml, clusterctl
templates under templates/, and release docs that explicitly call this work
unfinished.
Current State
Makefilecan generate local clusterctl assets underdist/clusterctl/infrastructure-stackit/<version>/.metadata.yamlmaps release series0.1to the Cluster APIv1beta2contract.hack/clusterctl-local.yamlpoints clusterctl at a local provider repository, but currently referenceslatestwhile the Makefile writes a concrete version directory.- CI has lint, unit/envtest, and kind e2e workflows only.
go.modcurrently uses Cluster APIv1.13.2and Kubernetes librariesv0.35.4.- Production docs say the release path is unfinished and needs versioned images, published clusterctl assets, upgrade guidance, support matrix, image signing, and SBOMs.
Upstream Requirements And Guidance
- Cluster API providers must publish a provider repository with
metadata.yaml, a components YAML, and optional cluster templates. For an infrastructure provider, the recommended components asset name isinfrastructure-components.yaml. - Clusterctl local repositories use
infrastructure-<provider>/<semver>/...; each version folder must contain the components, metadata, and templates for that release. metadata.yamldocuments release series to Cluster API contract mappings and is strictly validated by newer clusterctl versions.clusterctl upgradeupgrades provider components and CRDs/controllers, but it does not upgrade workload Cluster API objects or workload Kubernetes versions.- Sigstore Cosign supports keyless container signing through OIDC in GitHub Actions. Sign images by digest, not by tag.
- GitHub artifact attestations and/or Cosign attestations can publish provenance and SBOM attestations.
Useful references:
- https://cluster-api.sigs.k8s.io/developer/providers/contracts/clusterctl
- https://cluster-api.sigs.k8s.io/clusterctl/commands/upgrade
- https://cluster-api.sigs.k8s.io/reference/versions
- https://docs.sigstore.dev/cosign/signing/signing_with_containers/
- https://github.com/actions/attest
Implementation Steps
1. Choose The Distribution Contract
Use GitHub Releases as the clusterctl provider repository and GHCR as the controller image registry.
Primary distribution:
clusterctl init --infrastructure stackit:<version>using release assets.- Controller image published to
ghcr.io/<owner>/cluster-api-provider-stackit.
Secondary distribution:
- Attach
install.yamlto the same release for users who do not use clusterctl.
Keep Helm out of scope unless maintainers explicitly choose it later.
Update:
docs/src/development/release-packaging.mddocs/src/quick-start.mdREADME.md
2. Harden Clusterctl Packaging
Update make clusterctl-release so CLUSTERCTL_RELEASE_VERSION is derived from
the release tag in CI instead of defaulting permanently to v0.1.0.
Required generated assets:
metadata.yamlinfrastructure-components.yamlcluster-template.yamlcluster-template-bastion.yamlcluster-template-development.yamlcluster-template-topology.yamlclusterclass.yamladdons/*.yaml
Fix the local repository mismatch:
- Either generate a
latestsymlink/copy underdist/clusterctl/infrastructure-stackit/latest. - Or update
hack/clusterctl-local.yamlto point at the explicitCLUSTERCTL_RELEASE_VERSION.
Add make verify-clusterctl-release:
- Generate assets.
- Create or reuse an isolated kind cluster.
- Run
clusterctl initagainsthack/clusterctl-local.yaml. - Verify the provider Deployment, CRDs, and metadata are accepted.
3. Publish Versioned Images
Add .github/workflows/release.yml, triggered by v* tags.
Build and publish multi-architecture images:
- Required:
linux/amd64,linux/arm64. - Optional:
linux/s390x,linux/ppc64leonly after explicit test coverage.
Publish immutable tags:
ghcr.io/<owner>/cluster-api-provider-stackit:<tag>- Optional stable aliases:
<major>.<minor>andlatestfor stable releases only.
Render release manifests with the exact version tag or digest. Prefer digest pinning if clusterctl/template UX remains acceptable.
4. Publish Release Assets
Release workflow sequence:
-
Checkout full history.
-
Setup Go from
go.mod. -
Run
make test. -
Build and push the multi-arch image.
-
Run:
make clusterctl-release \ IMG=ghcr.io/<owner>/cluster-api-provider-stackit:<tag> \ CLUSTERCTL_RELEASE_VERSION=<tag> -
Run:
make build-installer IMG=ghcr.io/<owner>/cluster-api-provider-stackit:<tag> -
Generate checksums for all release assets.
-
Upload assets to the GitHub Release.
Add a PR dry-run workflow that verifies image build and asset generation without publishing anything.
5. Add Signing, SBOMs, And Provenance
Use GitHub Actions OIDC with keyless Sigstore/Cosign.
Workflow permissions needed:
permissions:
contents: write
packages: write
id-token: write
attestations: write
artifact-metadata: write
Image signing:
- Install Cosign.
- Sign the pushed image by digest.
- Publish verification instructions in docs.
SBOM:
- Generate SPDX JSON and CycloneDX JSON using Syft or Docker BuildKit SBOM support.
- Upload SBOM files as release assets.
- Attach SBOM attestations to the image.
Provenance:
- Publish build provenance through GitHub artifact attestations and/or Cosign attestations.
Docs must include examples:
cosign verify ghcr.io/<owner>/cluster-api-provider-stackit@sha256:<digest>
cosign verify-attestation ghcr.io/<owner>/cluster-api-provider-stackit@sha256:<digest>
sha256sum -c checksums.txt
6. Publish Provider Upgrade Guidance
Create docs/src/usage/provider-upgrade.md.
Cover management-cluster provider upgrades:
clusterctl upgrade plan
clusterctl upgrade apply --infrastructure stackit:<version>
State explicitly:
- Clusterctl upgrades provider components.
- Clusterctl does not upgrade workload
Cluster,MachineDeployment,Machine, or Kubernetes versions. - Workload Kubernetes upgrades are handled separately through CAPI object changes.
Move or rewrite cluster-upgrade.md into the docs tree as workload Kubernetes
upgrade guidance.
Initial support policy:
- Support adjacent provider minor upgrades only.
- Do not support skip-minor upgrades until validated by e2e tests.
- Pre-release versions require explicit version selection for all providers.
7. Create The Support Matrix
Fill docs/src/topics/reference/versions.md.
Matrix columns:
- CAPSTK version.
- Cluster API version.
- Cluster API contract.
- Management Kubernetes versions.
- Workload Kubernetes versions.
- Cloud provider STACKIT image version requirement.
- Tested CNI versions.
- Tested STACKIT regions.
- Tested architectures.
- Upgrade paths validated by e2e.
Seed current values:
- CAPSTK
v0.1.x - Cluster API
v1.13.x - Contract
v1beta2 - Kubernetes libraries
v0.35.x - Workload cluster minors already documented as
v1.33.xthroughv1.36.xuntil validated/updated.
Make the support matrix release-owned: every release must update or explicitly confirm it.
8. Add Release Validation Gates
Create a release checklist and encode as much as possible in CI.
Required gates:
make lintmake test- generated files clean after
make manifests generate make clusterctl-releasemake build-installer- local
clusterctl initfrom generated assets - release asset checksums generated
- image pushed by immutable tag
- image signed by digest
- SBOM generated and attached
Billable e2e gates for release candidates:
make test-e2emake test-e2e-workload-noderefmake test-e2e-workload-scalemake test-e2e-workload-upgrade-workersmake test-e2e-workload-upgrade-control-planemake test-e2e-workload-topology
Run billable e2e only in a dedicated STACKIT project with cleanup permissions and cost controls.
9. Fix Contract And Documentation Mismatches Before First Production Release
Verify CRD contract metadata:
- Docs claim CRDs carry
cluster.x-k8s.io/v1beta2: v1alpha1. - Generated CRDs should be checked and fixed if that label is missing.
metadata.yamlcontract must match the actual CRD contract metadata.
Add CAPI RBAC aggregation if missing:
- Ship roles with
cluster.x-k8s.io/aggregate-to-manager: "true"where required so CAPI core controllers can manage provider resources correctly.
Validate ClusterClass behavior:
- Existing topology e2e covers a small topology cluster.
- Add release validation for template metadata and HA topology variants before claiming broader production ClusterClass support.
Acceptance Criteria
The release story is complete when an AI agent or maintainer can:
- Tag a release.
- CI builds and publishes a versioned multi-arch image.
- CI signs the image by digest.
- CI generates SBOMs and provenance.
- CI publishes clusterctl assets and
install.yamlto a GitHub Release. - A fresh kind management cluster can run
clusterctl initfrom the published release assets. - Docs explain install, verify, upgrade, support matrix, and workload Kubernetes upgrade behavior.
- Release validation covers unit/envtest, local clusterctl init, and the selected billable STACKIT e2e paths.
Make Targets
Common targets:
| Target | Purpose |
|---|---|
make test | Run generated checks, formatting, vet, and non-e2e tests |
make manifests | Regenerate CRDs and RBAC |
make generate | Regenerate DeepCopy code |
make install | Install CRDs into the current cluster |
make uninstall | Remove CRDs from the current cluster |
make docker-build | Build the controller image |
make deploy | Deploy the controller into the current cluster |
make undeploy | Remove the controller deployment |
make build-installer | Generate dist/install.yaml |
make clusterctl-release | Generate local clusterctl release assets |
make cleanup-stackit | Delete tagged e2e resources through STACKIT APIs |
make install-workload-cni | Install Cilium, Calico, or a custom CNI manifest into a workload cluster |
make test-e2e-workload-noderef | Run billable workload NodeRef/providerID and Ready Node e2e |
make test-e2e-workload-scale | Run billable workload worker scale and Ready Node e2e |
make test-e2e-workload-upgrade-workers | Run billable workload worker upgrade and Ready replacement Node e2e |
make test-e2e-workload-upgrade-control-plane | Run billable workload control-plane upgrade and Ready replacement Node e2e |
make test-e2e-workload-topology | Run billable ClusterClass topology create/ready/delete e2e |
Documentation targets inside docs/:
| Target | Purpose |
|---|---|
make build | Build the mdBook |
make serve | Serve the mdBook locally |
make clean | Remove generated book output |
Troubleshooting
ClusterClass is rejected
If the API server rejects ClusterClass, KubeadmControlPlaneTemplate, or
Cluster.spec.topology, check the CAPI manager feature gates:
kubectl -n capi-system get deployment capi-controller-manager \
-o jsonpath='{.spec.template.spec.containers[0].args}'
kubectl -n capi-kubeadm-control-plane-system \
get deployment capi-kubeadm-control-plane-controller-manager \
-o jsonpath='{.spec.template.spec.containers[0].args}'
Both must include ClusterTopology=true.
Machine waits for bootstrap data
Check the owning Machine and bootstrap Secret:
kubectl describe machine "${MACHINE_NAME}" -n "${NAMESPACE}"
kubectl get secret -n "${NAMESPACE}"
StackitMachine intentionally does not contain user data directly. Bootstrap
data comes from the CAPI bootstrap Secret.
STACKIT CLI might also be valueable to check server logs during boot: stackit server log <server-id>
Cloud resources remain after a failed e2e run
Run direct cleanup by e2e tags:
make cleanup-stackit
Then verify no tagged STACKIT VMs or load balancers remain.
CRD Reference
tbd