Upgrade a Helm-managed Lamassu deployment, including the 3.8.0 to 4.0.0 chart migration and the KMS and VA persistent-data steps.

Upgrade the Helm chart

Use this guide to upgrade a Helm-managed deployment. It walks through the 3.8.0 to 4.0.0 chart migration in full: the values to review, the database changes the chart applies automatically and the manual steps for installations that keep KMS keys or VA CRLs on local volumes. Version-specific notes for other upgrades live in the charts/lamassu/CHANGELOG/ directory of the lamassu-helm repository.

Fresh installs have nothing to migrate

If you are not upgrading an existing release, go straight to Install with Helm. The manual data steps on this page only apply to deployments upgraded from 3.8.x that use local KMS or VA storage.

What changes in 4.0.0

Version 4.0.0 is a security-hardening release. Three changes affect upgrades:

Release-scoped resource names. Every chart-managed resource is now named <release>-lamassu-<component>, for example lamassu-kms, instead of the fixed, unscoped names used in 3.8.0 (kms, va, ca, ...). With a release named prod the StatefulSets become prod-lamassu-kms and prod-lamassu-va, not prod-kms. The nameOverride and fullnameOverride values change the prefix.
Non-root containers. Every workload runs as the numeric user 65532:65532 with a RuntimeDefault seccomp profile and dropped capabilities. Pods do not mount the service-account token by default, and the chart creates a dedicated ServiceAccount.
Strict values validation. values.schema.json rejects unknown top-level keys, so leftover 3.8.0 blocks fail helm lint and helm upgrade outright instead of being silently ignored.

services.connectors also becomes a map keyed by connector ID, and the chart no longer injects pod anti-affinity or topology spread rules.

The rename is not cosmetic for KMS and VA

Kubernetes derives the PVC name from the StatefulSet pod name (<volumeClaimTemplate>-<pod-name>). Renaming the kms and va StatefulSets means helm upgrade binds brand-new, empty PVCs instead of the existing ones. Your KMS key material and VA CRLs are not deleted — the old PVCs are orphaned, not removed — but the upgraded pods fail to find them with no such file or directory (KMS) or code=NotFound (VA) until you move the data across yourself. All other renamed resources are stateless and are recreated without data-loss risk.

What the upgrade job does automatically

Before any service starts, the chart's pre-install/pre-upgrade job prepares PostgreSQL:

Creates the pki, authz and wfx databases when they do not exist.
Ensures one schema per service database (alerts, ca, va, devicemanager, dmsmanager, kms) inside the shared pki database, and folds 3.8.0's separate per-service databases into it, importing their data.
Runs the database migrations for every schema.
Migrates authz, preloads its policies and seeds the services.authz.bootstrap principals.

The job is idempotent, so a failed upgrade can be retried. It requires a PostgreSQL role allowed to create databases and schemas, as described in Install with Helm.

Before you upgrade

Record the names of your existing local-storage PVCs:

export NAMESPACE=lamassu   # your 3.8.0 namespace
export RELEASE=lamassu     # your Helm release name

kubectl get pvc -n $NAMESPACE -l app.kubernetes.io/instance=$RELEASE
# on a 3.8.0 install you can also list the fixed names directly:
kubectl get pvc -n $NAMESPACE golang-engine-storage-kms-0 local-crl-file-storage-va-0

helm upgrade does not touch or delete the old PVCs, but nothing references them after the upgrade. You need their names to copy data and to clean up later; do not delete them until the new volumes are verified.

Remove values the strict schema now rejects, for example the old toolbox: block. Update your values file lists every rename.

Update your values file

Add services.authz (required). 3.8.0 had no authorization service: every route accepted any valid token. 4.0.0 adds fine-grained, entity-level authorization, and a 3.8.0 values file has nothing to migrate from — you must add the configuration:

services:
  authz:
    credentials:
      pki:
        database: pki
      authz:
        database: authz

authz is the authorization service's own storage for principals, grants and policies (services.authz.database, authz by default).
pki is the database the authorization engine queries when it evaluates policies against entities such as CAs, certificates, devices, DMSs, KMS keys and VA roles.
Only the database field of each credentials entry is read. Connection settings always come from the global postgres.hostname, postgres.port, postgres.username and postgres.password values.
Confirm services.authz.bootstrap (present by default in the chart) matches how your OIDC provider assigns the pki-admin realm role. Without a principal that matches the bootstrap rule, nobody can administer the PKI — there is no implicit superuser.

Rename toolbox to connectivityTest. The helm test connectivity hook now uses the upstream-maintained curlimages/curl image pinned by digest instead of the chart-owned toolbox image:

# OLD (3.8.0)
toolbox:
  image: ghcr.io/lamassuiot/toolbox:2.2.0

# NEW (4.0.0)
connectivityTest:
  image: curlimages/curl@sha256:58adaa4e8dca9c988bae2aba4ab3434a0bb2da16bbe3f92dec39ec7785166777

Remove per-service identity overrides. services.kms.securityContext.runAsGroup: 0 and services.authz.securityContext.runAsUser/runAsGroup: 999 no longer exist. Every service runs under the chart-wide 65532:65532, and KMS volume and PKCS#11 socket group access is handled automatically by the pod-level fsGroup: 65532.

Convert services.connectors to a map. Each entry is keyed by its stable connector ID and declares type and image:

services:
  connectors:
    aws.myconnector:
      type: awsiot
      image: ghcr.io/lamassuiot/lamassu-aws-connector:dev-v4

Check the TLS issuer values. With tls.certManagerOptions.issuer empty (the default), the chart creates a self-signed, release-scoped issuer — for example lamassu-downstream-ca-selfsigned-issuer with release lamassu. The legacy literal downstream-ca-selfsigned-issuer from 3.8.0-era values is still accepted and resolves to the same release-scoped issuer; point tls.certManagerOptions.clusterIssuer at your own issuer for corporate TLS.

Review scheduling and ports. The chart no longer injects pod anti-affinity or topology spread constraints — configure affinity and topologySpreadConstraints explicitly where your cluster needs them. The services.ui.port default is now 8085, the unprivileged chart-wide port of the rootless image, so update overrides that pinned 80. The shared probes: block is replaced by independent livenessProbe, readinessProbe and startupProbe blocks.

Run the upgrade

helm upgrade --install "$RELEASE" lamassu/lamassu \
  --namespace $NAMESPACE \
  --values values.yaml \
  --wait

Helm runs the database job before applying any workload, so services never start against an unprepared database. If validation rejects unknown values, fix the values file and re-run.

Migrate the KMS and VA data

Skip this section on fresh installs, or when services.kms.cryptoEngines never used a filesystem engine and services.va.fileStore.type was never local.

Find the new StatefulSets and PVCs. Their names depend on the release name and any nameOverride/fullnameOverride; the PVC name is <volumeClaimTemplate>-<StatefulSet name>-0:

KMS_STS=$(kubectl get statefulset -n "$NAMESPACE" \
  -l "app.kubernetes.io/instance=$RELEASE,app.kubernetes.io/component=kms" \
  -o jsonpath='{.items[0].metadata.name}')
VA_STS=$(kubectl get statefulset -n "$NAMESPACE" \
  -l "app.kubernetes.io/instance=$RELEASE,app.kubernetes.io/component=va" \
  -o jsonpath='{.items[0].metadata.name}')

KMS_PVC="golang-engine-storage-${KMS_STS}-0"
VA_PVC="local-crl-file-storage-${VA_STS}-0"

kubectl get pvc -n "$NAMESPACE" \
  golang-engine-storage-kms-0 "$KMS_PVC" \
  local-crl-file-storage-va-0 "$VA_PVC"

With release lamassu and the default chart name, the new PVCs are golang-engine-storage-lamassu-kms-0 and local-crl-file-storage-lamassu-va-0; with release prod they are golang-engine-storage-prod-lamassu-kms-0 and local-crl-file-storage-prod-lamassu-va-0. Confirm both old and new PVCs exist before continuing.

Scale the new StatefulSets down so nothing writes to either volume while you copy:

kubectl scale statefulset -n "$NAMESPACE" "$KMS_STS" "$VA_STS" --replicas=0

Run a short-lived helper pod that mounts all four claims:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: pvc-migrate-helper
  namespace: $NAMESPACE
spec:
  restartPolicy: Never
  containers:
    - name: migrate
      image: docker.io/library/alpine:3.20
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts:
        - {name: old-kms, mountPath: /old/kms}
        - {name: new-kms, mountPath: /new/kms}
        - {name: old-va,  mountPath: /old/va}
        - {name: new-va,  mountPath: /new/va}
  volumes:
    - {name: old-kms, persistentVolumeClaim: {claimName: golang-engine-storage-kms-0}}
    - {name: new-kms, persistentVolumeClaim: {claimName: $KMS_PVC}}
    - {name: old-va,  persistentVolumeClaim: {claimName: local-crl-file-storage-va-0}}
    - {name: new-va,  persistentVolumeClaim: {claimName: $VA_PVC}}
EOF

kubectl wait -n $NAMESPACE --for=condition=Ready pod/pvc-migrate-helper --timeout=60s

Inspect before writing so you do not clobber data the new pods may already have written:

kubectl exec -n $NAMESPACE pvc-migrate-helper -- sh -c 'find /new/kms /new/va -type f'

An empty result is the common case: a StatefulSet's own pod rarely gets far enough to write anything when its dependent data is missing.

Copy the data and align ownership:

kubectl exec -n $NAMESPACE pvc-migrate-helper -- sh -c '
  cp -a /old/kms/. /new/kms/ &&
  cp -a /old/va/. /new/va/ &&
  chown -R 65532:65532 /new/kms /new/va
'

The chown is required even if the old files already look non-root-owned. 3.8.0 did not pin a container identity, so files could carry any image-declared UID; 4.0.0 requires exactly 65532:65532 and does not fall back to whatever the image declares.

Scale back up and remove the helper:

kubectl delete pod -n $NAMESPACE pvc-migrate-helper
kubectl scale statefulset -n "$NAMESPACE" "$KMS_STS" "$VA_STS" --replicas=1

Verify the migration

KMS and VA logs show no no such file or directory or code=NotFound errors.
A previously issued CRL is servable again: GET /crl/<ca-ski> on VA.
helm test "$RELEASE" -n $NAMESPACE --logs passes; the connectivity hook checks the CA, DMS Manager, Device Manager and VA health endpoints and the UI response.
Sign in with an administrator identity and confirm operations are authorized.

Delete the old volumes

Only once everything works, delete the orphaned 3.8.0 volumes to reclaim storage:

kubectl delete pvc -n $NAMESPACE golang-engine-storage-kms-0 local-crl-file-storage-va-0

Troubleshooting

KMS or VA pods crash with no such file or directory or code=NotFound right after the upgrade. The new pods are bound to the new empty PVCs and the data migration has not run yet, or was copied into the wrong PVC. Follow Migrate the KMS and VA data and check that all four claims exist.
helm upgrade fails with an unknown-key validation error. Remove or rename the legacy values — typically the toolbox: block — because the strict schema rejects top-level keys it does not know.
The upgrade fails at the database job. The PostgreSQL role cannot create databases or schemas; grant the required privileges, or create the pki, authz and wfx databases yourself, then re-run the upgrade.
Nobody can log in after the upgrade. Check that services.authz.bootstrap matches a role your OIDC provider actually assigns (see Access control) and that the migration Job completed successfully.

Install with Helm

Prepare dependencies, configure values and install a controlled deployment.

Expose the Gateway

Choose an external traffic design and handle Gateway addressing.

Upgrade the Helm chart

Install with Helm

Expose the Gateway

On this page