3.8.0 to 4.0.0

What changes in the 4.0.0 architecture, which changes break a 3.8.0 deployment, how to upgrade a Helm release step by step and how to validate the result.

Lamassu 4.0.0 is a major release. Storage consolidates into a single PostgreSQL database, a new authz service authorizes every request, the Helm chart is hardened, and several APIs and stored configurations change shape. Most data converts automatically during the upgrade, but some changes need operator action before, during or after it.

This page is organized in four parts. Read them in order before you upgrade production:

  1. Architecture changes: what is different in 4.0.0 and what the upgrade migrates on its own.
  2. Breaking changes: every change that can break a 3.8.0 deployment, who it affects and what to do.
  3. Helm migration guide: the step-by-step procedure for a Helm-managed release.
  4. Validation checklist: how to confirm the upgrade worked before you remove the old data.

Backend migration steps are provisional

The 4.0.0 release is upcoming and the backend migration steps on this page are provisional. Verify each procedure against the final release artifacts before upgrading production.

Architecture changes

One PostgreSQL database with schemas

3.8.0 gave each service its own database. The 4.0.0 backend connects to a single PostgreSQL database named pki and isolates each service in a schema: kms, ca, va, devicemanager, dmsmanager and alerts. The authz service keeps its principals, grants and policies in a separate authz database (the development monolith uses an authz schema inside pki instead), and WFX keeps its own wfx database.

PostgreSQL is now the only storage engine. SQLite support is removed, and the couch_db and dynamo_db engines are not registered in 4.0.0.

Authorization moves into the authz service

In 3.8.0 the API Gateway validated tokens: a JWT policy on every private route checked the signature and only let through tokens that carried the administrator role. Every route was open to any token that passed that check.

4.0.0 removes the gateway policy. The /api/* routes reach the services directly, and each service authorizes every request through authz, which validates the token against the identity provider's JWKS endpoint and applies fine-grained, entity-level policies to CAs, certificates, devices, DMSs, KMS keys and CRL settings. There is no implicit superuser: the first administrator comes from a bootstrap principal that you configure. The access control page documents principals, policies and the first administrator.

Device events get their own store

Historical device events move from the devices.events column to a dedicated device_events store with their own endpoints. The events endpoint can also stream events over Server-Sent Events (SSE); to fan events out across service instances, each Device Manager instance binds its own durable AMQP queue.

A hardened Helm chart

Version 4.0.0 of the chart is a security-hardening release:

  • Release-scoped resource names. Every chart-managed resource is named <release>-lamassu-<component> instead of the fixed names used in 3.8.0 (kms, va, ca, ...). With a release named prod the StatefulSets become prod-lamassu-kms and prod-lamassu-va. The nameOverride and fullnameOverride values change the prefix.
  • Non-root containers. Every workload runs as the numeric user 65532:65532 with a RuntimeDefault seccomp profile and dropped capabilities. Pods do not mount the service-account token by default, and the chart creates a dedicated ServiceAccount.
  • Strict values validation. values.schema.json rejects unknown keys, so leftover 3.8.0 blocks fail helm lint and helm upgrade instead of being silently ignored.
  • No injected scheduling rules. The chart no longer adds pod anti-affinity or topology spread constraints on its own.

What the upgrade migrates automatically

Several migrations run without intervention when you upgrade a Helm release:

  • Before any service starts, the chart's pre-install/pre-upgrade job prepares PostgreSQL. It creates the pki, authz and wfx databases when they do not exist, creates one schema per service inside pki, and imports each 3.8.0 per-service database into its schema. Each import runs as a single transaction, so a failed import leaves no partial schema and a retried job imports again. The job then runs the database migrations for every schema, migrates authz, preloads its policies and seeds the services.authz.bootstrap principals. It is idempotent and requires a PostgreSQL role allowed to create databases and schemas, as described in Install with Helm.
  • Stored DMS configuration is rewritten to the 4.0.0 layout, and historical device events are transformed into the new store.
  • The KMS migration removes the stale lamassu.io/kms/binded-resources key metadata, and the VA migration resolves each role's CRL signer (see VA CRL signer).
  • When migrations.caToKms.enabled is set, a pre-upgrade hook registers legacy CA keys in the KMS.

Two things never migrate on their own: the KMS and VA data on local volumes, which stays on volumes the upgraded pods no longer use, and the values file, which you update by hand. Deployments that do not use the Helm chart also have to move their database data themselves; see Deployments without Helm.

Breaking changes

ChangeWho is affectedSeverityAction
Release-scoped resource namesHelm releases with a filesystem KMS engine or local VA CRL storageHigh: KMS and VA start on empty volumesCopy the data into the new volumes
Strict values validationEvery Helm releaseHigh: helm upgrade failsRemove or rename legacy keys
Required authorization configurationEvery deploymentHigh: without a bootstrap principal there is no administratorConfigure services.authz
Gateway token validation removedEvery Helm releaseHigh: helm upgrade fails on the old keysMove JWKS and roles into services.authz
PostgreSQL consolidation3.8.0 deployments with per-service databasesHigh: services find no data, or a migration abortsClean duplicate CA IDs and empty engine_id values first; move data manually outside Helm
SQLite storage removedDeployments on SQLite, couch_db or dynamo_dbHigh: the backend no longer starts with themMove to PostgreSQL before upgrading
CRL settings keyed by CA IDEvery deployment; API clients, custom policies and alert subscriptions are also affectedHigh: the /v1/roles routes are removed and the migration is irreversibleFix the data the migration aborts on, and back up the database and the CRL bucket
Webhooks with file referencesDMSs whose webhook mTLS certificate or key were file pathsHigh: enrollments calling the webhook failRe-enter certificate and key as PEM, or remove the file fields
Non-root containersReleases with per-service identity overrides or mounted key materialMedium: services cannot read their filesRemove the overrides and fix volume ownership
AMQP over TLSReleases that set amqp.tlsMedium: clients connect to the wrong portPoint amqp.port at the TLS listener, or disable TLS
Legacy CA keys in the KMSInstallations whose CA keys predate the KMSMedium: CA keys stay unmanagedRun the CA-to-KMS migration
Receiver certificate verificationDMSs whose webhook had validate_server_cert: falseMedium: credentials go to an unverified receiverSet insecure_skip_verify to false
Re-enrollment trust setScripts and consoles that edit DMSs through the APIMedium: an old-layout edit silently empties validation_casRead the DMS before editing and use the new field
CRL signer fieldClients that create or update CRL settings through the APIMedium: requests with the old field return an errorReplace crl_options.subject_key_id_signer with the required crl_options.signer_serial_number
Renamed VA eventsAlert subscriptions and consumers of the VA eventsMedium: subscription types are renamed automatically, their conditions are notReview conditions filtering on the old event payload or subject
Device eventsConsumers that read the device events fieldMedium: the field disappearsUse the new event endpoints
VA CRL signerDeployments with VA roles whose signer cannot be resolved uniquelyMedium: the VA migration abortsFix the affected roles before upgrading
Renamed and reshaped valuesReleases that override toolbox, services.connectors, ports, probes or schedulingLow to medium: validation errors or changed defaultsUpdate the values file
DMS update APIClients that send the DMS ID in the update bodyLow: the ID in the URL winsSend the ID in the URL
CA IDs are always assigned by the backendClients that send an id when creating or importing CAsLow: the ID in the body is ignored, and the field is removedLet the backend assign the ID and use the returned one
Certificate type valuesConsumers that match on certificate typeLow: two values are renamedAdapt consumers
Independent re-enrollment authenticationAny DMSLow: informationalRead both blocks before editing

Helm chart and values

Release-scoped resource names

Kubernetes derives a StatefulSet's PVC name from its pod name (<volumeClaimTemplate>-<pod-name>). Renaming the kms and va StatefulSets means helm upgrade binds brand-new, empty PVCs instead of the existing ones. The KMS key material and VA CRLs are not deleted — the old PVCs are orphaned, not removed — but the upgraded pods fail with no such file or directory (KMS) or code=NotFound (VA) until you copy the data across. This affects KMS when services.kms.cryptoEngines uses a filesystem engine and VA when services.va.fileStore.type is local. Every other renamed resource is stateless and is recreated without data-loss risk.

Step 4 of the migration guide copies the data.

Strict values validation

values.schema.json rejects unknown keys, so every block that 4.0.0 removed or renamed — toolbox, auth.oidc.apiGateway, auth.authorization and others — fails helm lint and helm upgrade outright. Step 2 lists every change.

Required authorization configuration

3.8.0 had no authorization service, so a 3.8.0 values file has nothing to migrate from: you must add the services.authz configuration yourself. It needs the JWKS endpoint of your identity provider, the database mappings the authorization engine uses, and a bootstrap principal that matches how your provider assigns the administrator role. Without a matching principal nobody can administer the PKI.

Gateway token validation removed

auth.oidc.apiGateway.jwks, auth.authorization.rolesClaim and auth.authorization.roles.admin are removed with no compatibility aliases, and the schema rejects them. Their settings move into services.authz: the JWKS URL becomes services.authz.jwkUrl, and the role claim and administrator role become a claim rule of the bootstrap principal. auth.oidc.frontend is unchanged.

Non-root containers

Service images run on a distroless base as the numeric user 65532; the authz image is Ubuntu with a non-root user that has the same UID. The per-service overrides services.kms.securityContext.runAsGroup: 0 and services.authz.securityContext.runAsUser/runAsGroup: 999 are removed; group access to the KMS volume and PKCS#11 socket is now handled by the pod-level fsGroup: 65532.

3.8.0 did not pin a container identity, so existing files can carry any UID. Any volume that holds keys or CRLs must be owned by 65532:65532, or at least readable (and, where needed, writable) by that identity.

AMQP over TLS

In 3.8.0 amqp.tls was not applied and event-bus clients always connected over plaintext. In 4.0.0 amqp.tls: true switches them to AMQPS. The default amqp.port stays 5672 and does not change when TLS is enabled, so a values file that already set amqp.tls: true must point amqp.port at the broker's TLS listener (amqps, commonly 5671), or set amqp.tls: false to keep plaintext. Clients must also trust the broker's certificate.

Renamed and reshaped values

  • toolbox is renamed to connectivityTest, and the helm test hook uses the upstream curlimages/curl image pinned by digest.
  • services.connectors is a map keyed by connector ID instead of a list.
  • The services.ui.port default is 8085, the unprivileged port of the rootless image.
  • The shared probes: block is replaced by independent livenessProbe, readinessProbe and startupProbe blocks.
  • With tls.certManagerOptions.issuer empty, the chart creates a release-scoped self-signed issuer, for example lamassu-downstream-ca-selfsigned-issuer. The legacy literal downstream-ca-selfsigned-issuer is still accepted and resolves to it.
  • Anti-affinity and topology spread must be configured explicitly through affinity and topologySpreadConstraints.

Data and storage

PostgreSQL consolidation

Every 3.8.0 per-service database must end up as a schema of pki, keeping its structure, indexes, constraints, data and migration history. The Helm chart does this automatically; deployments without Helm do it by hand (see Deployments without Helm).

Two schema migrations abort on data that 3.8.0 allowed, so clean the data before upgrading:

  • The CA migration adds a UNIQUE constraint on ca_certificates.id and aborts if duplicate CA IDs exist. Imports already reject duplicates with 409, so new ones cannot normally appear.
  • The KMS migration promotes the primary key to (key_id, engine_id) and aborts with cannot promote (key_id, engine_id) to primary key: N row(s) have no engine_id; backfill them before migrating when kms_keys rows lack engine_id. Backfill engine_id first.

The same KMS migration deletes, with a NOTICE, aliases that equal an existing key ID, because identifiers now resolve as a key ID before an alias. Affected aliases are dropped, not redirected.

SQLite storage removed

The SQLite storage engine is removed, and the couch_db and dynamo_db engines are not registered in 4.0.0: their configuration constants remain, but they select no engine. Deployments on any of them must move to PostgreSQL and consolidate their data into the pki schemas before upgrading.

Legacy CA keys in the KMS

CA certificates of type MANAGED or IMPORTED_WITH_KEY whose private keys predate the KMS service have the key stored only in the crypto engine, under the certificate's subject_key_id, with no KMS record. The ca-to-kms-migration tool registers them:

  • The key_id is the certificate's subject_key_id; it matches the SHA-256 digest of the public key only when the SKI was computed that way.
  • The tool is idempotent: keys already in the KMS are skipped, so it can be run several times.
  • Certificates that cannot be read, or lack an engine or a subject key ID, are skipped with a warning. Review the logs and resolve the affected records.
  • When CA certificates sharing a subject_key_id declare conflicting engine_ids, the tool fails that key and exits with code 1, which fails a Helm pre-upgrade hook. Resolve the conflict manually before rerunning.

Once the keys are registered, certificates reference them in the database and the KMS refuses to delete a key that a certificate still uses (409 key is in use). Deleting a certificate record removes it from the CRL and OCSP, so read Before deleting a key before freeing a key.

Optionally, installations with imported CAs from before 3.7 can start the KMS once with crypto_engines.migrate_keys_format: true: the KMS renames those SKI-named legacy keys to the SHA-256 digest of their public key (the old name is kept as an alias, and the certificate references follow automatically). Run it once and disable the option again; running it again is safe.

With Helm, enable the hook for the upgrade in step 2; without Helm, run the tool by hand.

References between certificates, keys and CRL settings

Certificates now reference their KMS key, and CRL settings reference their signer certificate, so a key or certificate in use cannot be deleted. The KMS migration drops the legacy lamassu.io/kms/binded-resources key metadata. Before 4.0.0, deleting a CA could remove its key while leaving key references in its retained certificate records. The CA migration registers those missing keys in the KMS with only their public key to satisfy the retained references. These entries cannot be deleted while the retained certificates reference them; they do not restore the private key. In 4.0.0, deleting a CA releases the references from its own current certificate and linked reissued certificates before attempting to delete the key; other certificates that still reference it prevent deletion.

APIs and stored configuration

DMS webhooks

Stored webhook configs are rewritten to the current layout and their fields renamed:

Field in 3.8.0Field in 4.0.0
validate_server_certinsecure_skip_verify (value negated)
oidcjwt_options
apikeyapikey_options
mtlsmtls_options

When the legacy mTLS certificate and key are both PEM, they move into mtls_options.cert_content and mtls_options.key_content and need no action. Two cases do need it.

Webhooks with file references. A mixed config, with a PEM certificate and a file-path key, is migrated to mtls_options.cert_file and mtls_options.key_file. Webhook delivery rejects those fields on every call, whatever the configured authentication mode, so every enroll and re-enroll that calls that webhook fails until the config is fixed. If the webhook uses mTLS, re-enter the certificate and key as Base64-encoded or plain PEM; if it does not, remove the leftover file fields. Metadata-only updates keep working in the meantime (PUT or PATCH on /v1/dms/:id/metadata). In a variant of the same problem, a stored config with file paths inside mtls_options.cert_content or mtls_options.key_content makes every full settings update (POST /v1/dms or PUT /v1/dms/:id) fail validation; settings are replaced wholesale, so resend the config with the certificate and key as PEM.

Receiver certificate verification. A config with validate_server_cert: false becomes insecure_skip_verify: true, so receiver verification stays skipped until that field is set to false. Correct it on every migrated config: the webhook authorization channel carries the API key or bearer token, and skipping verification exposes those credentials to a network-positioned attacker. Keep insecure_skip_verify: true only when the receiver certificate is deliberately unverifiable.

Re-enrollment trust set

The flat field reenrollment_settings.additional_validation_cas moves into the nested field reenrollment_settings.est_rfc7030_settings.client_certificate_settings.validation_cas:

{
  "reenrollment_settings": {
    "est_rfc7030_settings": {
      "client_certificate_settings": {
        "validation_cas": ["<CA_ID>"]
      }
    }
  }
}

The effective trust set does not change, because the nested value was never consulted when validating re-enrollments. If the flat list was missing or null, it becomes an empty list.

What changes is how API requests are handled. Bodies that still carry additional_validation_cas are accepted, but the value is silently dropped because unknown JSON fields are ignored, and GET responses no longer include it. An update body built from the old layout therefore empties validation_cas without any error. Always read the DMS before sending an edit and reuse the layout the API returns.

Independent re-enrollment authentication

Re-enrollment authentication settings are now stored independently from initial authentication. Nothing needs migrating, but read both blocks before editing a DMS so that you do not modify one while believing you are editing the other. The console exposes independent editors for both blocks; the DMS guide describes their fields, and you can also edit them through the DMS API.

DMS update API

PUT /v1/dms/:id takes the DMS ID from the URL and validates the request body. An ID in the body is ignored, so update clients that relied on it.

Certificate type values

Stored certificate type values are renamed: IMPORTED becomes IMPORTED_WITH_KEY and EXTERNAL becomes IMPORTED_WITHOUT_KEY. MANAGED is unchanged. Adapt consumers that read or match on those values.

Device events

The events field no longer appears in the device representation. Point consumers at the new endpoints:

  • GET /v1/devices/:id/events returns a device's events.
  • POST /v1/devices/:id/events creates an event.

When the request Accept header includes text/event-stream, the events endpoint streams over SSE with a keepalive every 15 seconds, and returns 501 when SSE is disabled in the deployment. Each Device Manager instance binds a durable AMQP queue named after its service ID, device-manager-sse-<uuid>. Queues from crashed or replaced instances linger until removed, so configure an expiration or cleanup policy on the broker.

CRL signer field

crl_options.subject_key_id_signer is replaced by the required crl_options.signer_serial_number, the serial number of the existing certificate that signs the CRL. A new CA's initial value is the serial number of its current certificate. Requests that still carry subject_key_id_signer return an error. The authorization entity that holds these settings is also renamed; see CRL settings keyed by CA ID.

CRL settings keyed by CA ID

In 3.8.0 the VA keyed each CRL by the CA's SKI, and GET/PUT /v1/roles/{ca-ski} read and updated it. In 4.0.0 the CRL settings of a CA are keyed by its CA ID, and every CA has its own, even when two CAs share a key:

RouteOperation
GET /v1/crl-settingsLists the CRL settings; filterable at least by ca_id, status and legacy_ski.
GET /v1/crl-settings/{ca-id}Reads the settings of a CA.
PUT /v1/crl-settings/{ca-id}Replaces crl_options.
DELETE /v1/crl-settings/{ca-id}Removes the archived settings of a CA.

The options keep their meaning and their defaults: 7 days of validity, 6 days and 23 hours of refresh_interval, and regenerate_on_revoke true. status (ACTIVE, FINAL or ARCHIVED), legacy_ski and latest_crl are read-only. A PUT validates the signer only when it changes, does not regenerate the CRL, and returns 409 when the settings are not ACTIVE.

The public endpoint GET /crl/{id} takes the CA ID — the form the CDPs of newly issued certificates use — and keeps resolving the SKI form of certificates issued before the upgrade. Certificates issued before the upgrade keep their /crl/<ski> distribution points; newly issued ones carry /crl/<ca-id>, and both resolve to the same settings. The VA scheduled CRL job requests a new list on its first run, and the new list includes both URLs in its Issuing Distribution Point. Until it succeeds, the old list has only the legacy /crl/<ski> IDP: verifiers that compare it with the CDP of newly issued certificates cannot validate those certificates. The default schedule can delay the first run by up to about an hour; failures can extend the window.

The VA migration is irreversible

The migration that turns VA roles into CRL settings cannot be rolled back: its rollback does nothing. CRLs generated after the upgrade are stored under the CA ID, where 3.8.0 does not look for them, and the archived settings of CAs deleted before the upgrade have no signer to go back to. Returning to 3.8.0 requires restoring both the database and the CRL bucket — the object storage that holds the published lists — from the backups taken before the upgrade.

The migration aborts, and the upgrade fails, on five kinds of data:

  • CA IDs containing /: they cannot be used in CRL URLs. Recreate those CAs with a different ID.
  • An SKI shared by more than one CA: the error names the SKIs and asks you to recreate all but one of them on a different key.
  • A VA role whose signer certificate has no key: point the role to a signer with a key, or fix the key reference of the certificate, before upgrading.
  • A CA whose ID equals the legacy-<ski> ID the migration reserves for the archived settings of another role whose CA was deleted: recreate that CA with a different ID.
  • A CA ID equal to the SKI of another VA role: their CRL URLs would be ambiguous.

The settings of a CA deleted before the upgrade are migrated as ARCHIVED, with a legacy-<ski> ID and no signer: their last CRL is still served, but never regenerated, and PUT returns 409. There is no API way back to ACTIVE. The CRL guide describes the full behavior.

Custom authorization policies. The entity va_role is renamed to crl_settings, with ca_id as its primary key and signer_serial_number as its relation to certificates. Update policies referencing the old entity. The entity gains the delete atomic action, a signer_key_id relation to KMS keys, and the filterable columns signer_engine_id and signer_key_id; grant the new action to whoever may delete archived CRL settings. Replace the 3.8.0 signer filter crl_subject_key_id_signer (or crl_signer_certificate_serial_number if used by an intermediate version) with signer_serial_number. Replace SKI filter values with the corresponding certificate serial numbers; renaming the filter alone is insufficient.

Renamed VA events

va.role.update becomes va.crl-settings.update and va.role.crl.init becomes va.crl-settings.init; va.role.crl.create keeps its name and its payload. Every subject changes from crl/<ski> to crl/<ca-id>, and the payload of the renamed events changes from the old VA role to the CRL settings of the CA, with ca_id instead of ca_ski. The alerts migration renames the event types of existing subscriptions automatically, but it does not rewrite their conditions — JSON Path, JSON Schema or JavaScript: review conditions that filter on the old payload or on the old crl/<ski> subject, including those for va.role.crl.create. Direct event-bus consumers and audit searches must adopt the new names and payload themselves.

CA IDs are always assigned by the backend

Creating or importing a CA registers it under a backend-generated UUID. Any id sent in the request body is ignored, and the field is removed from the create and import bodies. Scripts that supplied their own IDs and addressed CAs by them must read the ID the backend returns instead.

VA CRL signer

The VA migration resolves each role's signer from its stored SKI. It first looks for CA certificates with that SKI; if several match, it selects the oldest CA and logs a warning. If no CA certificate matches, it looks among all certificates with that SKI. The migration aborts with cannot resolve signer certificate for VA role <ca-ski>, SKI <ski> if that fallback finds zero or multiple candidates. Fix affected roles before upgrading.

Helm migration guide

The upgrade runs in a fixed order: prepare the data and values, run helm upgrade, then copy the KMS and VA data into the new volumes. The copy must run after the upgrade, because the destination StatefulSets and PVCs do not exist before it. If you are not upgrading an existing release, go straight to Install with Helm.

Step 1: Prepare

  1. Take a verifiable backup of PostgreSQL and of the CRL object storage — the VA bucket, or the local volume that holds the CRLs — and schedule downtime. The upgrade moves every service's data into pki, and the VA migration is irreversible: returning to 3.8.0 needs both backups (see CRL settings keyed by CA ID).
  2. If the deployment runs on SQLite, couch_db or dynamo_db, move it to PostgreSQL first.
  3. Resolve duplicate CA IDs and backfill engine_id in kms_keys, as described in PostgreSQL consolidation.
  4. Optionally, run the pre-upgrade check against the consolidated 3.8.0 data before applying the migrations.
  5. Record the names of your local-storage PVCs:
export NAMESPACE=lamassu   # your 3.8.0 namespace
export RELEASE=lamassu     # your Helm release name

kubectl get pvc -n $NAMESPACE -l app.kubernetes.io/instance=$RELEASE
# on a 3.8.0 install you can also list the fixed names directly:
kubectl get pvc -n $NAMESPACE golang-engine-storage-kms-0 local-crl-file-storage-va-0

helm upgrade does not touch or delete the old PVCs, but nothing references them after the upgrade. You need their names to copy the data and to clean up later.

Pre-upgrade check

The 4.0.0 database migrations refuse data they cannot migrate safely: they abort and name the offending objects. The upgrade-check tool reports all of those problems before the upgrade, in one run, with the same wording the migration would use. It is read-only: it never writes database rows or keys.

  • Image: ghcr.io/lamassuiot/lamassu-upgrade-check. Configure it with a KMS-style YAML file passed through LAMASSU_CONFIG_FILE; only the postgres storage provider is supported.
  • Run it on the 3.8.0 data after it has been consolidated into the pki schemas, and before any 4.0.0 migration runs or any 4.0.0 service starts.
  • Exit codes: 0 means no problems were found, 1 means at least one problem was found, and 2 means the command could not run (bad configuration, unreachable database, or a database that is not a 3.8.0 one), or a check could not run.
  • It covers the VA migrations only: an all-clear does not prove that the CA and KMS migration guards pass, so it does not replace the duplicate CA ID and engine_id checks described in PostgreSQL consolidation.

Step 2: Update the values file

Add services.authz. Configure the JWKS endpoint, the database mappings and the bootstrap principal:

services:
  authz:
    jwkUrl: https://idp.example.com/realms/lamassu/protocol/openid-connect/certs
    credentials:
      pki:
        database: pki
      authz:
        database: authz
    bootstrap:
      - principal_id: "oidc:pki-admin"
        principal_name: "PKI Admin"
        principal_type: "oidc"
        policy_ids: [...]   # keep the defaults from values.yaml
        auth_config:
          claims:
            - claim: realm_access.roles   # was auth.authorization.rolesClaim
              operator: contains
              value: pki-admin            # was auth.authorization.roles.admin
  • jwkUrl is fetched from the authz pod, not from the gateway, so a Service in the release namespace can be referenced by its short name. The default points at a Keycloak Service named auth-keycloak; set it explicitly if your identity provider lives anywhere else.
  • authz is the authorization service's own storage for principals, grants and policies. pki is the database the authorization engine queries when it evaluates policies against entities. Only the database field of each entry is read; the connection always uses the global postgres.hostname, postgres.port, postgres.username and postgres.password values.
  • If you kept the 3.8.0 role defaults (realm_access.roles and pki-admin), the default services.authz.bootstrap already matches them.

Delete the gateway token validation blocks. Remove auth.oidc.apiGateway and auth.authorization once their settings are in services.authz:

# REMOVE (3.8.0)
auth:
  oidc:
    apiGateway:
      jwks:
        - name: oidc-authn
          uri: https://idp.example.com/realms/lamassu/protocol/openid-connect/certs
  authorization:
    rolesClaim: realm_access.roles
    roles:
      admin: pki-admin

Rename toolbox to connectivityTest:

# OLD (3.8.0)
toolbox:
  image: ghcr.io/lamassuiot/toolbox:2.2.0

# NEW (4.0.0)
connectivityTest:
  image: curlimages/curl@sha256:58adaa4e8dca9c988bae2aba4ab3434a0bb2da16bbe3f92dec39ec7785166777

Remove the per-service identity overrides services.kms.securityContext.runAsGroup and services.authz.securityContext.runAsUser/runAsGroup.

Convert services.connectors to a map keyed by connector ID, each entry with its type and image:

services:
  connectors:
    aws.myconnector:
      type: awsiot
      image: ghcr.io/lamassuiot/lamassu-aws-connector:dev-v4

Check AMQP. If you set amqp.tls: true, also set amqp.port to the broker's TLS listener (commonly 5671), or set amqp.tls: false to stay on plaintext.

Enable the CA-to-KMS migration if your CA keys predate the KMS:

migrations:
  caToKms:
    enabled: true

Review the remaining values: overrides that pinned services.ui.port to 80, the old probes: block, the TLS issuer, and explicit affinity and topologySpreadConstraints where your cluster needs them. Renamed and reshaped values describes each one.

Run helm lint with the new values before upgrading; the strict schema reports any key that is still unknown.

Step 3: Run the upgrade

helm upgrade --install "$RELEASE" lamassu/lamassu \
  --namespace $NAMESPACE \
  --values values.yaml \
  --wait

Helm runs the pre-upgrade hooks — the CA-to-KMS migration when enabled, and the database job — before applying any workload, so services never start against an unprepared database. If validation rejects unknown values, fix the values file and re-run.

If the deployment uses local KMS or VA storage, those pods fail until you complete step 4, so --wait cannot succeed: Helm waits for them to become ready, times out, exits with an error and marks the release failed. The hooks have already run by then, and every manifest, including the new StatefulSets and their PVCs, has been applied. Helm does not roll anything back without --atomic. Continue with step 4, then re-run the same helm upgrade command so it completes and Helm records the release as deployed. Both hooks are idempotent, so they are safe to run again.

Step 4: Migrate the KMS and VA data

Skip this step when services.kms.cryptoEngines never used a filesystem engine and services.va.fileStore.type was never local.

  1. Find the new StatefulSets and PVCs. Their names depend on the release name and any nameOverride or fullnameOverride; the PVC name is <volumeClaimTemplate>-<StatefulSet name>-0:
KMS_STS=$(kubectl get statefulset -n "$NAMESPACE" \
  -l "app.kubernetes.io/instance=$RELEASE,app.kubernetes.io/component=kms" \
  -o jsonpath='{.items[0].metadata.name}')
VA_STS=$(kubectl get statefulset -n "$NAMESPACE" \
  -l "app.kubernetes.io/instance=$RELEASE,app.kubernetes.io/component=va" \
  -o jsonpath='{.items[0].metadata.name}')

KMS_PVC="golang-engine-storage-${KMS_STS}-0"
VA_PVC="local-crl-file-storage-${VA_STS}-0"

# Fails if a lookup came back empty or a new claim is the old one.
# Item 3 calls it again before creating the helper pod.
claims_ok() {
  local pair sts new old
  for pair in "$KMS_STS:$KMS_PVC:golang-engine-storage-kms-0" \
              "$VA_STS:$VA_PVC:local-crl-file-storage-va-0"; do
    IFS=: read -r sts new old <<< "$pair"
    if [ -z "$sts" ] || [ "$new" = "$old" ]; then
      echo "ABORT: new claim '$new' is missing or equals the 3.8.0 claim '$old'" >&2
      return 1
    fi
  done
}

claims_ok && kubectl get pvc -n "$NAMESPACE" \
  golang-engine-storage-kms-0 "$KMS_PVC" \
  local-crl-file-storage-va-0 "$VA_PVC"

With release lamassu and the default chart name, the new PVCs are golang-engine-storage-lamassu-kms-0 and local-crl-file-storage-lamassu-va-0; with release prod they are golang-engine-storage-prod-lamassu-kms-0 and local-crl-file-storage-prod-lamassu-va-0. Confirm both old and new PVCs exist before continuing. If claims_ok prints ABORT, the block stops before listing the claims: either a StatefulSet lookup failed, or a new claim resolves to the 3.8.0 claim. In the second case the copy would verify against itself and the final cleanup would delete the live volume. The chart always appends -kms and -va to the release-scoped name, so a collision means the claims are not what this guide expects; investigate before going on. Item 3 runs the same check and refuses to create the helper pod, so nothing touches the volumes. The check returns instead of calling exit, which would close your shell and lose the variables set so far. If you only migrate one of the two services, drop the other one's claims from the commands that follow.

  1. Record the replica counts, scale the new StatefulSets down and wait for every pod that uses the claims to exit, so nothing writes to either volume while you copy. Helm removes the 3.8.0 kms and va StatefulSets during the upgrade, but their pods kms-0 and va-0 can still be terminating and holding the old claims. While they run, the helper pod either cannot attach the claims or copies a volume that is still being written. The wait returns at once for pods that are already gone:
KMS_REPLICAS=$(kubectl get statefulset -n "$NAMESPACE" "$KMS_STS" -o jsonpath='{.spec.replicas}')
VA_REPLICAS=$(kubectl get statefulset -n "$NAMESPACE" "$VA_STS" -o jsonpath='{.spec.replicas}')

kubectl scale statefulset -n "$NAMESPACE" "$KMS_STS" "$VA_STS" --replicas=0

kubectl wait -n "$NAMESPACE" --for=delete --timeout=300s \
  pod/kms-0 pod/va-0 "pod/${KMS_STS}-0" "pod/${VA_STS}-0"
  1. Run a short-lived helper pod that mounts all four claims. It runs as root so it can change file ownership, so pin its image by digest and do not mount the service-account token:
claims_ok && cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: pvc-migrate-helper
  namespace: $NAMESPACE
spec:
  restartPolicy: Never
  automountServiceAccountToken: false
  containers:
    - name: migrate
      image: docker.io/library/alpine@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc # 3.20
      command: ["sh", "-c", "sleep 3600"]
      volumeMounts:
        - {name: old-kms, mountPath: /old/kms}
        - {name: new-kms, mountPath: /new/kms}
        - {name: old-va,  mountPath: /old/va}
        - {name: new-va,  mountPath: /new/va}
  volumes:
    - {name: old-kms, persistentVolumeClaim: {claimName: golang-engine-storage-kms-0}}
    - {name: new-kms, persistentVolumeClaim: {claimName: $KMS_PVC}}
    - {name: old-va,  persistentVolumeClaim: {claimName: local-crl-file-storage-va-0}}
    - {name: new-va,  persistentVolumeClaim: {claimName: $VA_PVC}}
EOF

kubectl wait -n $NAMESPACE --for=condition=Ready pod/pvc-migrate-helper --timeout=60s
  1. Inspect before writing so you do not clobber data the new pods may already have written:
kubectl exec -n $NAMESPACE pvc-migrate-helper -- sh -c 'find /new/kms /new/va -type f'

An empty result is the common case: a StatefulSet's own pod rarely gets far enough to write anything when its dependent data is missing. If the output is not empty, record the files and confirm with the service owner that they are disposable before you run the copy; do not overlay the 3.8.0 data onto an unknown store.

  1. Copy the data, align ownership and verify the copy. The second command must print copy verified; any diff output means the copy is incomplete, so do not continue:
kubectl exec -n $NAMESPACE pvc-migrate-helper -- sh -c '
  cp -a /old/kms/. /new/kms/ &&
  cp -a /old/va/. /new/va/ &&
  chown -R 65532:65532 /new/kms /new/va
'

kubectl exec -n $NAMESPACE pvc-migrate-helper -- sh -c '
  diff -r /old/kms /new/kms &&
  diff -r /old/va /new/va &&
  echo "copy verified"
'

The chown is required even if the old files already look non-root-owned, because 4.0.0 requires exactly 65532:65532 and does not fall back to whatever the image declares.

  1. Remove the helper and restore the recorded replica counts:
kubectl delete pod -n $NAMESPACE pvc-migrate-helper
kubectl scale statefulset -n "$NAMESPACE" "$KMS_STS" --replicas="$KMS_REPLICAS"
kubectl scale statefulset -n "$NAMESPACE" "$VA_STS" --replicas="$VA_REPLICAS"
  1. Re-run the step 3 helm upgrade command. The KMS and VA pods now start against the copied data, so --wait succeeds and the release moves from failed to deployed.

Step 5: Remove the one-off migration values

After the upgrade that performed the CA-to-KMS migration succeeds, remove migrations.caToKms.enabled from your values (it defaults to false). The hook reads the legacy 3.8.0 kms-config ConfigMap while it still exists and falls back to the release-scoped <release>-lamassu-kms-config once it is gone.

Then go through the validation checklist. Keep the old PVCs until it passes.

Deployments without Helm

Without the chart's database job, no tool moves data between databases: the migration tooling only applies migrations to a schema that already exists. Move each per-service database into its pki schema yourself, preserving structure, indexes, constraints, data and migration history, with writes stopped and a verified backup in hand.

Run the migrations in the required order, one schema at a time: kms first, then the ca-to-kms-migration tool (needed only for legacy deployments whose CA keys are named by SKI), then ca, then va. The devicemanager, dmsmanager and alerts schemas are independent and can be migrated in any order. With Helm, the chart's pre-install job applies the schemas in the correct order automatically.

If kms has not been migrated, the ca migration fails with kms schema not found: migrate the kms schema before the ca schema. The va migration requires ca and kms to be migrated first: it fails with ca schema not found: migrate the ca schema before the va schema or kms schema not found: migrate the kms schema before the va schema if one of them is missing.

The schemas reference each other, so the PostgreSQL role running the migrations needs cross-schema privileges: the ca schema needs USAGE on kms plus SELECT, REFERENCES and INSERT on kms.kms_keys, and the va schema needs USAGE on ca plus SELECT and REFERENCES on ca.certificates, SELECT on ca.ca_certificates, and USAGE on kms plus SELECT and REFERENCES on kms.kms_keys. Without them the migration aborts; grant the missing privileges and rerun the failed schema. After migrating ca, if the CA service runs as a different user than the migration user, grant it SELECT on ca.certificates_view (GRANT SELECT ON ca.certificates_view TO service_user). Without this grant, certificate and CA reads fail with permission denied for table certificates_view. After migrating va, if the CA service runs as a different user, grant it USAGE on the va schema and SELECT on va.crl_settings (GRANT USAGE ON SCHEMA va TO service_user; GRANT SELECT ON va.crl_settings TO service_user): the CA service reads va.crl_settings when it deletes a CA, and without these grants every deletion of a CA that has a crypto engine fails with permission denied for schema va (or for table crl_settings).

To apply migrations by hand, goose-lamassu must run against the pki database with search_path=<schema> in the connection string. The binary validates only search_path, not the database name, so check dbname=pki before running it, and repeat for kms, ca, va, devicemanager, dmsmanager and alerts:

./goose-lamassu "host=$DB_HOST port=5432 user=$DB_USER dbname=pki search_path=kms" up

Run ca-to-kms-migration first with --dry-run, which reports without writing to KMS storage, and then for real:

LAMASSU_CONFIG_FILE=migrate.yaml ./ca-to-kms-migration --dry-run
LAMASSU_CONFIG_FILE=migrate.yaml ./ca-to-kms-migration

Point the tool's storage configuration at the pki database, where the CA and KMS data now live in the ca and kms schemas. The tool's options are described in backend/cmd/ca-to-kms-migration/README.md in the lamassuiot repository.

Both tools' READMEs document the 4.0.0 pki layout and match this page: goose-lamassu uses dbname=pki examples and does not validate the database name, and the ca-to-kms-migration README describes the single pki database and the migration order.

Troubleshooting

  • KMS or VA pods crash with no such file or directory or code=NotFound right after the upgrade. The pods are bound to the new, empty PVCs and the data migration has not run yet, or was copied into the wrong PVC. Follow step 4 and check that all four claims exist.
  • helm upgrade --wait times out and helm status reports the release as failed. With local KMS or VA storage this is expected on the first run: the new pods cannot become ready before their data is copied. The hooks ran and the manifests and PVCs were applied, so complete step 4 and re-run the same helm upgrade command. If other pods are also not ready, check their logs, because something else is wrong.
  • The helper pod stays Pending or ContainerCreating and kubectl wait times out. Another pod still holds one of the claims; on block storage kubectl describe pod pvc-migrate-helper shows a Multi-Attach error. Check with kubectl get pods -n $NAMESPACE that kms-0, va-0 and the new KMS and VA pods are gone, then re-run the wait.
  • helm upgrade fails with an unknown-key validation error. Remove or rename the legacy values — typically toolbox, auth.oidc.apiGateway or auth.authorization — because the strict schema rejects keys it does not know.
  • The upgrade fails at the database job. The PostgreSQL role cannot create databases or schemas, or a schema migration aborted: on duplicate CA IDs or missing engine_id values, or in the VA schema on a CA ID containing /, an SKI shared by more than one CA (the error asks you to recreate all but one of them on a different key), a role whose signer certificate has no key, a CA ID reserved for the archived settings of another role, or a CA ID equal to another role's SKI. The abort message names the offending objects; fix them and re-run the upgrade.
  • The upgrade fails at the CA-to-KMS hook. CA certificates that share a subject_key_id declare conflicting engine_ids. Resolve the conflict and re-run.
  • Nobody can log in after the upgrade. Check that services.authz.jwkUrl is reachable from the authz pod, that services.authz.bootstrap matches a role your OIDC provider actually assigns, and that the database job completed.
  • A migration aborts with kms schema not found. The schemas must be migrated in order: run the kms schema migration before the ca and va schema migrations.
  • A migration aborts with ca schema not found. Run the ca schema migration before the va schema migration.
  • The VA migration fails with cannot resolve signer certificate for VA role ... N candidates. The migration cannot uniquely identify a signer certificate from the SKI stored on the role. Review the SKI and candidate certificates reported in the error; correct the pre-migration data with a verified backup and rerun the migration. The new signer_serial_number field is created during that migration, so it cannot be set through the VA API while the migration is failing.
  • A migration or service fails with permission denied for table ... or permission denied for schema .... The PostgreSQL role is missing the cross-schema privileges listed in Deployments without Helm; this includes certificates_view when the post-migration GRANT SELECT was skipped, and permission denied for schema va (or for table crl_settings) when deleting a CA without the va grants. Grant the missing privileges and re-run.

Validation checklist

After the upgrade, check in this order:

  1. Databases. Each service reads the pki database with its own schema, authz and wfx data are reachable, and the migration status is registered per schema, including the CA unique-ID and KMS composite-key migrations and the foreign-key migrations that run per schema in the required order.
  2. KMS and VA start cleanly. Their logs show no no such file or directory or code=NotFound errors.
  3. Existing KMS keys work. Issue a certificate from a CA that existed before the upgrade. None of the other checks reads a KMS key, so this is the only one that proves the key material was copied.
  4. Existing CRLs are served. GET /crl/<ca-id> on VA returns a previously issued CRL, and GET /crl/<ca-ski> with the legacy SKI of a certificate issued before the upgrade also resolves. The VA scheduled CRL job has issued a new list under the CA ID with both URLs in its IDP; check its logs if regeneration has not succeeded.
  5. Renamed VA events. Alert subscriptions to va.role.update and va.role.crl.init now subscribe to va.crl-settings.update and va.crl-settings.init, and conditions for all VA events, including va.role.crl.create, have been reviewed against the new payload (ca_id instead of ca_ski) and subject (crl/<ca-id> instead of crl/<ski>). Direct consumers and audit searches use the new event names.
  6. Legacy CA keys. If the CA-to-KMS migration ran, review its report and warnings and confirm the expected keys appear in the KMS. Migrated CA certificates now report key_id and has_private_key, which you can use to confirm the registration.
  7. Connectivity. helm test "$RELEASE" -n $NAMESPACE --logs passes; the hook checks the CA, DMS Manager, Device Manager and VA health endpoints and the UI response.
  8. Authorization. Sign in as the bootstrap principal and confirm administration works, check that an identity without permissions receives 403, and review client and integration permissions.
  9. AMQP. The backend connects to the broker with the expected TLS settings, and the cleanup policy for SSE queues is in place.
  10. Device events. GET /v1/devices/:id/events returns the historical events of a device that had them.
  11. Webhooks without file fields. For each DMS that uses a webhook for enrollment or re-enrollment, the config block carries insecure_skip_verify and the expected option block (apikey_options, jwt_options or mtls_options), with the certificate and key as PEM and no cert_file or key_file.
  12. Receiver verification. insecure_skip_verify is false, unless the receiver certificate is intentionally unverified.
  13. Webhook delivery. One enroll or re-enroll through each migrated webhook completes its authorization call.
  14. Trust set. est_rfc7030_settings.client_certificate_settings.validation_cas holds the expected CAs, in particular those of the old CA during rotation overlaps.
  15. API clients. Scripts and consoles no longer use the old DMS layout (flat additional_validation_cas, or the legacy mtls, apikey and oidc fields), send DMS updates with the ID in the URL, and match on the new certificate type values.

Delete the old volumes

Only once every check passes, delete the orphaned 3.8.0 volumes to reclaim storage. The delete is irreversible and the old KMS volume may hold the last copy of your CA key material, so take a VolumeSnapshot or a backup of both old PVCs first and keep it until you are confident the upgraded deployment is healthy. These are the fixed 3.8.0 names recorded in step 1; before deleting, confirm that neither is mounted by a running pod, for example with kubectl describe pvc -n $NAMESPACE golang-engine-storage-kms-0 local-crl-file-storage-va-0, where Used By must be <none>:

kubectl delete pvc -n $NAMESPACE golang-engine-storage-kms-0 local-crl-file-storage-va-0

The 3.8.0 per-service databases are no longer used either. Drop them only after the same checks pass and your PostgreSQL backup is verified.

On this page