3.8.0 to 4.0.0
What changes in the 4.0.0 architecture, which changes break a 3.8.0 deployment, how to upgrade a Helm release step by step and how to validate the result.
Lamassu 4.0.0 is a major release. Storage consolidates into a single PostgreSQL database, a new authz service authorizes every request, the Helm chart is hardened, and several APIs and stored configurations change shape. Most data converts automatically during the upgrade, but some changes need operator action before, during or after it.
This page is organized in four parts. Read them in order before you upgrade production:
- Architecture changes: what is different in 4.0.0 and what the upgrade migrates on its own.
- Breaking changes: every change that can break a 3.8.0 deployment, who it affects and what to do.
- Helm migration guide: the step-by-step procedure for a Helm-managed release.
- Validation checklist: how to confirm the upgrade worked before you remove the old data.
Backend migration steps are provisional
The 4.0.0 release is upcoming and the backend migration steps on this page are provisional. Verify each procedure against the final release artifacts before upgrading production.
Architecture changes
One PostgreSQL database with schemas
3.8.0 gave each service its own database. The 4.0.0 backend connects to a single PostgreSQL database named pki and isolates each service in a schema: kms, ca, va, devicemanager, dmsmanager and alerts. The authz service keeps its principals, grants and policies in a separate authz database (the development monolith uses an authz schema inside pki instead), and WFX keeps its own wfx database.
PostgreSQL is now the only storage engine. SQLite support is removed, and the couch_db and dynamo_db engines are not registered in 4.0.0.
Authorization moves into the authz service
In 3.8.0 the API Gateway validated tokens: a JWT policy on every private route checked the signature and only let through tokens that carried the administrator role. Every route was open to any token that passed that check.
4.0.0 removes the gateway policy. The /api/* routes reach the services directly, and each service authorizes every request through authz, which validates the token against the identity provider's JWKS endpoint and applies fine-grained, entity-level policies to CAs, certificates, devices, DMSs, KMS keys and CRL settings. There is no implicit superuser: the first administrator comes from a bootstrap principal that you configure. The access control page documents principals, policies and the first administrator.
Device events get their own store
Historical device events move from the devices.events column to a dedicated device_events store with their own endpoints. The events endpoint can also stream events over Server-Sent Events (SSE); to fan events out across service instances, each Device Manager instance binds its own durable AMQP queue.
A hardened Helm chart
Version 4.0.0 of the chart is a security-hardening release:
- Release-scoped resource names. Every chart-managed resource is named
<release>-lamassu-<component>instead of the fixed names used in 3.8.0 (kms,va,ca, ...). With a release namedprodthe StatefulSets becomeprod-lamassu-kmsandprod-lamassu-va. ThenameOverrideandfullnameOverridevalues change the prefix. - Non-root containers. Every workload runs as the numeric user
65532:65532with aRuntimeDefaultseccomp profile and dropped capabilities. Pods do not mount the service-account token by default, and the chart creates a dedicated ServiceAccount. - Strict values validation.
values.schema.jsonrejects unknown keys, so leftover 3.8.0 blocks failhelm lintandhelm upgradeinstead of being silently ignored. - No injected scheduling rules. The chart no longer adds pod anti-affinity or topology spread constraints on its own.
What the upgrade migrates automatically
Several migrations run without intervention when you upgrade a Helm release:
- Before any service starts, the chart's pre-install/pre-upgrade job prepares PostgreSQL. It creates the
pki,authzandwfxdatabases when they do not exist, creates one schema per service insidepki, and imports each 3.8.0 per-service database into its schema. Each import runs as a single transaction, so a failed import leaves no partial schema and a retried job imports again. The job then runs the database migrations for every schema, migratesauthz, preloads its policies and seeds theservices.authz.bootstrapprincipals. It is idempotent and requires a PostgreSQL role allowed to create databases and schemas, as described in Install with Helm. - Stored DMS configuration is rewritten to the 4.0.0 layout, and historical device events are transformed into the new store.
- The KMS migration removes the stale
lamassu.io/kms/binded-resourceskey metadata, and the VA migration resolves each role's CRL signer (see VA CRL signer). - When
migrations.caToKms.enabledis set, a pre-upgrade hook registers legacy CA keys in the KMS.
Two things never migrate on their own: the KMS and VA data on local volumes, which stays on volumes the upgraded pods no longer use, and the values file, which you update by hand. Deployments that do not use the Helm chart also have to move their database data themselves; see Deployments without Helm.
Breaking changes
| Change | Who is affected | Severity | Action |
|---|---|---|---|
| Release-scoped resource names | Helm releases with a filesystem KMS engine or local VA CRL storage | High: KMS and VA start on empty volumes | Copy the data into the new volumes |
| Strict values validation | Every Helm release | High: helm upgrade fails | Remove or rename legacy keys |
| Required authorization configuration | Every deployment | High: without a bootstrap principal there is no administrator | Configure services.authz |
| Gateway token validation removed | Every Helm release | High: helm upgrade fails on the old keys | Move JWKS and roles into services.authz |
| PostgreSQL consolidation | 3.8.0 deployments with per-service databases | High: services find no data, or a migration aborts | Clean duplicate CA IDs and empty engine_id values first; move data manually outside Helm |
| SQLite storage removed | Deployments on SQLite, couch_db or dynamo_db | High: the backend no longer starts with them | Move to PostgreSQL before upgrading |
| CRL settings keyed by CA ID | Every deployment; API clients, custom policies and alert subscriptions are also affected | High: the /v1/roles routes are removed and the migration is irreversible | Fix the data the migration aborts on, and back up the database and the CRL bucket |
| Webhooks with file references | DMSs whose webhook mTLS certificate or key were file paths | High: enrollments calling the webhook fail | Re-enter certificate and key as PEM, or remove the file fields |
| Non-root containers | Releases with per-service identity overrides or mounted key material | Medium: services cannot read their files | Remove the overrides and fix volume ownership |
| AMQP over TLS | Releases that set amqp.tls | Medium: clients connect to the wrong port | Point amqp.port at the TLS listener, or disable TLS |
| Legacy CA keys in the KMS | Installations whose CA keys predate the KMS | Medium: CA keys stay unmanaged | Run the CA-to-KMS migration |
| Receiver certificate verification | DMSs whose webhook had validate_server_cert: false | Medium: credentials go to an unverified receiver | Set insecure_skip_verify to false |
| Re-enrollment trust set | Scripts and consoles that edit DMSs through the API | Medium: an old-layout edit silently empties validation_cas | Read the DMS before editing and use the new field |
| CRL signer field | Clients that create or update CRL settings through the API | Medium: requests with the old field return an error | Replace crl_options.subject_key_id_signer with the required crl_options.signer_serial_number |
| Renamed VA events | Alert subscriptions and consumers of the VA events | Medium: subscription types are renamed automatically, their conditions are not | Review conditions filtering on the old event payload or subject |
| Device events | Consumers that read the device events field | Medium: the field disappears | Use the new event endpoints |
| VA CRL signer | Deployments with VA roles whose signer cannot be resolved uniquely | Medium: the VA migration aborts | Fix the affected roles before upgrading |
| Renamed and reshaped values | Releases that override toolbox, services.connectors, ports, probes or scheduling | Low to medium: validation errors or changed defaults | Update the values file |
| DMS update API | Clients that send the DMS ID in the update body | Low: the ID in the URL wins | Send the ID in the URL |
| CA IDs are always assigned by the backend | Clients that send an id when creating or importing CAs | Low: the ID in the body is ignored, and the field is removed | Let the backend assign the ID and use the returned one |
| Certificate type values | Consumers that match on certificate type | Low: two values are renamed | Adapt consumers |
| Independent re-enrollment authentication | Any DMS | Low: informational | Read both blocks before editing |
Helm chart and values
Release-scoped resource names
Kubernetes derives a StatefulSet's PVC name from its pod name (<volumeClaimTemplate>-<pod-name>). Renaming the kms and va StatefulSets means helm upgrade binds brand-new, empty PVCs instead of the existing ones. The KMS key material and VA CRLs are not deleted — the old PVCs are orphaned, not removed — but the upgraded pods fail with no such file or directory (KMS) or code=NotFound (VA) until you copy the data across. This affects KMS when services.kms.cryptoEngines uses a filesystem engine and VA when services.va.fileStore.type is local. Every other renamed resource is stateless and is recreated without data-loss risk.
Step 4 of the migration guide copies the data.
Strict values validation
values.schema.json rejects unknown keys, so every block that 4.0.0 removed or renamed — toolbox, auth.oidc.apiGateway, auth.authorization and others — fails helm lint and helm upgrade outright. Step 2 lists every change.
Required authorization configuration
3.8.0 had no authorization service, so a 3.8.0 values file has nothing to migrate from: you must add the services.authz configuration yourself. It needs the JWKS endpoint of your identity provider, the database mappings the authorization engine uses, and a bootstrap principal that matches how your provider assigns the administrator role. Without a matching principal nobody can administer the PKI.
Gateway token validation removed
auth.oidc.apiGateway.jwks, auth.authorization.rolesClaim and auth.authorization.roles.admin are removed with no compatibility aliases, and the schema rejects them. Their settings move into services.authz: the JWKS URL becomes services.authz.jwkUrl, and the role claim and administrator role become a claim rule of the bootstrap principal. auth.oidc.frontend is unchanged.
Non-root containers
Service images run on a distroless base as the numeric user 65532; the authz image is Ubuntu with a non-root user that has the same UID. The per-service overrides services.kms.securityContext.runAsGroup: 0 and services.authz.securityContext.runAsUser/runAsGroup: 999 are removed; group access to the KMS volume and PKCS#11 socket is now handled by the pod-level fsGroup: 65532.
3.8.0 did not pin a container identity, so existing files can carry any UID. Any volume that holds keys or CRLs must be owned by 65532:65532, or at least readable (and, where needed, writable) by that identity.
AMQP over TLS
In 3.8.0 amqp.tls was not applied and event-bus clients always connected over plaintext. In 4.0.0 amqp.tls: true switches them to AMQPS. The default amqp.port stays 5672 and does not change when TLS is enabled, so a values file that already set amqp.tls: true must point amqp.port at the broker's TLS listener (amqps, commonly 5671), or set amqp.tls: false to keep plaintext. Clients must also trust the broker's certificate.
Renamed and reshaped values
toolboxis renamed toconnectivityTest, and thehelm testhook uses the upstreamcurlimages/curlimage pinned by digest.services.connectorsis a map keyed by connector ID instead of a list.- The
services.ui.portdefault is8085, the unprivileged port of the rootless image. - The shared
probes:block is replaced by independentlivenessProbe,readinessProbeandstartupProbeblocks. - With
tls.certManagerOptions.issuerempty, the chart creates a release-scoped self-signed issuer, for examplelamassu-downstream-ca-selfsigned-issuer. The legacy literaldownstream-ca-selfsigned-issueris still accepted and resolves to it. - Anti-affinity and topology spread must be configured explicitly through
affinityandtopologySpreadConstraints.
Data and storage
PostgreSQL consolidation
Every 3.8.0 per-service database must end up as a schema of pki, keeping its structure, indexes, constraints, data and migration history. The Helm chart does this automatically; deployments without Helm do it by hand (see Deployments without Helm).
Two schema migrations abort on data that 3.8.0 allowed, so clean the data before upgrading:
- The CA migration adds a UNIQUE constraint on
ca_certificates.idand aborts if duplicate CA IDs exist. Imports already reject duplicates with409, so new ones cannot normally appear. - The KMS migration promotes the primary key to
(key_id, engine_id)and aborts withcannot promote (key_id, engine_id) to primary key: N row(s) have no engine_id; backfill them before migratingwhenkms_keysrows lackengine_id. Backfillengine_idfirst.
The same KMS migration deletes, with a NOTICE, aliases that equal an existing key ID, because identifiers now resolve as a key ID before an alias. Affected aliases are dropped, not redirected.
SQLite storage removed
The SQLite storage engine is removed, and the couch_db and dynamo_db engines are not registered in 4.0.0: their configuration constants remain, but they select no engine. Deployments on any of them must move to PostgreSQL and consolidate their data into the pki schemas before upgrading.
Legacy CA keys in the KMS
CA certificates of type MANAGED or IMPORTED_WITH_KEY whose private keys predate the KMS service have the key stored only in the crypto engine, under the certificate's subject_key_id, with no KMS record. The ca-to-kms-migration tool registers them:
- The
key_idis the certificate'ssubject_key_id; it matches the SHA-256 digest of the public key only when the SKI was computed that way. - The tool is idempotent: keys already in the KMS are skipped, so it can be run several times.
- Certificates that cannot be read, or lack an engine or a subject key ID, are skipped with a warning. Review the logs and resolve the affected records.
- When CA certificates sharing a
subject_key_iddeclare conflictingengine_ids, the tool fails that key and exits with code1, which fails a Helm pre-upgrade hook. Resolve the conflict manually before rerunning.
Once the keys are registered, certificates reference them in the database and the KMS refuses to delete a key that a certificate still uses (409 key is in use). Deleting a certificate record removes it from the CRL and OCSP, so read Before deleting a key before freeing a key.
Optionally, installations with imported CAs from before 3.7 can start the KMS once with crypto_engines.migrate_keys_format: true: the KMS renames those SKI-named legacy keys to the SHA-256 digest of their public key (the old name is kept as an alias, and the certificate references follow automatically). Run it once and disable the option again; running it again is safe.
With Helm, enable the hook for the upgrade in step 2; without Helm, run the tool by hand.
References between certificates, keys and CRL settings
Certificates now reference their KMS key, and CRL settings reference their signer certificate, so a key or certificate in use cannot be deleted. The KMS migration drops the legacy lamassu.io/kms/binded-resources key metadata. Before 4.0.0, deleting a CA could remove its key while leaving key references in its retained certificate records. The CA migration registers those missing keys in the KMS with only their public key to satisfy the retained references. These entries cannot be deleted while the retained certificates reference them; they do not restore the private key. In 4.0.0, deleting a CA releases the references from its own current certificate and linked reissued certificates before attempting to delete the key; other certificates that still reference it prevent deletion.
APIs and stored configuration
DMS webhooks
Stored webhook configs are rewritten to the current layout and their fields renamed:
| Field in 3.8.0 | Field in 4.0.0 |
|---|---|
validate_server_cert | insecure_skip_verify (value negated) |
oidc | jwt_options |
apikey | apikey_options |
mtls | mtls_options |
When the legacy mTLS certificate and key are both PEM, they move into mtls_options.cert_content and mtls_options.key_content and need no action. Two cases do need it.
Webhooks with file references. A mixed config, with a PEM certificate and a file-path key, is migrated to mtls_options.cert_file and mtls_options.key_file. Webhook delivery rejects those fields on every call, whatever the configured authentication mode, so every enroll and re-enroll that calls that webhook fails until the config is fixed. If the webhook uses mTLS, re-enter the certificate and key as Base64-encoded or plain PEM; if it does not, remove the leftover file fields. Metadata-only updates keep working in the meantime (PUT or PATCH on /v1/dms/:id/metadata). In a variant of the same problem, a stored config with file paths inside mtls_options.cert_content or mtls_options.key_content makes every full settings update (POST /v1/dms or PUT /v1/dms/:id) fail validation; settings are replaced wholesale, so resend the config with the certificate and key as PEM.
Receiver certificate verification. A config with validate_server_cert: false becomes insecure_skip_verify: true, so receiver verification stays skipped until that field is set to false. Correct it on every migrated config: the webhook authorization channel carries the API key or bearer token, and skipping verification exposes those credentials to a network-positioned attacker. Keep insecure_skip_verify: true only when the receiver certificate is deliberately unverifiable.
Re-enrollment trust set
The flat field reenrollment_settings.additional_validation_cas moves into the nested field reenrollment_settings.est_rfc7030_settings.client_certificate_settings.validation_cas:
{
"reenrollment_settings": {
"est_rfc7030_settings": {
"client_certificate_settings": {
"validation_cas": ["<CA_ID>"]
}
}
}
}The effective trust set does not change, because the nested value was never consulted when validating re-enrollments. If the flat list was missing or null, it becomes an empty list.
What changes is how API requests are handled. Bodies that still carry additional_validation_cas are accepted, but the value is silently dropped because unknown JSON fields are ignored, and GET responses no longer include it. An update body built from the old layout therefore empties validation_cas without any error. Always read the DMS before sending an edit and reuse the layout the API returns.
Independent re-enrollment authentication
Re-enrollment authentication settings are now stored independently from initial authentication. Nothing needs migrating, but read both blocks before editing a DMS so that you do not modify one while believing you are editing the other. The console exposes independent editors for both blocks; the DMS guide describes their fields, and you can also edit them through the DMS API.
DMS update API
PUT /v1/dms/:id takes the DMS ID from the URL and validates the request body. An ID in the body is ignored, so update clients that relied on it.
Certificate type values
Stored certificate type values are renamed: IMPORTED becomes IMPORTED_WITH_KEY and EXTERNAL becomes IMPORTED_WITHOUT_KEY. MANAGED is unchanged. Adapt consumers that read or match on those values.
Device events
The events field no longer appears in the device representation. Point consumers at the new endpoints:
GET /v1/devices/:id/eventsreturns a device's events.POST /v1/devices/:id/eventscreates an event.
When the request Accept header includes text/event-stream, the events endpoint streams over SSE with a keepalive every 15 seconds, and returns 501 when SSE is disabled in the deployment. Each Device Manager instance binds a durable AMQP queue named after its service ID, device-manager-sse-<uuid>. Queues from crashed or replaced instances linger until removed, so configure an expiration or cleanup policy on the broker.
CRL signer field
crl_options.subject_key_id_signer is replaced by the required crl_options.signer_serial_number, the serial number of the existing certificate that signs the CRL. A new CA's initial value is the serial number of its current certificate. Requests that still carry subject_key_id_signer return an error. The authorization entity that holds these settings is also renamed; see CRL settings keyed by CA ID.
CRL settings keyed by CA ID
In 3.8.0 the VA keyed each CRL by the CA's SKI, and GET/PUT /v1/roles/{ca-ski} read and updated it. In 4.0.0 the CRL settings of a CA are keyed by its CA ID, and every CA has its own, even when two CAs share a key:
| Route | Operation |
|---|---|
GET /v1/crl-settings | Lists the CRL settings; filterable at least by ca_id, status and legacy_ski. |
GET /v1/crl-settings/{ca-id} | Reads the settings of a CA. |
PUT /v1/crl-settings/{ca-id} | Replaces crl_options. |
DELETE /v1/crl-settings/{ca-id} | Removes the archived settings of a CA. |
The options keep their meaning and their defaults: 7 days of validity, 6 days and 23 hours of refresh_interval, and regenerate_on_revoke true. status (ACTIVE, FINAL or ARCHIVED), legacy_ski and latest_crl are read-only. A PUT validates the signer only when it changes, does not regenerate the CRL, and returns 409 when the settings are not ACTIVE.
The public endpoint GET /crl/{id} takes the CA ID — the form the CDPs of newly issued certificates use — and keeps resolving the SKI form of certificates issued before the upgrade. Certificates issued before the upgrade keep their /crl/<ski> distribution points; newly issued ones carry /crl/<ca-id>, and both resolve to the same settings. The VA scheduled CRL job requests a new list on its first run, and the new list includes both URLs in its Issuing Distribution Point. Until it succeeds, the old list has only the legacy /crl/<ski> IDP: verifiers that compare it with the CDP of newly issued certificates cannot validate those certificates. The default schedule can delay the first run by up to about an hour; failures can extend the window.
The VA migration is irreversible
The migration that turns VA roles into CRL settings cannot be rolled back: its rollback does nothing. CRLs generated after the upgrade are stored under the CA ID, where 3.8.0 does not look for them, and the archived settings of CAs deleted before the upgrade have no signer to go back to. Returning to 3.8.0 requires restoring both the database and the CRL bucket — the object storage that holds the published lists — from the backups taken before the upgrade.
The migration aborts, and the upgrade fails, on five kinds of data:
- CA IDs containing
/: they cannot be used in CRL URLs. Recreate those CAs with a different ID. - An SKI shared by more than one CA: the error names the SKIs and asks you to recreate all but one of them on a different key.
- A VA role whose signer certificate has no key: point the role to a signer with a key, or fix the key reference of the certificate, before upgrading.
- A CA whose ID equals the
legacy-<ski>ID the migration reserves for the archived settings of another role whose CA was deleted: recreate that CA with a different ID. - A CA ID equal to the SKI of another VA role: their CRL URLs would be ambiguous.
The settings of a CA deleted before the upgrade are migrated as ARCHIVED, with a legacy-<ski> ID and no signer: their last CRL is still served, but never regenerated, and PUT returns 409. There is no API way back to ACTIVE. The CRL guide describes the full behavior.
Custom authorization policies. The entity va_role is renamed to crl_settings, with ca_id as its primary key and signer_serial_number as its relation to certificates. Update policies referencing the old entity. The entity gains the delete atomic action, a signer_key_id relation to KMS keys, and the filterable columns signer_engine_id and signer_key_id; grant the new action to whoever may delete archived CRL settings. Replace the 3.8.0 signer filter crl_subject_key_id_signer (or crl_signer_certificate_serial_number if used by an intermediate version) with signer_serial_number. Replace SKI filter values with the corresponding certificate serial numbers; renaming the filter alone is insufficient.
Renamed VA events
va.role.update becomes va.crl-settings.update and va.role.crl.init becomes va.crl-settings.init; va.role.crl.create keeps its name and its payload. Every subject changes from crl/<ski> to crl/<ca-id>, and the payload of the renamed events changes from the old VA role to the CRL settings of the CA, with ca_id instead of ca_ski. The alerts migration renames the event types of existing subscriptions automatically, but it does not rewrite their conditions — JSON Path, JSON Schema or JavaScript: review conditions that filter on the old payload or on the old crl/<ski> subject, including those for va.role.crl.create. Direct event-bus consumers and audit searches must adopt the new names and payload themselves.
CA IDs are always assigned by the backend
Creating or importing a CA registers it under a backend-generated UUID. Any id sent in the request body is ignored, and the field is removed from the create and import bodies. Scripts that supplied their own IDs and addressed CAs by them must read the ID the backend returns instead.
VA CRL signer
The VA migration resolves each role's signer from its stored SKI. It first looks for CA certificates with that SKI; if several match, it selects the oldest CA and logs a warning. If no CA certificate matches, it looks among all certificates with that SKI. The migration aborts with cannot resolve signer certificate for VA role <ca-ski>, SKI <ski> if that fallback finds zero or multiple candidates. Fix affected roles before upgrading.
Helm migration guide
The upgrade runs in a fixed order: prepare the data and values, run helm upgrade, then copy the KMS and VA data into the new volumes. The copy must run after the upgrade, because the destination StatefulSets and PVCs do not exist before it. If you are not upgrading an existing release, go straight to Install with Helm.
Step 1: Prepare
- Take a verifiable backup of PostgreSQL and of the CRL object storage — the VA bucket, or the local volume that holds the CRLs — and schedule downtime. The upgrade moves every service's data into
pki, and the VA migration is irreversible: returning to 3.8.0 needs both backups (see CRL settings keyed by CA ID). - If the deployment runs on SQLite,
couch_dbordynamo_db, move it to PostgreSQL first. - Resolve duplicate CA IDs and backfill
engine_idinkms_keys, as described in PostgreSQL consolidation. - Optionally, run the pre-upgrade check against the consolidated 3.8.0 data before applying the migrations.
- Record the names of your local-storage PVCs:
export NAMESPACE=lamassu # your 3.8.0 namespace
export RELEASE=lamassu # your Helm release name
kubectl get pvc -n $NAMESPACE -l app.kubernetes.io/instance=$RELEASE
# on a 3.8.0 install you can also list the fixed names directly:
kubectl get pvc -n $NAMESPACE golang-engine-storage-kms-0 local-crl-file-storage-va-0helm upgrade does not touch or delete the old PVCs, but nothing references them after the upgrade. You need their names to copy the data and to clean up later.
Pre-upgrade check
The 4.0.0 database migrations refuse data they cannot migrate safely: they abort and name the offending objects. The upgrade-check tool reports all of those problems before the upgrade, in one run, with the same wording the migration would use. It is read-only: it never writes database rows or keys.
- Image:
ghcr.io/lamassuiot/lamassu-upgrade-check. Configure it with a KMS-style YAML file passed throughLAMASSU_CONFIG_FILE; only thepostgresstorage provider is supported. - Run it on the 3.8.0 data after it has been consolidated into the
pkischemas, and before any 4.0.0 migration runs or any 4.0.0 service starts. - Exit codes:
0means no problems were found,1means at least one problem was found, and2means the command could not run (bad configuration, unreachable database, or a database that is not a 3.8.0 one), or a check could not run. - It covers the VA migrations only: an all-clear does not prove that the CA and KMS migration guards pass, so it does not replace the duplicate CA ID and
engine_idchecks described in PostgreSQL consolidation.
Step 2: Update the values file
Add services.authz. Configure the JWKS endpoint, the database mappings and the bootstrap principal:
services:
authz:
jwkUrl: https://idp.example.com/realms/lamassu/protocol/openid-connect/certs
credentials:
pki:
database: pki
authz:
database: authz
bootstrap:
- principal_id: "oidc:pki-admin"
principal_name: "PKI Admin"
principal_type: "oidc"
policy_ids: [...] # keep the defaults from values.yaml
auth_config:
claims:
- claim: realm_access.roles # was auth.authorization.rolesClaim
operator: contains
value: pki-admin # was auth.authorization.roles.adminjwkUrlis fetched from theauthzpod, not from the gateway, so a Service in the release namespace can be referenced by its short name. The default points at a Keycloak Service namedauth-keycloak; set it explicitly if your identity provider lives anywhere else.authzis the authorization service's own storage for principals, grants and policies.pkiis the database the authorization engine queries when it evaluates policies against entities. Only thedatabasefield of each entry is read; the connection always uses the globalpostgres.hostname,postgres.port,postgres.usernameandpostgres.passwordvalues.- If you kept the 3.8.0 role defaults (
realm_access.rolesandpki-admin), the defaultservices.authz.bootstrapalready matches them.
Delete the gateway token validation blocks. Remove auth.oidc.apiGateway and auth.authorization once their settings are in services.authz:
# REMOVE (3.8.0)
auth:
oidc:
apiGateway:
jwks:
- name: oidc-authn
uri: https://idp.example.com/realms/lamassu/protocol/openid-connect/certs
authorization:
rolesClaim: realm_access.roles
roles:
admin: pki-adminRename toolbox to connectivityTest:
# OLD (3.8.0)
toolbox:
image: ghcr.io/lamassuiot/toolbox:2.2.0
# NEW (4.0.0)
connectivityTest:
image: curlimages/curl@sha256:58adaa4e8dca9c988bae2aba4ab3434a0bb2da16bbe3f92dec39ec7785166777Remove the per-service identity overrides services.kms.securityContext.runAsGroup and services.authz.securityContext.runAsUser/runAsGroup.
Convert services.connectors to a map keyed by connector ID, each entry with its type and image:
services:
connectors:
aws.myconnector:
type: awsiot
image: ghcr.io/lamassuiot/lamassu-aws-connector:dev-v4Check AMQP. If you set amqp.tls: true, also set amqp.port to the broker's TLS listener (commonly 5671), or set amqp.tls: false to stay on plaintext.
Enable the CA-to-KMS migration if your CA keys predate the KMS:
migrations:
caToKms:
enabled: trueReview the remaining values: overrides that pinned services.ui.port to 80, the old probes: block, the TLS issuer, and explicit affinity and topologySpreadConstraints where your cluster needs them. Renamed and reshaped values describes each one.
Run helm lint with the new values before upgrading; the strict schema reports any key that is still unknown.
Step 3: Run the upgrade
helm upgrade --install "$RELEASE" lamassu/lamassu \
--namespace $NAMESPACE \
--values values.yaml \
--waitHelm runs the pre-upgrade hooks — the CA-to-KMS migration when enabled, and the database job — before applying any workload, so services never start against an unprepared database. If validation rejects unknown values, fix the values file and re-run.
If the deployment uses local KMS or VA storage, those pods fail until you complete step 4, so --wait cannot succeed: Helm waits for them to become ready, times out, exits with an error and marks the release failed. The hooks have already run by then, and every manifest, including the new StatefulSets and their PVCs, has been applied. Helm does not roll anything back without --atomic. Continue with step 4, then re-run the same helm upgrade command so it completes and Helm records the release as deployed. Both hooks are idempotent, so they are safe to run again.
Step 4: Migrate the KMS and VA data
Skip this step when services.kms.cryptoEngines never used a filesystem engine and services.va.fileStore.type was never local.
- Find the new StatefulSets and PVCs. Their names depend on the release name and any
nameOverrideorfullnameOverride; the PVC name is<volumeClaimTemplate>-<StatefulSet name>-0:
KMS_STS=$(kubectl get statefulset -n "$NAMESPACE" \
-l "app.kubernetes.io/instance=$RELEASE,app.kubernetes.io/component=kms" \
-o jsonpath='{.items[0].metadata.name}')
VA_STS=$(kubectl get statefulset -n "$NAMESPACE" \
-l "app.kubernetes.io/instance=$RELEASE,app.kubernetes.io/component=va" \
-o jsonpath='{.items[0].metadata.name}')
KMS_PVC="golang-engine-storage-${KMS_STS}-0"
VA_PVC="local-crl-file-storage-${VA_STS}-0"
# Fails if a lookup came back empty or a new claim is the old one.
# Item 3 calls it again before creating the helper pod.
claims_ok() {
local pair sts new old
for pair in "$KMS_STS:$KMS_PVC:golang-engine-storage-kms-0" \
"$VA_STS:$VA_PVC:local-crl-file-storage-va-0"; do
IFS=: read -r sts new old <<< "$pair"
if [ -z "$sts" ] || [ "$new" = "$old" ]; then
echo "ABORT: new claim '$new' is missing or equals the 3.8.0 claim '$old'" >&2
return 1
fi
done
}
claims_ok && kubectl get pvc -n "$NAMESPACE" \
golang-engine-storage-kms-0 "$KMS_PVC" \
local-crl-file-storage-va-0 "$VA_PVC"With release lamassu and the default chart name, the new PVCs are golang-engine-storage-lamassu-kms-0 and local-crl-file-storage-lamassu-va-0; with release prod they are golang-engine-storage-prod-lamassu-kms-0 and local-crl-file-storage-prod-lamassu-va-0. Confirm both old and new PVCs exist before continuing. If claims_ok prints ABORT, the block stops before listing the claims: either a StatefulSet lookup failed, or a new claim resolves to the 3.8.0 claim. In the second case the copy would verify against itself and the final cleanup would delete the live volume. The chart always appends -kms and -va to the release-scoped name, so a collision means the claims are not what this guide expects; investigate before going on. Item 3 runs the same check and refuses to create the helper pod, so nothing touches the volumes. The check returns instead of calling exit, which would close your shell and lose the variables set so far. If you only migrate one of the two services, drop the other one's claims from the commands that follow.
- Record the replica counts, scale the new StatefulSets down and wait for every pod that uses the claims to exit, so nothing writes to either volume while you copy. Helm removes the 3.8.0
kmsandvaStatefulSets during the upgrade, but their podskms-0andva-0can still be terminating and holding the old claims. While they run, the helper pod either cannot attach the claims or copies a volume that is still being written. Thewaitreturns at once for pods that are already gone:
KMS_REPLICAS=$(kubectl get statefulset -n "$NAMESPACE" "$KMS_STS" -o jsonpath='{.spec.replicas}')
VA_REPLICAS=$(kubectl get statefulset -n "$NAMESPACE" "$VA_STS" -o jsonpath='{.spec.replicas}')
kubectl scale statefulset -n "$NAMESPACE" "$KMS_STS" "$VA_STS" --replicas=0
kubectl wait -n "$NAMESPACE" --for=delete --timeout=300s \
pod/kms-0 pod/va-0 "pod/${KMS_STS}-0" "pod/${VA_STS}-0"- Run a short-lived helper pod that mounts all four claims. It runs as root so it can change file ownership, so pin its image by digest and do not mount the service-account token:
claims_ok && cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: pvc-migrate-helper
namespace: $NAMESPACE
spec:
restartPolicy: Never
automountServiceAccountToken: false
containers:
- name: migrate
image: docker.io/library/alpine@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc # 3.20
command: ["sh", "-c", "sleep 3600"]
volumeMounts:
- {name: old-kms, mountPath: /old/kms}
- {name: new-kms, mountPath: /new/kms}
- {name: old-va, mountPath: /old/va}
- {name: new-va, mountPath: /new/va}
volumes:
- {name: old-kms, persistentVolumeClaim: {claimName: golang-engine-storage-kms-0}}
- {name: new-kms, persistentVolumeClaim: {claimName: $KMS_PVC}}
- {name: old-va, persistentVolumeClaim: {claimName: local-crl-file-storage-va-0}}
- {name: new-va, persistentVolumeClaim: {claimName: $VA_PVC}}
EOF
kubectl wait -n $NAMESPACE --for=condition=Ready pod/pvc-migrate-helper --timeout=60s- Inspect before writing so you do not clobber data the new pods may already have written:
kubectl exec -n $NAMESPACE pvc-migrate-helper -- sh -c 'find /new/kms /new/va -type f'An empty result is the common case: a StatefulSet's own pod rarely gets far enough to write anything when its dependent data is missing. If the output is not empty, record the files and confirm with the service owner that they are disposable before you run the copy; do not overlay the 3.8.0 data onto an unknown store.
- Copy the data, align ownership and verify the copy. The second command must print
copy verified; anydiffoutput means the copy is incomplete, so do not continue:
kubectl exec -n $NAMESPACE pvc-migrate-helper -- sh -c '
cp -a /old/kms/. /new/kms/ &&
cp -a /old/va/. /new/va/ &&
chown -R 65532:65532 /new/kms /new/va
'
kubectl exec -n $NAMESPACE pvc-migrate-helper -- sh -c '
diff -r /old/kms /new/kms &&
diff -r /old/va /new/va &&
echo "copy verified"
'The chown is required even if the old files already look non-root-owned, because 4.0.0 requires exactly 65532:65532 and does not fall back to whatever the image declares.
- Remove the helper and restore the recorded replica counts:
kubectl delete pod -n $NAMESPACE pvc-migrate-helper
kubectl scale statefulset -n "$NAMESPACE" "$KMS_STS" --replicas="$KMS_REPLICAS"
kubectl scale statefulset -n "$NAMESPACE" "$VA_STS" --replicas="$VA_REPLICAS"- Re-run the step 3
helm upgradecommand. The KMS and VA pods now start against the copied data, so--waitsucceeds and the release moves fromfailedtodeployed.
Step 5: Remove the one-off migration values
After the upgrade that performed the CA-to-KMS migration succeeds, remove migrations.caToKms.enabled from your values (it defaults to false). The hook reads the legacy 3.8.0 kms-config ConfigMap while it still exists and falls back to the release-scoped <release>-lamassu-kms-config once it is gone.
Then go through the validation checklist. Keep the old PVCs until it passes.
Deployments without Helm
Without the chart's database job, no tool moves data between databases: the migration tooling only applies migrations to a schema that already exists. Move each per-service database into its pki schema yourself, preserving structure, indexes, constraints, data and migration history, with writes stopped and a verified backup in hand.
Run the migrations in the required order, one schema at a time: kms first, then the ca-to-kms-migration tool (needed only for legacy deployments whose CA keys are named by SKI), then ca, then va. The devicemanager, dmsmanager and alerts schemas are independent and can be migrated in any order. With Helm, the chart's pre-install job applies the schemas in the correct order automatically.
If kms has not been migrated, the ca migration fails with kms schema not found: migrate the kms schema before the ca schema. The va migration requires ca and kms to be migrated first: it fails with ca schema not found: migrate the ca schema before the va schema or kms schema not found: migrate the kms schema before the va schema if one of them is missing.
The schemas reference each other, so the PostgreSQL role running the migrations needs cross-schema privileges: the ca schema needs USAGE on kms plus SELECT, REFERENCES and INSERT on kms.kms_keys, and the va schema needs USAGE on ca plus SELECT and REFERENCES on ca.certificates, SELECT on ca.ca_certificates, and USAGE on kms plus SELECT and REFERENCES on kms.kms_keys. Without them the migration aborts; grant the missing privileges and rerun the failed schema. After migrating ca, if the CA service runs as a different user than the migration user, grant it SELECT on ca.certificates_view (GRANT SELECT ON ca.certificates_view TO service_user). Without this grant, certificate and CA reads fail with permission denied for table certificates_view. After migrating va, if the CA service runs as a different user, grant it USAGE on the va schema and SELECT on va.crl_settings (GRANT USAGE ON SCHEMA va TO service_user; GRANT SELECT ON va.crl_settings TO service_user): the CA service reads va.crl_settings when it deletes a CA, and without these grants every deletion of a CA that has a crypto engine fails with permission denied for schema va (or for table crl_settings).
To apply migrations by hand, goose-lamassu must run against the pki database with search_path=<schema> in the connection string. The binary validates only search_path, not the database name, so check dbname=pki before running it, and repeat for kms, ca, va, devicemanager, dmsmanager and alerts:
./goose-lamassu "host=$DB_HOST port=5432 user=$DB_USER dbname=pki search_path=kms" upRun ca-to-kms-migration first with --dry-run, which reports without writing to KMS storage, and then for real:
LAMASSU_CONFIG_FILE=migrate.yaml ./ca-to-kms-migration --dry-run
LAMASSU_CONFIG_FILE=migrate.yaml ./ca-to-kms-migrationPoint the tool's storage configuration at the pki database, where the CA and KMS data now live in the ca and kms schemas. The tool's options are described in backend/cmd/ca-to-kms-migration/README.md in the lamassuiot repository.
Both tools' READMEs document the 4.0.0 pki layout and match this page: goose-lamassu uses dbname=pki examples and does not validate the database name, and the ca-to-kms-migration README describes the single pki database and the migration order.
Troubleshooting
- KMS or VA pods crash with
no such file or directoryorcode=NotFoundright after the upgrade. The pods are bound to the new, empty PVCs and the data migration has not run yet, or was copied into the wrong PVC. Follow step 4 and check that all four claims exist. helm upgrade --waittimes out andhelm statusreports the release asfailed. With local KMS or VA storage this is expected on the first run: the new pods cannot become ready before their data is copied. The hooks ran and the manifests and PVCs were applied, so complete step 4 and re-run the samehelm upgradecommand. If other pods are also not ready, check their logs, because something else is wrong.- The helper pod stays
PendingorContainerCreatingandkubectl waittimes out. Another pod still holds one of the claims; on block storagekubectl describe pod pvc-migrate-helpershows aMulti-Attach error. Check withkubectl get pods -n $NAMESPACEthatkms-0,va-0and the new KMS and VA pods are gone, then re-run the wait. helm upgradefails with an unknown-key validation error. Remove or rename the legacy values — typicallytoolbox,auth.oidc.apiGatewayorauth.authorization— because the strict schema rejects keys it does not know.- The upgrade fails at the database job. The PostgreSQL role cannot create databases or schemas, or a schema migration aborted: on duplicate CA IDs or missing
engine_idvalues, or in the VA schema on a CA ID containing/, an SKI shared by more than one CA (the error asks you to recreate all but one of them on a different key), a role whose signer certificate has no key, a CA ID reserved for the archived settings of another role, or a CA ID equal to another role's SKI. The abort message names the offending objects; fix them and re-run the upgrade. - The upgrade fails at the CA-to-KMS hook. CA certificates that share a
subject_key_iddeclare conflictingengine_ids. Resolve the conflict and re-run. - Nobody can log in after the upgrade. Check that
services.authz.jwkUrlis reachable from theauthzpod, thatservices.authz.bootstrapmatches a role your OIDC provider actually assigns, and that the database job completed. - A migration aborts with
kms schema not found. The schemas must be migrated in order: run thekmsschema migration before thecaandvaschema migrations. - A migration aborts with
ca schema not found. Run thecaschema migration before thevaschema migration. - The VA migration fails with
cannot resolve signer certificate for VA role ... N candidates. The migration cannot uniquely identify a signer certificate from the SKI stored on the role. Review the SKI and candidate certificates reported in the error; correct the pre-migration data with a verified backup and rerun the migration. The newsigner_serial_numberfield is created during that migration, so it cannot be set through the VA API while the migration is failing. - A migration or service fails with
permission denied for table ...orpermission denied for schema .... The PostgreSQL role is missing the cross-schema privileges listed in Deployments without Helm; this includescertificates_viewwhen the post-migrationGRANT SELECTwas skipped, andpermission denied for schema va(orfor table crl_settings) when deleting a CA without thevagrants. Grant the missing privileges and re-run.
Validation checklist
After the upgrade, check in this order:
- Databases. Each service reads the
pkidatabase with its own schema,authzandwfxdata are reachable, and the migration status is registered per schema, including the CA unique-ID and KMS composite-key migrations and the foreign-key migrations that run per schema in the required order. - KMS and VA start cleanly. Their logs show no
no such file or directoryorcode=NotFounderrors. - Existing KMS keys work. Issue a certificate from a CA that existed before the upgrade. None of the other checks reads a KMS key, so this is the only one that proves the key material was copied.
- Existing CRLs are served.
GET /crl/<ca-id>on VA returns a previously issued CRL, andGET /crl/<ca-ski>with the legacy SKI of a certificate issued before the upgrade also resolves. The VA scheduled CRL job has issued a new list under the CA ID with both URLs in its IDP; check its logs if regeneration has not succeeded. - Renamed VA events. Alert subscriptions to
va.role.updateandva.role.crl.initnow subscribe tova.crl-settings.updateandva.crl-settings.init, and conditions for all VA events, includingva.role.crl.create, have been reviewed against the new payload (ca_idinstead ofca_ski) and subject (crl/<ca-id>instead ofcrl/<ski>). Direct consumers and audit searches use the new event names. - Legacy CA keys. If the CA-to-KMS migration ran, review its report and warnings and confirm the expected keys appear in the KMS. Migrated CA certificates now report
key_idandhas_private_key, which you can use to confirm the registration. - Connectivity.
helm test "$RELEASE" -n $NAMESPACE --logspasses; the hook checks the CA, DMS Manager, Device Manager and VA health endpoints and the UI response. - Authorization. Sign in as the bootstrap principal and confirm administration works, check that an identity without permissions receives
403, and review client and integration permissions. - AMQP. The backend connects to the broker with the expected TLS settings, and the cleanup policy for SSE queues is in place.
- Device events.
GET /v1/devices/:id/eventsreturns the historical events of a device that had them. - Webhooks without file fields. For each DMS that uses a webhook for enrollment or re-enrollment, the
configblock carriesinsecure_skip_verifyand the expected option block (apikey_options,jwt_optionsormtls_options), with the certificate and key as PEM and nocert_fileorkey_file. - Receiver verification.
insecure_skip_verifyisfalse, unless the receiver certificate is intentionally unverified. - Webhook delivery. One enroll or re-enroll through each migrated webhook completes its authorization call.
- Trust set.
est_rfc7030_settings.client_certificate_settings.validation_casholds the expected CAs, in particular those of the old CA during rotation overlaps. - API clients. Scripts and consoles no longer use the old DMS layout (flat
additional_validation_cas, or the legacymtls,apikeyandoidcfields), send DMS updates with the ID in the URL, and match on the new certificatetypevalues.
Delete the old volumes
Only once every check passes, delete the orphaned 3.8.0 volumes to reclaim storage. The delete is irreversible and the old KMS volume may hold the last copy of your CA key material, so take a VolumeSnapshot or a backup of both old PVCs first and keep it until you are confident the upgraded deployment is healthy. These are the fixed 3.8.0 names recorded in step 1; before deleting, confirm that neither is mounted by a running pod, for example with kubectl describe pvc -n $NAMESPACE golang-engine-storage-kms-0 local-crl-file-storage-va-0, where Used By must be <none>:
kubectl delete pvc -n $NAMESPACE golang-engine-storage-kms-0 local-crl-file-storage-va-0The 3.8.0 per-service databases are no longer used either. Drop them only after the same checks pass and your PostgreSQL backup is verified.