Skip to content

Integration: ACME

The API Gateway Manager can renew the certificates of private keys on your Layer7 gateways automatically. It acts as an ACME client (RFC 8555) and requests the certificates from your own ACME server or any ACME compatible server, for example Secardeo certACME, EJBCA or step-ca.

  • The private key stays on the gateway. The API Gateway Manager only asks the gateway for a certificate signing request (CSR), sends it to the ACME server and puts the issued certificate chain back onto the key.
  • The alias of the key does not change, so policies and listen ports that use the key keep working.
  • A renewal can be started manually ("Renew now") or runs automatically once a day.
  • Failed renewals can trigger a notification (e-mail, Teams, ...) via Automations.

Set Up ACME Step by Step

This is the short way from nothing to a certificate that is renewed automatically. The sections below explain every field.

Before you start

Ask the team that runs your ACME server (your PKI or CA team) for:

  • the directory URL of the ACME server, with the certificate profile if the server has several (EJBCA, step-ca),
  • the EAB key ID and HMAC key, if the server requires External Account Binding,
  • the CA certificate (PEM) that issued the TLS certificate of the ACME server, if it is an internal CA,
  • how the server authorizes your domains: without a challenge (whitelist, pre-authorization, EAB) or via DNS-01. For DNS-01 ask the DNS team for a TSIG key that may update TXT records in the zone,
  • whether the issued certificates have a subject (CN=<domain>). The gateway needs it, see the prerequisites below.

Check on the API Gateway Manager:

  • The gateway runs version 11.1.1 or newer (11.2.1 or newer to restart the listen ports automatically) and its version is set in the node settings.
  • The node connects via Graphman (Check Connectivity on the node).
  • The API Gateway Manager reaches the ACME server via HTTPS (and the DNS server via TCP port 53 for DNS-01).

Steps

  1. Prepare the private key on the gateway. Use the key that the listen port or policy already uses, or create one in the Security tab of the node with Create New Private Key: alias e.g. api-example, DN CN=api.example.com. Its self-signed certificate is replaced with the first renewal. Assign the key to the listen port in the Policy Manager (Manage Listen Ports, tab SSL/TLS Settings) if it is a new one.
  2. Create the ACME server under Integrations, section Security, icon ACME: directory URL, contact email, EAB and CA certificate if needed. Click Test, then save. See 1. Configure the ACME Server.
  3. Only for DNS-01: create the DNS server under Integrations, icon DNS (RFC 2136): DNS server, zone, TSIG key. Click Test, then save. See Configure the DNS Server (DNS-01).
  4. Configure the key. In the Security tab of the node click the button ACME of the key: switch on "Renew automatically via ACME", choose the ACME server, the challenge (and the DNS server), enter the domains (api.example.com, one per line), choose the target (the node or the cluster of the key) and keep "Restart listen ports" on. Save. See 2. Enable the Renewal for a Private Key.
  5. Renew now in the same dialog. The background job shows every step. Afterwards the column ACME shows "Managed" and the key its new expiry date.
  6. Check the result (see Check the renewed certificate): the listen port serves the new certificate, issued by your CA.
  7. Get notified if a renewal fails: create an automation for the ACME advices, see 6. Notifications.

From now on the daily check renews the certificate "Renew days before expiry" (default 30) days before it expires.

Check the renewed certificate

  • In the Security tab the private key shows the new expiry date. The column ACME shows "Managed", "Last Renewal" in the ACME dialog the time of the renewal.
  • The listen port serves the new certificate, e.g. with openssl s_client -connect <gateway>:<port> -servername api.example.com </dev/null | openssl x509 -noout -subject -issuer -enddate.
  • If the ACME dialog shows Action needed (column ACME: "Restart needed"), some listen ports still serve the old certificate until they are restarted, see Listen ports serve the new certificate after a restart.

Certificates: what the clients need

The renewed certificate is issued by the CA of your ACME server, not self-signed any more. Clients of the listen port (browsers, API consumers, other gateways) must trust this CA: install its root certificate (and intermediate certificates if they are not sent by the gateway) in their trust store before the first renewal. The gateway sends the whole chain it received from the ACME server.

  • Pinned certificates: clients that pin the certificate or its public key of the listen port must be updated after every renewal. With "Keep private key, renew certificate only" the public key stays the same, so public key pinning keeps working; with "Create a new private key on every renewal" it changes every time.
  • Mutual TLS: the trusted certificates of the gateway (for client certificates and backends) are not touched by ACME.
  • Subject: the first domain becomes the common name (CN=<domain>), all domains become subject alternative names. Clients check the host name against the subject alternative names.

Prerequisites

  • An ACME server that the API Gateway Manager can reach over HTTPS.
  • The ACME server must authorize the domains, either

    • without a challenge, for example by a whitelist, a pre-authorization or External Account Binding (EAB), or
    • via the challenge DNS-01: the API Gateway Manager sets a TXT record on the DNS server of the domains. The DNS server must accept dynamic updates (RFC 2136) with a TSIG key, see Configure the DNS Server (DNS-01).

    HTTP-01 and TLS-ALPN-01 are not supported. * One private key per subject on a gateway: the gateway refuses a certificate whose subject another private key of the gateway already has. Renew every domain with one key only. If the ACME server issues certificates without subject (only subject alternative names, as some servers and certificate profiles do), they all have the same, empty subject: then only one key per gateway can be renewed. Use an ACME server or certificate profile that sets the subject (common name CN=<domain>), as internal CAs usually do. * The private key already exists on the gateway. The API Gateway Manager renews the certificate of an existing key, it does not create keys. * The API Gateway Manager can reach the gateway via Graphman (preferred) or Restman (fallback). * Gateway 11.1.1 or newer. From 11.2.1 the API Gateway Manager also restarts the listen ports using the key, see Listen ports serve the new certificate after a restart.

1. Configure the ACME Server

Go to the Integrations menu item and select the ACME icon in the Security section of the Integration Center tab.

Field Description
Name Any name, shown when you choose the ACME server for a private key
Directory URL The ACME directory endpoint. With EJBCA or step-ca the URL also selects the certificate profile, e.g. https://ejbca.example.com/ejbca/acme/ALIAS/directory
Contact Email Registered with the ACME account, used by the CA for notices
EAB Key ID / EAB HMAC Key External Account Binding, only if the ACME server requires it (e.g. certACME Enterprise). The HMAC key is base64url encoded as provided by the CA
Verify SSL Certificate Checks the TLS certificate of the ACME server. On by default, only disable it for tests
CA Certificate (PEM) Optional. The certificate of your company CA if the TLS certificate of the ACME server is issued by it. Several certificates can be pasted one after another. The system CAs stay trusted
Automatic Renewal Renews the private keys of this ACME server once a day. Disable it e.g. during a maintenance of the ACME server
Daily at Time of the daily check (time zone of the API Gateway Manager)
Next Run When the daily check runs next
Save even if the test fails Saves the ACME server although the connection test fails, see below

Click Test to check the settings without saving them. The test checks the TLS connection, the directory, the nonce endpoint and whether External Account Binding is configured as the ACME server requires. Whether the ACME server actually accepts the EAB key is only known when the API Gateway Manager registers its account during the first renewal.

The same test runs when you save the ACME server. If it fails (e.g. wrong directory URL, server not reachable, untrusted certificate), the ACME server is not saved and the errors are shown in the dialog. Check Save even if the test fails to save it anyway, e.g. if the ACME server is not reachable yet. Warnings (e.g. SSL verification disabled) do not prevent saving.

The ACME account (an EC P-256 key) is created automatically on the first renewal and stored encrypted. It is registered again if the directory URL or the EAB key ID changes, or if the ACME server no longer knows the account. Terms of service of the ACME server are accepted automatically; the test shows them.

After creating your integration, you'll find it listed in the "Existing Integrations" tab. You can edit or delete it from there.

Configure the DNS Server (DNS-01)

Only needed for the challenge DNS-01. Go to the Integrations menu item and select the icon DNS (RFC 2136) in the Security section of the Integration Center tab.

Field Description
Name Any name, shown when you choose the DNS server for a private key
DNS Server Host name or IP address of the primary DNS server of the zone, optionally with port (default 53), e.g. ns1.example.com or 10.0.0.53:53. The updates are sent via TCP
Zone The zone that contains the domains of the certificates, e.g. example.com
TSIG Key Name Name of the TSIG key that may update the zone
TSIG Algorithm HMAC-SHA256 (default), HMAC-SHA512 or HMAC-SHA1, as configured for the key on the DNS server
TSIG Secret The secret of the TSIG key, base64 encoded as in the configuration of the DNS server. Stored encrypted
Wait after update (seconds) Additional time after the DNS server returns the TXT record, e.g. until the secondary DNS servers have it. 0 = validate right away, at most 600

The DNS server must allow the TSIG key to update TXT records in the zone. Example for BIND:

key "dagm-key" {
    algorithm hmac-sha256;
    secret "<base64 secret>";
};
zone "example.com" {
    type primary;
    file "example.com.zone";
    update-policy { grant dagm-key zonesub TXT; };
};

Other DNS servers with dynamic updates and TSIG, e.g. PowerDNS or Knot DNS, work the same way. Create the secret e.g. with tsig-keygen -a hmac-sha256 dagm-key.

Click Test to check the settings: the API Gateway Manager sets a TXT record _dagm-test.<zone>, reads it back and deletes it again.

The DNS server must be reachable from the API Gateway Manager via TCP, and the ACME server must resolve the domains via this DNS server (or a secondary of the zone).

2. Enable the Renewal for a Private Key

Open the node and go to the Security tab. In the Private Keys section, click the button ACME of the key. You need write rights on the environment of the node.

Field Description
Renew automatically via ACME Only enabled keys are renewed by the daily check
ACME Server One of the ACME integrations
Challenge "None (pre-authorized / External Account Binding)": the ACME server authorizes the domains itself. "DNS-01 (TXT record on a DNS server)": the API Gateway Manager answers the challenge via a DNS server
DNS Server Only for DNS-01: one of the "DNS (RFC 2136)" integrations, the DNS server of the zone of the domains
Domains One domain per line. The first one becomes the common name, all of them subject alternative names. Wildcards like *.example.com are allowed if the ACME server supports them. With DNS-01 every domain must be in the zone of the DNS server, this is checked when saving
Renew days before expiry 1 to 365, default 30
Private Key Handling "Keep private key, renew certificate only" (default), or "Create a new private key on every renewal" with key type, RSA key size or elliptic curve and signature hash, see New private key on every renewal
Target Gateways the renewed certificate is installed on (nodes, clusters or whole environments). Usually the node or cluster of the key
Restart listen ports On by default. Restarts the listen ports using the key right after the renewal, so they serve the new certificate, see Listen ports serve the new certificate after a restart. Switch it off to restart them yourself, e.g. in a maintenance window
Order Status / Last Renewal / Next Check / Last Error State of the renewal (read only)

While "Renew automatically via ACME" is off, the settings may be incomplete. When it is on, an ACME server, at least one domain and at least one target are required, and for DNS-01 a DNS server.

The column ACME in the private keys grid shows the state of every key:

State Meaning
Not managed No ACME settings
Disabled Settings saved, automatic renewal off
Managed Renewed automatically
Renewing An order is in progress
Waiting for approval The ACME server waits for a manual approval of the order
Restart needed Renewed, but a listen port serves the new certificate only after a restart, see "Action needed" in the dialog
Error The last renewal failed, see "Last Error" in the dialog

The button Manage ACME Servers above the grid opens the ACME integrations.

3. Renew Now

The button Renew now in the ACME dialog saves the settings of the dialog and renews the certificate immediately, regardless of its expiry date. It runs as background job and shows its progress. An ACME server must be selected.

4. Automatic Renewal

Every ACME server with "Automatic Renewal" on has one daily background job. At the configured time it renews every enabled private key of this ACME server that

  • expires within its "Renew days before expiry", or
  • has an order that is still open or waits for an approval.

A key whose renewal failed is retried every day, but only while it expires soon. One failing key does not stop the others; the background job is then marked as failed.

If the API Gateway Manager is not running at the configured time, for example during an update or maintenance, the daily check runs right after the API Gateway Manager has started again. It runs once, also after several days, and renews every key that is due by then. The next check runs at the configured time again. While the API Gateway Manager is down, nothing is renewed and no notification is sent: the gateway and the ACME server do not renew certificates by themselves. Choose "Renew days before expiry" large enough to cover a longer downtime plus the time to fix a failed renewal; with the default of 30 days there is plenty of time.

Approval workflows: Some ACME servers (e.g. certACME with "ACME acceptance") keep an order open until somebody approves it. The API Gateway Manager then shows "Waiting for approval", keeps the order and continues it with the next run. No new order is created.

5. What Happens During a Renewal

  1. The API Gateway Manager registers or loads its ACME account.
  2. It creates an order for the domains and makes sure the authorizations are valid. An open order, for example one that waits for an approval, is continued, but only for the same domains and for the private key its CSR was created for: after the domains or "Private Key Handling" were changed, a new order is created.

    With DNS-01, for every domain that is not authorized yet: it sets the TXT record _acme-challenge.<domain> on the DNS server (for *.example.com on example.com), waits until the DNS server returns it (at most 30 seconds, plus "Wait after update"), lets the ACME server validate it and deletes the record again, also if the validation fails. 3. It asks the gateway of the key for a CSR: subject CN=<first domain>, every domain as subject alternative name. The private key stays on the gateway. 4. The ACME server issues the certificate, the API Gateway Manager downloads the chain and checks that it fits the private key on the gateway. 5. The certificate chain of the key is replaced on every target, and the private keys of the target are read again. 6. The listen ports using the key are restarted, see below.

Graphman is used whenever it is available on a gateway, Restman only as fallback. The progress shows which one is used.

Several targets: Replacing the chain only works if the target has the same private key (same keystore and alias, e.g. distributed with "Create New Private Key" to several targets). A target with a different key under the same alias reports an error, the other targets are still updated.

Listen ports serve the new certificate after a restart

A gateway keeps serving the old certificate (or the old private key) on a listen port until the listen port is restarted, also after the certificate was replaced. Therefore the API Gateway Manager restarts the listen ports that use the key, on every target:

  • listen ports with the key as their own private key, and
  • listen ports without own private key (e.g. "Default HTTPS (8443)", "Default HTTPS (9443)") if the key is the default SSL key of the gateway.

Only enabled listen ports are restarted; their configuration is not changed. During the restart the listen port refuses new connections for a moment (usually less than a second), open connections stay. In a cluster the other nodes serve the new certificate a few seconds later.

Restarted automatically on gateway 11.2.1 or newer. Not restarted, but listed in a notice:

  • the listen port of the connection of the API Gateway Manager itself (usually "Default HTTPS (8443)" with the default SSL key): the gateway does not restart the listen port of the current management connection,
  • every listen port of gateways older than 11.2.1, which can not restart a single listen port safely,
  • every listen port if "Restart listen ports" is off.

The notice is shown at the end of the renewal and as Action needed in the ACME dialog, the column ACME shows Restart needed, and for keys with automatic renewal the advice ACME certificate needs a listen port restart is created. Restart the listen ports listed there, e.g. in the Policy Manager (Manage Listen Ports: disable and enable the listen port) or restart the gateway. The notice disappears with the next renewal that needs no restart by hand.

New private key on every renewal

With "Create a new private key on every renewal" the key pair is replaced as well:

  1. The gateway of the key generates a new key pair under the temporary alias <alias>-acme-next. The old key stays in use, also while an order waits for an approval.
  2. The CSR is created for the new key.
  3. The new key is exported from the gateway, put together with the issued certificate chain and replaces the old key on every target in one step, under the same alias: listen ports and policies keep working. On the way it is protected by a one-time password.
  4. The temporary key is deleted.

Requirements: the gateway of the key needs version 11.2.0 or newer (set the gateway version in the node settings) and Graphman; all targets need Graphman. This is checked when the ACME settings are saved. A target without Graphman reports an error, the other targets are still updated.

How Private Keys Are Protected

  • New keys are generated on the gateway. With ACME the private key never leaves the gateway: only the certificate signing request (CSR) and the issued certificate are transferred.
  • Copying a key to other gateways (new private key on every renewal, "Create New Private Key" for several targets): the key is transferred as PKCS#12 file encrypted with a random one-time password. The password only exists in memory for this transfer, it is not stored or logged. No copy of the key is stored on the server of the API Gateway Manager.
  • Exporting a key in the security tab: the PKCS#12 file is encrypted with the password you enter, no copy is stored on the server.
  • Connections to the gateways and to the ACME server use HTTPS. The TLS certificate of the ACME server is verified (option "Verify SSL Certificate").

6. Notifications

After every renewal the API Gateway Manager updates three advices (see Advisor):

  • ACME renewal failed, with the error and a button to the security tab
  • ACME order waits for approval
  • ACME certificate needs a listen port restart: renewed, but a listen port serves the new certificate only after a restart

They exist only for keys with automatic renewal enabled and disappear with the next successful renewal. They can not be suppressed: a suppressed advice would hide the next problem of the key. To get an e-mail or Teams message:

  1. Create a notification integration (e.g. E-Mail or Teams), if not done yet.
  2. Go to Automations and create an automation with the recipe "Send a notification to ... when a new advice of type ... is created in ...".
  3. Choose the advice type "ACME renewal failed" (and optionally "ACME order waits for approval" and "ACME certificate needs a listen port restart"), the integration and the environment. One automation per environment.

A notification is sent once when the advice is created, not again every day while the problem persists.

Limitations

  • Challenges: pre-authorized orders ("None") and DNS-01 via dynamic DNS updates (RFC 2136) with TSIG. HTTP-01, TLS-ALPN-01 and the APIs of DNS providers (e.g. Azure DNS, Route 53) are not supported.
  • DNS-01: one zone per DNS integration. Renew domains of different zones with separate private keys.
  • Gateway 11.1.1 or newer. Older gateways are not supported.
  • Listen ports are restarted automatically on gateway 11.2.1 or newer, except the listen port of the connection of the API Gateway Manager.
  • "Create a new private key on every renewal" needs gateway 11.2.0 or newer with Graphman on the node of the key, and Graphman on all targets.
  • Only certificates of existing private keys can be renewed. Trusted certificates of third parties cannot be renewed via ACME.

Troubleshooting

Message Cause / solution
The ACME server <URL> does not answer in time / refuses the connection / is unknown, its host name can not be resolved The directory URL of the ACME server is wrong, or the API Gateway Manager can not reach the server (firewall, proxy). Check the directory URL under Integrations
Unable to connect to the ACME server: ... cURL error 60 The TLS certificate of the ACME server is not trusted. Paste the company CA into "CA Certificate (PEM)"
The ACME server requires External Account Binding, but no EAB key is configured Enter EAB key ID and HMAC key of the CA
... (urn:ietf:params:acme:error:externalAccountRequired) / unauthorized EAB key ID or HMAC key wrong, or the account is not allowed for these domains
The ACME server did not authorize <domain> Challenge "None": the ACME server requires a challenge for this domain. Whitelist or pre-authorize the domain on the ACME server, or use DNS-01. DNS-01: the ACME server did not find the TXT record, e.g. because it asks a DNS server that does not have the record yet. Increase "Wait after update" of the DNS integration
DNS-01 for <domain>: ... The TXT record could not be set, see the following rows
Update of zone ... on ... failed: check TSIG key name, algorithm and secret (NOTAUTH) Wrong TSIG key name, algorithm or secret, or the server is not authoritative for the zone. Compare with the key in the configuration of the DNS server
Update of zone ... on ... failed: refused, allow updates for the TSIG key (REFUSED) The TSIG key may not update the zone. Allow it, e.g. update-policy { grant <key> zonesub TXT; }; (BIND)
"<domain>" is not in the zone ... of the DNS server ... (when saving) / _acme-challenge.<domain> is not in the zone ... The domain is not in the zone of the DNS integration: correct the domain (e.g. host.example.com instead of host), or choose the DNS integration of the right zone
The DNS server ... is not reachable / did not answer in time Wrong host or port, or a firewall blocks TCP to the DNS server
TXT record set, but ... does not return it The DNS server accepted the update but does not answer with the record. Enter the primary DNS server that holds the zone
No DNS server is selected for the challenge DNS-01 / The TSIG secret of ... is not valid base64 Choose a DNS server in the ACME dialog of the key / enter the secret base64 encoded
rejectedIdentifier The domain is not allowed by the policy of the ACME server
... not installed on all targets: <gateway>: ... The certificate was issued but a target rejected it, usually because the target has a different private key under the alias
<gateway>: restart the listen port(s) ... or the gateway, only then ... serve(s) the new certificate Not an error: the certificate is renewed, but these listen ports still serve the old one. Restart them (Policy Manager: disable and enable the listen port) or the gateway. See Listen ports serve the new certificate after a restart
The certificate was installed, but <gateway>: The listen port ... is DISABLED, enable it in the Policy Manager The listen port was switched off for the restart, but could not be switched on again (e.g. the connection to the gateway broke). Enable it at once in the Policy Manager (Manage Listen Ports)
The certificate was installed, but <gateway>: The listen port ... could not be restarted: ... The gateway refused the restart, the listen port is unchanged and still serves the old certificate. Restart it by hand
<gateway>: has another private key <alias> than <node> The target has another private key under the alias, the certificate does not fit it. Copy the private key of the node to the target, or choose "Create a new private key on every renewal"
Refused: another private key on the gateway probably has a certificate with the same subject (...) Another private key of the gateway has a certificate with this subject, the gateway does not accept a second one. Find it in the security tab (column subject), or use an ACME server or certificate profile that sets the subject (CN) if it is "none"
Graphman is not available ... falling back to Restman Only a warning: Graphman is not reachable on that gateway
"Create a new private key on every renewal" needs gateway 11.2.0 or newer with Graphman on ... Set the gateway version in the node settings, or update the gateway. Otherwise switch the private key handling back to "Keep private key"
...: Graphman is not available, the new private key can only be installed via Graphman Enable Graphman on that target, or remove it from the targets of the key
The ACME server of <key> does not exist anymore The ACME integration chosen for the key was deleted. Open the ACME dialog of the key, choose an ACME server and save