Configure Kubernetes Traefik Ingress with mTLS#

mTLS configuration using an Traefik ingress was more difficult than expected. Primary because important information is hidden well. In fact it is much simpler than one might expect.

Primer about mTLS#

If Mutual TLS is configured both, server and client, need to provide certificates.

First the handshake is done normally as with TLS. But next the client presents its own certificate and the server has to check it. In order to do so, it has to be configured to trust the certificate by checking it against a Certificate Authorities (CAs) certificate. Usually a custom CA is used to the client certificates. So, the server needs the custom CAs certificate to proceed.

Prepare the certificates#

Install OpenSSL first.

Create Certificate Authority#

First we need a custom Certificate Authority (CA).

Create configuration file openssl-ca.cnf like so:

HOME            = .
RANDFILE        = $ENV::HOME/.rnd

####################################################################
[ ca ]
default_ca    = CA_default      # The default ca section

[ CA_default ]

default_days     = 3650          # How long to certify for
default_crl_days = 30           # How long before next CRL
default_md       = sha256       # Use public key default MD
preserve         = no           # Keep passed DN ordering

x509_extensions = ca_extensions # The extensions to add to the cert

email_in_dn     = no            # Don't concat the email in the DN
copy_extensions = copy          # Required to copy SANs from CSR to cert

base_dir      = .
certificate   = $base_dir/cacert.pem   # The CA certifcate
private_key   = $base_dir/cakey.pem   # The CA private key
new_certs_dir = $base_dir              # Location for new certs after signing
database      = $base_dir/index.txt    # Database index file
serial        = $base_dir/serial.txt   # The current serial number

unique_subject = no  # Set to 'no' to allow creation of
                     # several certificates with same subject.

####################################################################
[ req ]
default_bits       = 2048
default_keyfile    = cakey.pem
distinguished_name = ca_distinguished_name
x509_extensions    = ca_extensions
string_mask        = utf8only

####################################################################
[ ca_distinguished_name ]
countryName         = Country Name (2 letter code)
countryName_default = AT
stateOrProvinceName         = State or Province Name (full name)
stateOrProvinceName_default = Tirol
localityName                = Locality Name (eg, city)
localityName_default        = Innsbruck
organizationName            = Organization Name (eg, company)
organizationName_default    = ACME GmbH
organizationalUnitName         = Organizational Unit (eg, division)
organizationalUnitName_default = Development
commonName         = Common Name (e.g. server FQDN or YOUR name)
commonName_default = Purpose of the CA
emailAddress         = Email Address
emailAddress_default = mail@acme.example.com

####################################################################
[ ca_extensions ]

subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always, issuer
basicConstraints       = critical, CA:true
keyUsage               = keyCertSign, cRLSign

####################################################################
[ signing_policy ]
countryName            = optional
stateOrProvinceName    = optional
localityName           = optional
organizationName       = optional
organizationalUnitName = optional
commonName             = supplied
emailAddress           = optional

####################################################################
[ signing_req ]
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints       = CA:FALSE
keyUsage               = digitalSignature, keyEncipherment

The execute OpenSSL to create the CA:

openssl req -new -x509 -config openssl-ca.cnf -days 365 -newkey rsa:4096 -sha256 -noenc -out cacert.pem -outform PEM

Prompt the questions given.

You should end up with two new files cacert.key and cacert.pem.

Create the client certificate#

First lets create a key with:

openssl genpkey -out client.key -algorithm RSA -pkeyopt rsa_keygen_bits:4096

For the Certificate Signing Request (CSR) we again need a configuration file, create openssl-csr.cnf:

HOME            = .
RANDFILE        = $ENV::HOME/.rnd

####################################################################
[ req ]
default_bits       = 4096
default_keyfile    = client.key
distinguished_name = server_distinguished_name
req_extensions     = server_req_extensions
string_mask        = utf8only

####################################################################
[ server_distinguished_name ]
countryName         = Country Name (2 letter code)
countryName_default = AT
stateOrProvinceName         = State or Province Name (full name)
stateOrProvinceName_default = Tirol
localityName                = Locality Name (eg, city)
localityName_default        = Innsbruck
organizationName            = Organization Name (eg, company)
organizationName_default    = ACME GmbH
organizationalUnitName         = Organizational Unit (eg, division)
organizationalUnitName_default = Development

commonName         = Common Name (e.g. server FQDN or YOUR name)
commonName_default = Purpose of the Key

emailAddress         = Email Address
emailAddress_default = mail@acme.exampel.com
####################################################################
[ server_req_extensions ]

subjectKeyIdentifier = hash
basicConstraints     = CA:FALSE
keyUsage             = digitalSignature, keyEncipherment
subjectAltName       = @alternate_names
nsComment            = "OpenSSL Generated Certificate"

####################################################################
[ alternate_names ]

DNS.1 = local.example.com

Then create the signing request:

openssl req -new -config openssl-csr.cnf -key client.key -out client.csr -outform PEM

And sign the key. The first time we need to create some files:

touch index.txt
echo "01" > serial.txt

The actually sign the key:

openssl ca -config openssl-ca.cnf -policy signing_policy -extensions signing_req -out client.pem -infiles client.csr

Provide the CA Certificate as Secret#

Lets load the CA certificate into cluster. The CA certificate is extracted from key tls.ca or ca.crt of the given secret! We use latter.

kubectl create secret generic ca-mtls -n example --from-file=ca.crt=./cacert.pem

Configure the Traefik Ingress#

In our example we are using a Letsencrypt cluster issuer for the normal servers TLS certificate. The issuer is named letsencrypt.

First we configure the Traefik specific TLSOption to be passed to the Ingress as annotation. The annotation is traefik.ingress.kubernetes.io/router.tls.options and the value has to be assembled form namespace and option name plus a prefix like so: ${NAMESPACE}-{TLSOPTION-NAME}@kubernetescrd.

Second we configure the Ingress itself, referencing the option.

---
apiVersion: traefik.io/v1alpha1
kind: TLSOption
metadata:
  name: mtlsoption
  namespace: example
spec:
  clientAuth:
    clientAuthType: RequireAndVerifyClientCert
    secretNames:
      - ca-mtls
  minVersion: VersionTLS12
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls.options: example-mtlsoption@kubernetescrd
  name: mtlsingress
  namespace: example
spec:
  ingressClassName: traefik
  rules:
    - host: acme.example.com
      http:
        paths:
          - backend:
              service:
                name: YOUR-SERVICE-NAME-HERE
                port:
                  number: YOUR-PORT-NUMBER-HERE
            path: /
            pathType: ImplementationSpecific
  tls:
    - hosts:
        - acme.example.com
      secretName: mtlsingress-tls

Thats it!

Test the setup#

There are several options to test the setup, all you need is a client providing the certificate - and one that don’t to check if it fails without.

One basic test on TLS level can be done using OpenSSL itself:

openssl s_client -connect acme.example.com:443 -key client.key  -cert client.pem -state

This should not return any errors!

Next with curl a real HTTPS request:

curl https://acme.example.com/ --cert client.pem --key client.key -v

This should give you the expected response and no errors!