Setup transport encryption in Eventing

This page describes the transport encryption feature which allows transporting data and events over secured and encrypted HTTPS connections using TLS.

OpenShift Serverless transport encryption for Eventing is a Developer Preview feature only. Developer Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Developer Preview features, see access.redhat.com/support/offerings/devpreview/.

Prerequisites

  • You have access to an OpenShift account with cluster administrator access.

  • Install the OpenShift CLI (oc).

  • Install the OpenShift Serverless Operator.

  • Install the OpenShift Cert-Manager Operator.

Setup a SelfSigned ClusterIssuer

ClusterIssuers, are Kubernetes resources that represent certificate authorities (CAs) that are able to generate signed certificates by honoring certificate signing requests. All cert-manager certificates require a referenced issuer that is in a ready condition to attempt to honor the request.
Reference: cert-manager.io/docs/concepts/issuer/

For the simplicity of this guide, we will use a SelfSigned issuer as root certificate, however, be aware of the implications and limitations as documented at cert-manager.io/docs/configuration/selfsigned/ of this method.
If you’re running your company specific Private Key Infrastructure (PKI), we recommend the CA issuer. Refer to the cert-manager documentation for more details: cert-manager.io/docs/configuration/ca/, however, you can use any other issuer that is usable for cluster-local services.

  1. Create a SelfSigned ClusterIssuer:

    apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: knative-eventing-selfsigned-issuer
spec:
  selfSigned: {}

  2. Apply the ClusterIssuer resource:

    $ oc apply -f <filename>

  3. Create a root certificate using the previously created SelfSigned ClusterIssuer:

    apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: knative-eventing-selfsigned-ca
  namespace: cert-manager (1)
spec:
  secretName: knative-eventing-ca (2)

  isCA: true
  commonName: selfsigned-ca
  privateKey:
    algorithm: ECDSA
    size: 256

  issuerRef:
    name: knative-eventing-selfsigned-issuer
    kind: ClusterIssuer
    group: cert-manager.io
    1 The OpenShift Cert-Manager Operator namespace, cert-manager by default.
    2 Secret name later used for the ClusterIssuer for Eventing

  4. Apply the Certificate resource:

    $ oc apply -f <filename>

Setup ClusterIssuer for Eventing

  1. Create the knative-eventing-ca-issuer ClusterIssuer for Eventing:

    # This is the issuer that every Eventing component should use to issue their server's certs.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: knative-eventing-ca-issuer
spec:
  ca:
    secretName: knative-eventing-ca (1)
    1 Secret name in the OpenShift Cert-Manager Operator namespace (cert-manager by default) containing the certificate that can then be used by Knative Eventing components for new certificates.

    The name of the ClusterIssuer must be knative-eventing-ca-issuer.

  2. Apply the ClusterIssuer resource:

    $ oc apply -f <filename>

Understanding the transport encryption configuration

The transport-encryption feature flag is an enum configuration that configures how Addressables (Broker, Channel, Sink) should or must accept events.

The possible values for transport-encryption are:

  • disabled (this is equivalent to the current behavior)

    • Addressables may accept events to HTTPS endpoints

    • Producers may send events to HTTPS endpoints

  • permissive

    • Addressables should accept events on both HTTP and HTTPS endpoints

    • Addressables should advertise both HTTP and HTTPS endpoints

    • Producers should prefer sending events to HTTPS endpoints, if available

  • strict

    • Addressables must not accept events to non-HTTPS endpoints

    • Addressables must only advertise HTTPS endpoints

Setup transport encryption in KnativeEventing

  1. Enabling transport-encryption in KnativeEventing:

    apiVersion: operator.knative.dev/v1beta1
kind: KnativeEventing
metadata:
  name: knative-eventing
  namespace: knative-eventing
spec:

  # Other spec fields omitted ...
  # ...

  config:
    features:
      transport-encryption: strict (1)
    1 Configure transport-encryption to strict.

  2. Apply the KnativeEventing resource:

    $ oc apply -f <filename>

Configure additional CA trust bundles

By default, Eventing clients trusts the OpenShift CA bundle that you have configured when using a custom PKI for OpenShift, as documented at docs.openshift.com/container-platform/4.14/networking/configuring-a-custom-pki.html.

If you need to add additional CA bundles for Eventing, you can do so by creating ConfigMaps in the knative-eventing namespace with label networking.knative.dev/trust-bundle: true:

Whenever CA bundles ConfigMaps are updated, the Eventing clients will automatically add them to their trusted CA bundles when a new connection is established.

  1. Create a CA bundle for Eventing:

kind: ConfigMap
metadata:
  name: my-org-eventing-bundle (1)
  namespace: knative-eventing
  labels:
    networking.knative.dev/trust-bundle: "true"
data: (2)
  ca.crt: ...
  ca1.crt: ...
  tls.crt: ...
1 Use a name that is unlikely to conflict with existing or future Eventing ConfigMap.
2 All keys containing valid PEM-encoded CA bundles will be trusted by Eventing clients.

Configure custom event sources to trust the Eventing CA

The recommended way of creating custom event sources is using a SinkBinding, SinkBinding will inject the configured CA trust bundles as projected volume into each container using the directory /knative-custom-certs.

Some organizations might inject company specific CA trust bundles into base container images and automatically configure runtimes (openjdk, node, etc) to trust that CA bundle.
In that case, you might not need to configure your clients.

Using the previous example of the my-org-eventing-bundle ConfigMap with data keys being ca.crt, ca1.crt and tls.crt, you will have a /knative-custom-certs directory that will have the following layout:

/knative-custom-certs/ca.crt
/knative-custom-certs/ca1.crt
/knative-custom-certs/tls.crt

Those files can then be used to add CA trust bundles to HTTP clients sending events to Eventing.

Depending on the runtime, programming language or library that you’re using, there are different ways of configuring custom CA certs files using command line flags, environment variables, or by reading the content of those files.
Refer to their documentation for more details.

Adding SelfSigned ClusterIssuer to CA trust bundles

In case you are using a SelfSigned ClusterIssuer as described in the Setup a SelfSigned ClusterIssuer section, you can add the CA to the Eventing CA trust bundles by running the following commands:

  1. Export the CA from the knative-eventing-ca secret in the OpenShift Cert-Manager Operator namespace, cert-manager by default:

    $ oc get secret -n cert-manager knative-eventing-ca -o=jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt

  2. Create a CA trust bundle in the knative-eventing namespace:

    $ oc create configmap -n knative-eventing my-org-selfsigned-ca-bundle --from-file=ca.crt

  3. Label the ConfigMap with networking.knative.dev/trust-bundle: "true" label:

    $ oc label configmap -n knative-eventing my-org-selfsigned-ca-bundle networking.knative.dev/trust-bundle=true

Ensure seamless CA rotation

Ensuring seamless CA rotation is essential to avoid service downtime, or to deal with an emergency. The following procedure explains how you can seamlessly rotate a CA.

  1. Create a new CA certificate.

  2. Add the public key of the new CA certificate to the CA trust bundles as described in the Configure additional CA trust bundles section.
    Make sure to also keep the public key of the existing CA.

  3. Ensure that all clients have consumed the latest set of CA trust bundles.
    Knative Eventing components will automatically reload the changed CA trust bundles. If you have custom workload consuming trust bundles as well, make sure to reload/restart them accordingly.

  4. Update the knative-eventing-ca-issuer ClusterIssuer to reference the secret containing the CA certificate created at step 1.

  5. Force cert-manager to renew certificates in the knative-eventing namespace.
    Refer to the cert-manager documentation for more details: cert-manager.io/docs/usage/certificate/#reissuance-triggered-by-user-actions.

  6. As soon as the CA rotation is fully completed, you can remove the public key of the old CA from the trust bundle ConfigMap.

Verification

  1. Create an InMemoryChannel:

    apiVersion: messaging.knative.dev/v1
kind: InMemoryChannel
metadata:
 name: transport-encryption-test

  2. Apply the InMemoryChannel resource:

    $ oc apply -f <filename>

  3. View the InMemoryChannel address:

    $ oc get inmemorychannels.messaging.knative.dev transport-encryption-test
    Example output
    NAME                        URL                                                                                           AGE   READY   REASON
transport-encryption-test   https://imc-dispatcher.knative-eventing.svc.cluster.local/default/transport-encryption-test   17s   True