# Kubernetes Client

Quarkus includes the `kubernetes-client` extension which enables the use of the Fabric8 Kubernetes Client in native mode while also making it easier to work with.

Having a Kubernetes Client extension in Quarkus is very useful in order to unlock the power of Kubernetes Operators. Kubernetes Operators are quickly emerging as a new class of Cloud Native applications. These applications essentially watch the Kubernetes API and react to changes on various resources and can be used to manage the lifecycle of all kinds of complex systems like databases, messaging systems and much more. Being able to write such operators in Java with the very low footprint that native images provide is a great match.

## Configuration

Once you have your Quarkus project configured you can add the `kubernetes-client` extension to your project by running the following command in your project base directory.

CLI

    quarkus extension add 'kubernetes-client'

Maven

    ./mvnw quarkus:add-extension -Dextensions='kubernetes-client'

Gradle

    ./gradlew addExtension --extensions='kubernetes-client'

This will add the following to your build file:

pom.xml

    <dependency>
        <groupId>io.quarkus</groupId>
        <artifactId>quarkus-kubernetes-client</artifactId>
    </dependency>

build.gradle

    implementation("io.quarkus:quarkus-kubernetes-client")

## Usage

Quarkus configures a Bean of type `KubernetesClient` which can be injected into application code using the well known CDI methods. This client can be configured using various properties as can be seen in the following example:

    quarkus.kubernetes-client.trust-certs=false
    quarkus.kubernetes-client.namespace=default

Note that the full list of properties is available in the Dev Services section of the configuration reference.

In dev mode and when running tests, Dev Services for Kubernetes automatically starts a Kubernetes API server.

### Customizing and overriding

Quarkus provides multiple integration points for influencing the Kubernetes Client provided as a CDI bean.

#### Kubernetes Client Config customization

The first integration point is the use of the `io.quarkus.kubernetes.client.KubernetesConfigCustomizer` interface. When such a bean exists, it allows for arbitrary customizations of the `io.fabric8.kubernetes.client.Config` created by Quarkus (which takes into account the `quarkus.kubernetes-client.*` properties).

Alternatively, application code can override the `io.fabric8.kubernetes.client.Config` or even the `io.fabric8.kubernetes.client.KubernetesClient` bean (which are normally provided by the extension) by simply declaring custom versions of those beans.

An example of this can be seen in the following snippet:

    @Singleton
    public class KubernetesClientProducer {

        @Produces
        public KubernetesClient kubernetesClient() {
            // here you would create a custom client
            return new DefaultKubernetesClient();
        }
    }

#### Kubernetes Client ObjectMapper customization

The Fabric8 Kubernetes Client uses its own `ObjectMapper` instance for serialization and deserialization of Kubernetes resources. This mapper is provided to the client through a `KubernetesSerialization` instance that’s injected into the `KubernetesClient` bean.

If for some reason you must customize the default `ObjectMapper` bean provided by this extension and used by the Kubernetes Client, you can do so by declaring a bean that implements the `KubernetesClientObjectMapperCustomizer` interface.

The following code snippet contains an example of a `KubernetesClientObjectMapperCustomizer` to set the `ObjectMapper` locale:

    @Singleton
    public static class Customizer implements KubernetesClientObjectMapperCustomizer {
        @Override
        public void customize(ObjectMapper objectMapper) {
            objectMapper.setLocale(Locale.ROOT);
        }
    }

Furthermore, if you must replace the default `ObjectMapper` bean used by the Kubernetes Client that the extension creates automatically, you can do so by declaring a bean of type `@KubernetesClientObjectMapper`. The following code snippet shows how you can declare this bean:

    @Singleton
    public class KubernetesObjectMapperProducer {
        @KubernetesClientObjectMapper
        @Singleton
        @Produces
        public ObjectMapper kubernetesClientObjectMapper() {
            return new ObjectMapper();
        }
    }

| The static `io.fabric8.kubernetes.client.utils.Serialization` utils class is deprecated and should not be used. Access to `Serialization.jsonMapper()` should be replaced by the usage of @KubernetesClientObjectMapperCustomizer` declared beans.

## Testing

To make testing against a mock Kubernetes API extremely simple, Quarkus provides the `WithKubernetesTestServer` annotation which automatically launches a mock of the Kubernetes API server and sets the proper environment variables needed so that the Kubernetes Client configures itself to use said mock. Tests can inject the mock server and set it up in any way necessary for the particular testing using the `@KubernetesTestServer` annotation.

Let’s assume we have a REST endpoint defined like so:

    @Path("/pod")
    public class Pods {

        private final KubernetesClient kubernetesClient;

        public Pods(KubernetesClient kubernetesClient) {
            this.kubernetesClient = kubernetesClient;
        }

        @GET
        @Path("/{namespace}")
        public List<Pod> pods(String namespace) {
            return kubernetesClient.pods().inNamespace(namespace).list().getItems();
        }
    }

We could write a test for this endpoint very easily like so:

    // you can even configure aspects like crud, https and port on this annotation
    @WithKubernetesTestServer
    @QuarkusTest
    public class KubernetesClientTest {

        @KubernetesTestServer
        KubernetesServer mockServer;

        @BeforeEach
        public void before() {
            final Pod pod1 = new PodBuilder().withNewMetadata().withName("pod1").withNamespace("test").and().build();
            final Pod pod2 = new PodBuilder().withNewMetadata().withName("pod2").withNamespace("test").and().build();

            // Set up Kubernetes so that our "pretend" pods are created
            mockServer.getClient().pods().create(pod1);
            mockServer.getClient().pods().create(pod2);
        }

        @Test
        public void testInteractionWithAPIServer() {
            RestAssured.when().get("/pod/test").then()
                    .body("size()", is(2));
        }

    }

Note that to take advantage of these features, the `quarkus-test-kubernetes-client` dependency needs to be added, for example like so:

pom.xml

    <dependency>
        <groupId>io.quarkus</groupId>
        <artifactId>quarkus-test-kubernetes-client</artifactId>
        <scope>test</scope>
    </dependency>

build.gradle

    testImplementation("io.quarkus:quarkus-test-kubernetes-client")

By default, the mock server will be in CRUD mode, so you have to use the client to build your state before your application can retrieve it, but you can also set it up in non-CRUD mode and mock all HTTP requests made to Kubernetes:

    // you can even configure aspects like crud, https and port on this annotation
    @WithKubernetesTestServer(crud = false)
    @QuarkusTest
    public class KubernetesClientTest {

        @KubernetesTestServer
        KubernetesServer mockServer;

        @BeforeEach
        public void before() {
            final Pod pod1 = new PodBuilder().withNewMetadata().withName("pod1").withNamespace("test").and().build();
            final Pod pod2 = new PodBuilder().withNewMetadata().withName("pod2").withNamespace("test").and().build();

            // Mock any HTTP request to Kubernetes pods so that our pods are returned
            mockServer.expect().get().withPath("/api/v1/namespaces/test/pods")
                    .andReturn(200,
                            new PodListBuilder().withNewMetadata().withResourceVersion("1").endMetadata().withItems(pod1, pod2)
                                    .build())
                    .always();
        }

        @Test
        public void testInteractionWithAPIServer() {
            RestAssured.when().get("/pod/test").then()
                    .body("size()", is(2));
        }

    }

You can also use the `setup` attribute on the `@WithKubernetesTestServer` annotation to provide a class that will configure the `KubernetesServer` instance:

    @WithKubernetesTestServer(setup = MyTest.Setup.class)
    @QuarkusTest
    public class MyTest {

        public static class Setup implements Consumer<KubernetesServer> {

            @Override
            public void accept(KubernetesServer server) {
              server.expect().get().withPath("/api/v1/namespaces/test/pods")
                .andReturn(200, new PodList()).always();
            }
        }

        // tests
    }

Alternately, you can create an extension of the `KubernetesServerTestResource` class to ensure all your `@QuarkusTest` enabled test classes share the same mock server setup via the `QuarkusTestResource` annotation:

    public class CustomKubernetesMockServerTestResource extends KubernetesServerTestResource {

        @Override
        protected void configureServer() {
            super.configureServer();
            server.expect().get().withPath("/api/v1/namespaces/test/pods")
              .andReturn(200, new PodList()).always();
        }
    }

and use this in your other test classes as follows:

    @QuarkusTestResource(CustomKubernetesMockServerTestResource.class)
    @QuarkusTest
    public class KubernetesClientTest {

        //tests will now use the configured server...
    }

## Note on implementing or extending generic types

Due to the restrictions imposed by GraalVM, extra care needs to be taken when implementing or extending generic types provided by the client if the application is intended to work in native mode. Essentially every implementation or extension of generic classes such as `Watcher`, `ResourceHandler` or `CustomResource` needs to specify their associated Kubernetes model class (or, in the case of `CustomResource`, regular Java types) at class definition time. To better understand this, suppose we want to watch for changes to Kubernetes `Pod` resources. There are a couple ways to write such a `Watcher` that are guaranteed to work in native:

    client.pods().watch(new Watcher<Pod>() {
        @Override
        public void eventReceived(Action action, Pod pod) {
            // do something
        }

        @Override
        public void onClose(KubernetesClientException e) {
            // do something
        }
    });

or

    public class PodResourceWatcher implements Watcher<Pod> {
        @Override
        public void eventReceived(Action action, Pod pod) {
            // do something
        }

        @Override
        public void onClose(KubernetesClientException e) {
            // do something
        }
    }

    ...

    client.pods().watch(new PodResourceWatcher());

Note that defining the generic type via a class hierarchy similar to the following example will also work correctly:

    public abstract class MyWatcher<S> implements Watcher<S> {
    }

    ...

    client.pods().watch(new MyWatcher<Pod>() {
        @Override
        public void eventReceived(Action action, Pod pod) {
            // do something
        }
    });

| The following example will not work in native mode because the generic type of watcher cannot be determined by looking at the class and method definitions thus making Quarkus unable to properly determine the Kubernetes model class for which reflection registration is needed:

    public class ResourceWatcher<T extends HasMetadata> implements Watcher<T> {
        @Override
        public void eventReceived(Action action, T resource) {
            // do something
        }

        @Override
        public void onClose(KubernetesClientException e) {
            // do something
        }
    }

    client.pods().watch(new ResourceWatcher<Pod>());

## Note on using Elliptic Curve keys

Please note that if you would like to use Elliptic Curve keys with Kubernetes Client then adding a BouncyCastle PKIX dependency is required:

pom.xml

    <dependency>
        <groupId>org.bouncycastle</groupId>
        <artifactId>bcpkix-jdk18on</artifactId>
    </dependency>

build.gradle

    implementation("org.bouncycastle:bcpkix-jdk18on")

Note that internally an `org.bouncycastle.jce.provider.BouncyCastleProvider` provider will be registered if it has not already been registered.

You can have this provider registered as described in the BouncyCastle or BouncyCastle FIPS sections.

## Access to the Kubernetes API

In many cases in order to access the Kubernetes API server a `ServiceAccount`, `Role` and `RoleBinding` will be necessary. An example that allows listing all pods could look something like this:

    ---
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: <applicationName>
      namespace: <namespace>
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: <applicationName>
      namespace: <namespace>
    rules:
      - apiGroups: [""]
        resources: ["pods"]
        verbs: ["list"]
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: <applicationName>
      namespace: <namespace>
    roleRef:
      kind: Role
      name: <applicationName>
      apiGroup: rbac.authorization.k8s.io
    subjects:
      - kind: ServiceAccount
        name: <applicationName>
        namespace: <namespace>

Replace `<applicationName>` and `<namespace>` with your values. Have a look at Configure Service Accounts for Pods to get further information.

## OpenShift Client

If the targeted Kubernetes cluster is an OpenShift cluster, it is possible to access it through the `openshift-client` extension, in a similar way. This leverages the dedicated fabric8 openshift client, and provides access to `OpenShift` proprietary objects (e.g. `Route`, `ProjectRequest`, `BuildConfig` …​)

Note that the configuration properties are shared with the `kubernetes-client` extension. In particular, they have the same `quarkus.kubernetes-client` prefix.

Add the extension with:

CLI

    quarkus extension add 'openshift-client'

Maven

    ./mvnw quarkus:add-extension -Dextensions='openshift-client'

Gradle

    ./gradlew addExtension --extensions='openshift-client'

Note that `openshift-client` extension has a dependency on the `kubernetes-client` extension.

To use the client, inject an `OpenShiftClient` instead of the `KubernetesClient`:

    @Inject
    private OpenShiftClient openshiftClient;

If you need to override the default `OpenShiftClient`, provide a producer such as:

    @Singleton
    public class OpenShiftClientProducer {

        @Produces
        public OpenShiftClient openshiftClient() {
            // here you would create a custom client
            return new DefaultOpenShiftClient();
        }
    }

Mock support is also provided in a similar fashion:

    @QuarkusTestResource(OpenShiftMockServerTestResource.class)
    @QuarkusTest
    public class OpenShiftClientTest {

        @MockServer
        private OpenShiftMockServer mockServer;
    ...

Or by using the `@WithOpenShiftTestServer` similar to the `@WithKubernetesTestServer` explained in the previous section:

    @WithOpenShiftTestServer
    @QuarkusTest
    public class OpenShiftClientTest {

        @OpenShiftTestServer
        private OpenShiftServer mockOpenShiftServer;
    ...

To use this feature, you have to add a dependency on `quarkus-test-openshift-client`:

pom.xml

    <dependency>
        <groupId>io.quarkus</groupId>
        <artifactId>quarkus-test-openshift-client</artifactId>
        <scope>test</scope>
    </dependency>

build.gradle

    testImplementation("io.quarkus:quarkus-test-openshift-client")

## Configuration Reference

Configuration property fixed at build time - All other configuration properties are overridable at runtime

Configuration property | Type | Default
--- | --- | ---
`quarkus.kubernetes-client.trust-certs` Whether the client should trust a self-signed certificate if so presented by the API server Environment variable: `QUARKUS_KUBERNETES_CLIENT_TRUST_CERTS` | boolean |
`quarkus.kubernetes-client.api-server-url` URL of the Kubernetes API server Environment variable: `QUARKUS_KUBERNETES_CLIENT_API_SERVER_URL` | string |
`quarkus.kubernetes-client.namespace` Default namespace to use Environment variable: `QUARKUS_KUBERNETES_CLIENT_NAMESPACE` | string |
`quarkus.kubernetes-client.ca-cert-file` CA certificate file Environment variable: `QUARKUS_KUBERNETES_CLIENT_CA_CERT_FILE` | string |
`quarkus.kubernetes-client.ca-cert-data` CA certificate data Environment variable: `QUARKUS_KUBERNETES_CLIENT_CA_CERT_DATA` | string |
`quarkus.kubernetes-client.client-cert-file` Client certificate file Environment variable: `QUARKUS_KUBERNETES_CLIENT_CLIENT_CERT_FILE` | string |
`quarkus.kubernetes-client.client-cert-data` Client certificate data Environment variable: `QUARKUS_KUBERNETES_CLIENT_CLIENT_CERT_DATA` | string |
`quarkus.kubernetes-client.client-key-file` Client key file Environment variable: `QUARKUS_KUBERNETES_CLIENT_CLIENT_KEY_FILE` | string |
`quarkus.kubernetes-client.client-key-data` Client key data Environment variable: `QUARKUS_KUBERNETES_CLIENT_CLIENT_KEY_DATA` | string |
`quarkus.kubernetes-client.client-key-algo` Client key algorithm Environment variable: `QUARKUS_KUBERNETES_CLIENT_CLIENT_KEY_ALGO` | string |
`quarkus.kubernetes-client.client-key-passphrase` Client key passphrase Environment variable: `QUARKUS_KUBERNETES_CLIENT_CLIENT_KEY_PASSPHRASE` | string |
`quarkus.kubernetes-client.username` Kubernetes auth username Environment variable: `QUARKUS_KUBERNETES_CLIENT_USERNAME` | string |
`quarkus.kubernetes-client.password` Kubernetes auth password Environment variable: `QUARKUS_KUBERNETES_CLIENT_PASSWORD` | string |
`quarkus.kubernetes-client.token` Kubernetes oauth token Environment variable: `QUARKUS_KUBERNETES_CLIENT_TOKEN` | string |
`quarkus.kubernetes-client.watch-reconnect-interval` Watch reconnect interval Environment variable: `QUARKUS_KUBERNETES_CLIENT_WATCH_RECONNECT_INTERVAL` | Duration | `PT1S`
`quarkus.kubernetes-client.watch-reconnect-limit` Maximum reconnect attempts in case of watch failure By default there is no limit to the number of reconnect attempts Environment variable: `QUARKUS_KUBERNETES_CLIENT_WATCH_RECONNECT_LIMIT` | int | `-1`
`quarkus.kubernetes-client.connection-timeout` Maximum amount of time to wait for a connection with the API server to be established Environment variable: `QUARKUS_KUBERNETES_CLIENT_CONNECTION_TIMEOUT` | Duration | `PT10S`
`quarkus.kubernetes-client.request-timeout` Maximum amount of time to wait for a request to the API server to be completed Environment variable: `QUARKUS_KUBERNETES_CLIENT_REQUEST_TIMEOUT` | Duration | `PT10S`
`quarkus.kubernetes-client.request-retry-backoff-limit` Maximum number of retry attempts for API requests that fail with an HTTP code of >= 500 Environment variable: `QUARKUS_KUBERNETES_CLIENT_REQUEST_RETRY_BACKOFF_LIMIT` | int | `0`
`quarkus.kubernetes-client.request-retry-backoff-interval` Time interval between retry attempts for API requests that fail with an HTTP code of >= 500 Environment variable: `QUARKUS_KUBERNETES_CLIENT_REQUEST_RETRY_BACKOFF_INTERVAL` | Duration | `PT1S`
`quarkus.kubernetes-client.http-proxy` HTTP proxy used to access the Kubernetes API server Environment variable: `QUARKUS_KUBERNETES_CLIENT_HTTP_PROXY` | string |
`quarkus.kubernetes-client.https-proxy` HTTPS proxy used to access the Kubernetes API server Environment variable: `QUARKUS_KUBERNETES_CLIENT_HTTPS_PROXY` | string |
`quarkus.kubernetes-client.proxy-username` Proxy username Environment variable: `QUARKUS_KUBERNETES_CLIENT_PROXY_USERNAME` | string |
`quarkus.kubernetes-client.proxy-password` Proxy password Environment variable: `QUARKUS_KUBERNETES_CLIENT_PROXY_PASSWORD` | string |
`quarkus.kubernetes-client.no-proxy` IP addresses or hosts to exclude from proxying Environment variable: `QUARKUS_KUBERNETES_CLIENT_NO_PROXY` | list of string |
`quarkus.kubernetes-client.generate-rbac` Enable the generation of the RBAC manifests. If enabled and no other role binding are provided using the properties `quarkus.kubernetes.rbac.`, it will generate a default role binding using the role "view" and the application service account. Environment variable: `QUARKUS_KUBERNETES_CLIENT_GENERATE_RBAC` | boolean | `true`
Dev Services | Type | Default
`quarkus.kubernetes-client.devservices.enabled` If Dev Services for Kubernetes should be used. (default to true) If this is true and kubernetes client is not configured then a kubernetes cluster will be started and will be used. Environment variable: `QUARKUS_KUBERNETES_CLIENT_DEVSERVICES_ENABLED` | boolean | `true`
`quarkus.kubernetes-client.devservices.api-version` The kubernetes api server version to use. If not set, Dev Services for Kubernetes will use the latest supported version of the given flavor. see https://github.com/dajudge/kindcontainer/blob/master/k8s-versions.json Environment variable: `QUARKUS_KUBERNETES_CLIENT_DEVSERVICES_API_VERSION` | string |
`quarkus.kubernetes-client.devservices.flavor` The flavor to use (kind, k3s or api-only). Default to api-only. Environment variable: `QUARKUS_KUBERNETES_CLIENT_DEVSERVICES_FLAVOR` | `kind`, `k3s`, `api-only` | `api-only`
`quarkus.kubernetes-client.devservices.override-kubeconfig` By default, if a kubeconfig is found, Dev Services for Kubernetes will not start. Set this to true to override the kubeconfig config. Environment variable: `QUARKUS_KUBERNETES_CLIENT_DEVSERVICES_OVERRIDE_KUBECONFIG` | boolean | `false`
`quarkus.kubernetes-client.devservices.shared` Indicates if the Kubernetes cluster managed by Quarkus Dev Services is shared. When shared, Quarkus looks for running containers using label-based service discovery. If a matching container is found, it is used, and so a second one is not started. Otherwise, Dev Services for Kubernetes starts a new container. The discovery uses the `quarkus-dev-service-kubernetes` label. The value is configured using the `service-name` property. Container sharing is only used in dev mode. Environment variable: `QUARKUS_KUBERNETES_CLIENT_DEVSERVICES_SHARED` | boolean | `true`
`quarkus.kubernetes-client.devservices.service-name` The value of the `quarkus-dev-service-kubernetes` label attached to the started container. This property is used when `shared` is set to `true`. In this case, before starting a container, Dev Services for Kubernetes looks for a container with the `quarkus-dev-service-kubernetes` label set to the configured value. If found, it will use this container instead of starting a new one. Otherwise, it starts a new container with the `quarkus-dev-service-kubernetes` label set to the specified value. This property is used when you need multiple shared Kubernetes clusters. Environment variable: `QUARKUS_KUBERNETES_CLIENT_DEVSERVICES_SERVICE_NAME` | string | `kubernetes`
`quarkus.kubernetes-client.devservices.container-env` Environment variables that are passed to the container. Environment variable: `QUARKUS_KUBERNETES_CLIENT_DEVSERVICES_CONTAINER_ENV` | `Map<String,String>` |

| About the Duration format The format for durations uses the standard `java.time.Duration` format. You can learn more about it in the Duration#parse() javadoc. You can also provide duration values starting with a number. In this case, if the value consists only of a number, the converter treats the value as seconds. Otherwise, `PT` is implicitly prepended to the value to obtain a standard `java.time.Duration` format.