Note: Impatient readers may head straight to Quick Start.

Using Kubebuilder v1? Check the legacy documentation

Who is this for

Users of Kubernetes

Users of Kubernetes will develop a deeper understanding of Kubernetes through learning the fundamental concepts behind how APIs are designed and implemented. This book will teach readers how to develop their own Kubernetes APIs and the principles from which the core Kubernetes APIs are designed.

Including:

  • The structure of Kubernetes APIs and Resources
  • API versioning semantics
  • Self-healing
  • Garbage Collection and Finalizers
  • Declarative vs Imperative APIs
  • Level-Based vs Edge-Base APIs
  • Resources vs Subresources

Kubernetes API extension developers

API extension developers will learn the principals and concepts behind implementing canonical Kubernetes APIs, as well as simple tools and libraries for rapid execution. This book covers pitfalls and misconceptions that extension developers commonly encounter.

Including:

  • How to batch multiple events into a single reconciliation call
  • How to configure periodic reconciliation
  • Forthcoming
    • When to use the lister cache vs live lookups
    • Garbage Collection vs Finalizers
    • How to use Declarative vs Webhook Validation
    • How to implement API versioning

Resources

Quick Start

This Quick Start guide will cover:

Installation

Install kubebuilder:

os=$(go env GOOS)
arch=$(go env GOARCH)

# download kubebuilder and extract it to tmp
curl -sL https://go.kubebuilder.io/dl/2.0.0-beta.0/${os}/${arch} | tar -xz -C /tmp/

# move to a long-term location and put it on your path
# (you'll need to set the KUBEBUILDER_ASSETS env var if you put it somewhere else)
sudo mv /tmp/kubebuilder_2.0.0-beta.0_${os}_${arch} /usr/local/kubebuilder
export PATH=$PATH:/usr/local/kubebuilder/bin

You can also install a KubeBuilder master snapshot from https://go.kubebuilder.io/dl/latest/${os}/${arch}.

Install kustomize v3.0.1+

Create a Project

Initialize a new project and Go module for your controllers:

kubebuilder init --domain my.domain

If you’re not in GOPATH, you’ll need to run go mod init <modulename> in order to tell kubebuilder and Go the base import path of your module.

Create an API

Create a new API group-version called webapp/v1, and a kind Guestbook in that API group-version:

kubebuilder create api --group webapp --version v1 --kind Guestbook

This will create the files api/v1/guestbook_types.go and controller/guestbook_controller.go for you to edit.

Optional: Edit the API definition or the reconciliation business logic. For more on this see What’s in a Controller and Designing an API.

Test It Out Locally

You’ll need a Kubernetes cluster to run against. You can use KIND to get a local cluster for testing, or run against a remote cluster.

Your controller will automatically use the current context in your kubeconfig file (i.e. whatever cluster kubectl cluster-info shows).

Install the CRDs into the cluster:

make install

Run your controller (this will run in the foreground, so switch to a new terminal if you want to leave it running):

make run

Install Samples

Create your samples (make sure to edit them first if you’ve changed the API definition):

kubectl apply -f config/samples/

Run It On the Cluster

Build and push your image to the location specified by IMG:

make docker-build docker-push IMG=<some-registry>/controller

Deploy the controller to the cluster:

make deploy

If you encounter RBAC errors, you may need to grant yourself cluster-admin privileges:

Tutorial: Building CronJob

Too many tutorials start out with some really contrived setup, or some toy application that gets the basics across, and then stalls out on the more complicated stuff. Instead, this tutorial should take you through (almost) the full gamut of complexity with Kubebuilder, starting off simple and building up to something pretty full-featured.

Let’s pretend (and sure, this is a teensy bit contrived) that we’ve finally gotten tired of the maintenance burden of the non-Kubebuilder implementation of the CronJob controller in Kubernetes, and we’d like to rewrite it using KubeBuilder.

The job (no pun intended) of the CronJob controller is to run one-off tasks on the Kubernetes cluster at regular intervals. It does this by building on top of the Job controller, whose task is to run one-off tasks once, seeing them to completion.

Instead of trying to tackle rewriting the Job controller as well, we’ll use this as an opportunity to see how to interact with external types.

Scaffolding Out Our Project

As covered in the quick start, we’ll need to scaffold out a new project. Make sure you’ve installed Kubebuilder, then scaffold out a new project:

# we'll use a domain of tutorial.kubebuilder.io,
# so all API groups will be <group>.tutorial.kubebuilder.io.
kubebuilder init --domain tutorial.kubebuilder.io

Now that we’ve got a project in place, let’s take a look at what Kubebuilder has scaffolded for us so far...

What’s in a basic project?

When scaffolding out a new project, Kubebuilder provides us with a few basic pieces of boilerplate.

Build Infrastructure

First up, basic infrastructure for building your project:

go.mod: A new Go module matching our project, with basic dependencies

module tutorial.kubebuilder.io/project

go 1.12

require (
    github.com/go-logr/logr v0.1.0
    github.com/gogo/protobuf v1.2.1 // indirect
    github.com/json-iterator/go v1.1.6 // indirect
    github.com/modern-go/reflect2 v1.0.1 // indirect
    github.com/onsi/ginkgo v1.8.0 // indirect
    github.com/onsi/gomega v1.5.0 // indirect
    github.com/robfig/cron v1.1.0
    github.com/spf13/pflag v1.0.3 // indirect
    golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09 // indirect
    golang.org/x/sys v0.0.0-20190429190828-d89cdac9e872 // indirect
    golang.org/x/text v0.3.2 // indirect
    gopkg.in/yaml.v2 v2.2.2 // indirect
    k8s.io/api v0.0.0-20190409021203-6e4e0e4f393b
    k8s.io/apimachinery v0.0.0-20190404173353-6a84e37a896d
    k8s.io/client-go v11.0.1-0.20190409021438-1a26190bd76a+incompatible
    sigs.k8s.io/controller-runtime v0.2.0-beta.4
)

Makefile: Make targets for building and deploying your controller


# Image URL to use all building/pushing image targets
IMG ?= controller:latest
# Produce CRDs that work back to Kubernetes 1.11 (no version conversion)
CRD_OPTIONS ?= "crd:trivialVersions=true"

all: manager

# Run tests
test: generate fmt vet manifests
    go test ./api/... ./controllers/... -coverprofile cover.out

# Build manager binary
manager: generate fmt vet
    go build -o bin/manager main.go

# Run against the configured Kubernetes cluster in ~/.kube/config
run: generate fmt vet
    go run ./main.go

# Install CRDs into a cluster
install: manifests
    kubectl apply -f config/crd/bases

# Deploy controller in the configured Kubernetes cluster in ~/.kube/config
deploy: manifests
    kubectl apply -f config/crd/bases
    kustomize build config/default | kubectl apply -f -

# Generate manifests e.g. CRD, RBAC etc.
manifests: controller-gen
    $(CONTROLLER_GEN) $(CRD_OPTIONS) rbac:roleName=manager-role webhook paths="./api/...;./controllers/..." output:crd:artifacts:config=config/crd/bases

# Run go fmt against code
fmt:
    go fmt ./...

# Run go vet against code
vet:
    go vet ./...

# Generate code
generate: controller-gen
    $(CONTROLLER_GEN) object:headerFile=./hack/boilerplate.go.txt paths=./api/...

# Build the docker image
docker-build: test
    docker build . -t ${IMG}
    @echo "updating kustomize image patch file for manager resource"
    sed -i'' -e 's@image: .*@image: '"${IMG}"'@' ./config/default/manager_image_patch.yaml

# Push the docker image
docker-push:
    docker push ${IMG}

# find or download controller-gen
# download controller-gen if necessary
controller-gen:
ifeq (, $(shell which controller-gen))
    go get sigs.k8s.io/controller-tools/cmd/controller-gen@v0.2.0-beta.3
CONTROLLER_GEN=$(shell go env GOPATH)/bin/controller-gen
else
CONTROLLER_GEN=$(shell which controller-gen)
endif

PROJECT: Kubebuilder metadata for scaffolding new components

version: "2"
domain: tutorial.kubebuilder.io
repo: tutorial.kubebuilder.io/project

Launch Configuration

We also get launch configurations under the config/ directory. Right now, it just contains Kustomize YAML definitions required to launch our controller on a cluster, but once we get started writing our controller, it’ll also hold our CustomResourceDefinitions, RBAC configuration, and WebhookConfigurations.

config/default contains a Kustomize base for launching the controller in a standard configuration.

Each other directory contains a different piece of configuration, refactored out into its own base:

  • config/manager: launch your controllers as pods in the cluster

  • config/rbac: permissions required to run your controllers under their own service account

The Entrypoint

Last, but certainly not least, Kubebuilder scaffolds out the basic entrypoint of our project: main.go. Let’s take a look at that next...

Every journey needs a start, every program a main

Apache License

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Our package starts out with some basic imports. Particularly:

  • The core controller-runtime library
  • The default controller-runtime logging, Zap (more on that a bit later)


package main

import (
    "flag"
    "os"

    "k8s.io/apimachinery/pkg/runtime"
    _ "k8s.io/client-go/plugin/pkg/client/auth/gcp"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/log/zap"
    // +kubebuilder:scaffold:imports
)

Every set of controllers needs a Scheme, which provides mappings between Kinds and their corresponding Go types. We’ll talk a bit more about Kinds when we write our API definition, so just keep this in mind for later.


var (
    scheme   = runtime.NewScheme()
    setupLog = ctrl.Log.WithName("setup")
)

func init() {

    // +kubebuilder:scaffold:scheme
}

At this point, our main function is fairly simple:

  • We set up some basic flags for metrics.

  • We instantiate a manager, which keeps track of running all of our controllers, as well as setting up shared caches and clients to the API server (notice we tell the manager about our Scheme).

  • We run our manager, which in turn runs all of our controllers and webhooks. The manager is set up to run until it receives a graceful shutdown signal. This way, when we’re running on Kubernetes, we behave nicely with graceful pod termination.

While we don’t have anything to run just yet, remember where that +kubebuilder:scaffold:builder comment is -- things’ll get interesting there soon.



func main() {
    var metricsAddr string
    flag.StringVar(&metricsAddr, "metrics-addr", ":8080", "The address the metric endpoint binds to.")
    flag.Parse()

    ctrl.SetLogger(zap.Logger(true))

    mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{Scheme: scheme, MetricsBindAddress: metricsAddr})
    if err != nil {
        setupLog.Error(err, "unable to start manager")
        os.Exit(1)
    }

    // +kubebuilder:scaffold:builder

    setupLog.Info("starting manager")
    if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
        setupLog.Error(err, "problem running manager")
        os.Exit(1)
    }
}

With that out of the way, we can get on to scaffolding our API!

Groups and Versions and Kinds, oh my!

Actually, before we get started with our API, we should talk terminology a bit.

When we talk about APIs in Kubernetes, we often use 4 terms: groups, versions, kinds, and resources.

Groups and Versions

An API Group in Kubernetes is simply a collection of related functionality. Each group has one or more versions, which, as the name suggests, allow us to change how an API works over time.

Kinds and Resources

Each API group-version contains one or more API types, which we call Kinds. While a Kind may change forms between versions, each form must be able to store all the data of the other forms, somehow (we can store the data in fields, or in annotations). This means that using an older API version won’t cause newer data to be lost or corrupted. See the Kubernetes API guidelines for more information.

You’ll also hear mention of resources on occasion. A resource is simply a use of a Kind in the API. Often, there’s a one-to-one mapping between Kinds and resources. For instance, the pods resource corresponds to the Pod Kind. However, sometimes, the same Kind may be returned by multiple resources. For instance, the Scale Kind is returned by all scale subresources, like deployments/scale or replicasets/scale. This is what allows the Kubernetes HorizontalPodAutoscaler to interact with different resources. With CRDs, however, each Kind will correspond to a single resource.

Notice that resources are always lowercase, and by convention are the lowercase form of the Kind.

So, how does that correspond to Go?

When we refer to a kind in a particular group-version, we’ll call it a GroupVersionKind, or GVK for short. Same with resources and GVR. As we’ll see shortly, each GVK corresponds to a given root Go type in a package.

Now that we have our terminology straight, we can actually create our API!

Err, but what’s that Scheme thing?

The Scheme we saw before is simply a way to keep track of what Go type corresponds to a given GVK (don’t be overwhelemed by its godocs).

For instance, suppose we mark that the "tutorial.kubebuilder.io/api/v1".CronJob{} type as being in the batch.tutorial.kubebuilder.io/v1 API group (implicitly saying it has the Kind CronJob).

Then, we can later construct a new &CronJob{} given some JSON from the API server that says

{
    "kind": "CronJob",
    "apiVersion": "batch.tutorial.kubebuilder.io/v1",
    ...
}

or properly look up the group-version when we go to submit a &CronJob{} in an update.

Adding a new API

To scaffold out a new Kind (you were paying attention to the last chapter, right?) and corresponding controller, we can use kubebuilder create api:

kubebuilder create api --group batch --version v1 --kind CronJob

The first time we call this command for each group-version, it will create a directory for the new group-version.

In this case, the api/v1/ directory is created, corresponding to the batch.tutorial.kubebuilder.io/v1 (remember our --domain setting from the beginning?).

It has also added a file for our CronJob Kind, api/v1/cronjob_types.go. Each time we call the command with a different kind, it’ll add a corresponding new file.

Let’s take a look at what we’ve been given out of the box, then we can move on to filling it out.

Apache License

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

We start out simply enough: we import the meta/v1 API group, which is not normally exposed by itself, but instead contains metadata common to all Kubernetes Kinds.


package v1

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

Next, we get types for the Spec and Status of our Kind. Kubernetes functions by reconciling desired state (Spec) with actual cluster state (other objects’ Status) and external state, and then recording what it observed (Status). Thus, every functional object includes spec and status. A few types, like ConfigMap don’t follow this pattern, since they don’t encode desired state, but most types do.


// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.

// CronJobSpec defines the desired state of CronJob
type CronJobSpec struct {
    // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
    // Important: Run "make" to regenerate code after modifying this file
}

// CronJobStatus defines the observed state of CronJob
type CronJobStatus struct {
    // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
    // Important: Run "make" to regenerate code after modifying this file
}

Next, we get the types corresponding to actual Kinds, CronJob and CronJobList. CronJob is our root type, and describes the CronJob kind. Like all Kubernetes objects, it contains TypeMeta (which describes API version and Kind), and also contains ObjectMeta, which holds things like name, namespace, and labels.

CronJobList is simply a container for multiple CronJobs. It’s the Kind used in bulk operations, like LIST.

In general, we never modify either of these -- all modifications go in either Spec or Status

That little +kubebuilder:object:root comment is called a marker. We’ll see more of them in a bit, but know that they act as extra metadata, telling controller-tools (our code and YAML generator) extra information. This particular one tells the object generator that this type represents a Kind. Then, the object generator generates an implementation of the runtime.Object interface for us, which is the standard interface that all types representing Kinds must implement.



// +kubebuilder:object:root=true

// CronJob is the Schema for the cronjobs API
type CronJob struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   CronJobSpec   `json:"spec,omitempty"`
    Status CronJobStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true

// CronJobList contains a list of CronJob
type CronJobList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []CronJob `json:"items"`
}

Finally, we add the Go types to the API group. This allows us to add the types in this API group to any Scheme.


func init() {
    SchemeBuilder.Register(&CronJob{}, &CronJobList{})
}

Now that we’ve seen the basic structure, let’s fill it out!

Designing an API

In Kubernetes, we have a few rules for how we design APIs. Namely, all serialized fields must be camelCase, so we use JSON struct tags to specify this. We can also use the omitempty struct tag to mark that a field should be omitted from serialization when empty.

Fields may use most of the primitive types. Numbers are the exception: for API compatibility purposes, we accept two forms of numbers: int32 for integers, and resource.Quantity for decimals.

Hold up, what’s a Quantity?

Quantities are a special notation for decimal numbers that have an explicitly fixed representation that makes them more portable across machines. You’ve probably noticed them when specifying resources requests and limits on pods in Kubernetes.

They conceptually work similar to floating point numbers: they have a significand, base, and exponent. Their serialize, human readable for uses whole numbers and suffixes to specify values much the way we describe computer storage.

For instance, the value 2m means 0.002 in decimal notation. 2Ki means 2048 in decimal, while 2K means 2000 in decimal. If we want to specify fractions, we switch to a suffix that lets us use a whole number: 2.5 is 2500m.

There are two supported bases: 10 and 2 (called decimal and binary, respectively). Decimal base is indicated with “normal” SI suffixes (e.g. M and K), while Binary base is specified in “mebi” notation (e.g. Mi and Ki). Think megabytes vs mebibytes.

There’s one other special type that we use: metav1.Time. This functions identically to time.Time, except that it has a fixed, portable serialization format.

With that out of the way, let’s take a look at what our CronJob object looks like!

Apache License

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Imports



package v1

import (
    batchv1beta1 "k8s.io/api/batch/v1beta1"
    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.

First, let’s take a look at our spec. As we discussed before, spec holds desired state, so any “inputs” to our controller go here.

Fundamentally a CronJob needs the following pieces:

  • A schedule (the cron in CronJob)
  • A template for the Job to run (the job in CronJob)

We’ll also want a few extras, which will make our users’ lives easier:

  • A deadline for starting jobs (if we miss this deadline, we’ll just wait till the next scheduled time)
  • What to do if multiple jobs would run at once (do we wait? stop the old one? run both?)
  • A way to pause the running of a CronJob, in case something’s wrong with it
  • Limits on old job history

Remember, since we never read our own status, we need to have some other way to keep track of whether a job has run. We can use at least one old job to do this.

We’ll use several markers (// +comment) to specify additional metadata. These will be used by controller-tools when generating our CRD manifest. As we’ll see in a bit, controller-tools will also use GoDoc to form descriptions for the fields.



// CronJobSpec defines the desired state of CronJob
type CronJobSpec struct {
    // +kubebuilder:validation:MinLength=0

    // The schedule in Cron format, see https://en.wikipedia.org/wiki/Cron.
    Schedule string `json:"schedule"`

    // +kubebuilder:validation:Minimum=0

    // Optional deadline in seconds for starting the job if it misses scheduled
    // time for any reason.  Missed jobs executions will be counted as failed ones.
    // +optional
    StartingDeadlineSeconds *int64 `json:"startingDeadlineSeconds,omitempty"`

    // Specifies how to treat concurrent executions of a Job.
    // Valid values are:
    // - "Allow" (default): allows CronJobs to run concurrently;
    // - "Forbid": forbids concurrent runs, skipping next run if previous run hasn't finished yet;
    // - "Replace": cancels currently running job and replaces it with a new one
    // +optional
    ConcurrencyPolicy ConcurrencyPolicy `json:"concurrencyPolicy,omitempty"`

    // This flag tells the controller to suspend subsequent executions, it does
    // not apply to already started executions.  Defaults to false.
    // +optional
    Suspend *bool `json:"suspend,omitempty"`

    // Specifies the job that will be created when executing a CronJob.
    JobTemplate batchv1beta1.JobTemplateSpec `json:"jobTemplate"`

    // +kubebuilder:validation:Minimum=0

    // The number of successful finished jobs to retain.
    // This is a pointer to distinguish between explicit zero and not specified.
    // +optional
    SuccessfulJobsHistoryLimit *int32 `json:"successfulJobsHistoryLimit,omitempty"`

    // +kubebuilder:validation:Minimum=0

    // The number of failed finished jobs to retain.
    // This is a pointer to distinguish between explicit zero and not specified.
    // +optional
    FailedJobsHistoryLimit *int32 `json:"failedJobsHistoryLimit,omitempty"`
}

We define a custom type to hold our concurrency policy. It’s actually just a string under the hood, but the type gives extra documentation, and allows us to attach validation on the type instead of the field, making the validation more easily reusable.



// ConcurrencyPolicy describes how the job will be handled.
// Only one of the following concurrent policies may be specified.
// If none of the following policies is specified, the default one
// is AllowConcurrent.
// +kubebuilder:validation:Enum=Allow;Forbid;Replace
type ConcurrencyPolicy string

const (
    // AllowConcurrent allows CronJobs to run concurrently.
    AllowConcurrent ConcurrencyPolicy = "Allow"

    // ForbidConcurrent forbids concurrent runs, skipping next run if previous
    // hasn't finished yet.
    ForbidConcurrent ConcurrencyPolicy = "Forbid"

    // ReplaceConcurrent cancels currently running job and replaces it with a new one.
    ReplaceConcurrent ConcurrencyPolicy = "Replace"
)

Next, let’s design our status, which holds observed state. It contains any information we want users or other controllers to be able to easily obtain.

We’ll keep a list of actively running jobs, as well as the last time that we succesfully ran our job. Notice that we use metav1.Time instead of time.Time to get the stable serialization, as mentioned above.



// CronJobStatus defines the observed state of CronJob
type CronJobStatus struct {
    // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
    // Important: Run "make" to regenerate code after modifying this file

    // A list of pointers to currently running jobs.
    // +optional
    Active []corev1.ObjectReference `json:"active,omitempty"`

    // Information when was the last time the job was successfully scheduled.
    // +optional
    LastScheduleTime *metav1.Time `json:"lastScheduleTime,omitempty"`
}

Finally, we have the rest of the boilerplate that we’ve already discussed. As previously noted, we don’t need to change this, except to mark that we want a status subresource, so that we behave like built-in kubernetes types.



// +kubebuilder:object:root=true
// +kubebuilder:subresource:status

// CronJob is the Schema for the cronjobs API
type CronJob struct {

Root Object Definitions


    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   CronJobSpec   `json:"spec,omitempty"`
    Status CronJobStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true

// CronJobList contains a list of CronJob
type CronJobList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []CronJob `json:"items"`
}

func init() {
    SchemeBuilder.Register(&CronJob{}, &CronJobList{})
}

Now that we have an API, we’ll need to write a controller to actually implement the functionality.

A Brief Aside: What’s the rest of this stuff?

If you’ve taken a peek at the rest of the files in the api/v1/ directory, you might have noticed two additional files beyond cronjob_types.go: groupversion_info.go and zz_generated.deepcopy.go.

Neither of these files ever needs to be edited (the former stays the same and the latter is autogenerated), but it’s useful to know what’s in them.

groupversion_info.go

groupversion_info.go contains common metadata about the group-version:

Apache License

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

First, we have some package-level markers that denote that there are Kubernetes objects in this package, and that this package represents the group batch.tutorial.kubebuilder.io. The object generator makes use of the former, while the latter is used by the CRD generator to generate the right metadata for the CRDs it creates from this package.



// Package v1 contains API Schema definitions for the batch v1 API group
// +kubebuilder:object:generate=true
// +groupName=batch.tutorial.kubebuilder.io
package v1

import (
    "k8s.io/apimachinery/pkg/runtime/schema"
    "sigs.k8s.io/controller-runtime/pkg/scheme"
)

Then, we have the commonly useful variables that help us set up our Scheme. Since we need to use all the types in this package in our controller, it’s helpful (and the convention) to have a convenient method to add all the types to some other Scheme. SchemeBuilder makes this easy for us.


var (
    // GroupVersion is group version used to register these objects
    GroupVersion = schema.GroupVersion{Group: "batch.tutorial.kubebuilder.io", Version: "v1"}

    // SchemeBuilder is used to add go types to the GroupVersionKind scheme
    SchemeBuilder = &scheme.Builder{GroupVersion: GroupVersion}

    // AddToScheme adds the types in this group-version to the given scheme.
    AddToScheme = SchemeBuilder.AddToScheme
)

zz_generated.deepcopy.go

zz_generated.deepcopy.go contains the autogenerated implementation of the aforementioned runtime.Object interface, which marks all of our root types as representing Kinds.

The core of the runtime.Object interface is a deep-copy method, DeepCopyObject.

The object generator in controller-tools also generates two other handy methods for each root type and all its sub-types: DeepCopy and DeepCopyInto.

What’s in a controller?

Controllers are the core of Kubernetes, and of any operator.

It’s a controller’s job to ensure that, for any given object, the actual state of the world (both the cluster state, and potentially external state like running containers for Kubelet or loadbalancers for a cloud provider) matches the desired state in the object. Each controller focuses on one root Kind, but may interact with other Kinds.

We call this process reconciling.

In controller-runtime, the logic that implements the reconciling for a specific kind is called a Reconciler. A reconciler takes the name of an object, and returns whether or not we need to try again (e.g. in case of errors or periodic controllers, like the HorizontalPodAutoscaler).

Apache License

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

First, we start out with some standard imports. As before, we need the core controller-runtime library, as well as the client package, and the package for our API types.


package controllers

import (
    "context"

    "github.com/go-logr/logr"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"

    batchv1 "tutorial.kubebuilder.io/project/api/v1"
)

Next, kubebuilder has scaffold out a basic reconciler struct for us. Pretty much every reconciler needs to log, and needs to be able to fetch objects, so these are added out of the box.



// CronJobReconciler reconciles a CronJob object
type CronJobReconciler struct {
    client.Client
    Log logr.Logger
}

Most controllers eventually end up running on the cluster, so they need RBAC permissions. These are the bare minimum permissions needed to run. As we add more functionality, we’ll need to revisit these.



// +kubebuilder:rbac:groups=batch.tutorial.kubebuilder.io,resources=cronjobs,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=batch.tutorial.kubebuilder.io,resources=cronjobs/status,verbs=get;update;patch

Reconcile actually performs the reconciling for a single named object. Our Request just has a name, but we can use the client to fetch that object from the cache.

We return an empty result and no error, which indicates to controller-runtime that we’ve succesfully reconciled this object and don’t need to try again until there’s some changes.

Most controllers need a logging handle and a context, so we set them up here.

The context is used to allow cancelation of requests, and potentially things like tracing. It’s the first argument to all client methods. The Background context is just a basic context without any extra data or timing restrictions.

The logging handle lets us log. controller-runtime uses structured logging through a library called logr. As we’ll see shortly, logging works by attaching key-value pairs to a static message. We can pre-assign some pairs at the top of our reconcile method to have those attached to all log lines in this reconciler.


func (r *CronJobReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    _ = context.Background()
    _ = r.Log.WithValues("cronjob", req.NamespacedName)

    // your logic here

    return ctrl.Result{}, nil
}

Finally, we add this reconciler to the manager, so that it gets started when the manager is started.

For now, we just note that this reconciler operates on CronJobs. Later, we’ll use this to mark that we care about related objects as well.

TODO: jump back to main?



func (r *CronJobReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&batchv1.CronJob{}).
        Complete(r)
}

Now that we’ve seen the basic structure of a reconciler, let’s fill out the logic for CronJobs.

Implementing a controller

The basic logic of our CronJob controller is this:

  1. Load the named CronJob

  2. List all active jobs, and update the status

  3. Clean up old jobs according to the history limits

  4. Check if we’re suspended (and don’t do anything else if we are)

  5. Get the next scheduled run

  6. Run a new job if it’s on schedule, not past the deadline, and not blocked by our concurrency policy

  7. Requeue when we either see a running job (done automatically) or it’s time for the next scheduled run.

Apache License

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

We’ll start out with some imports. You’ll see below that we’ll need a few more imports than those scaffolded for us. We’ll talk about each one when we use it.


package controllers

import (
    "context"
    "fmt"
    "sort"
    "time"

    "github.com/go-logr/logr"
    "github.com/robfig/cron"
    kbatch "k8s.io/api/batch/v1"
    corev1 "k8s.io/api/core/v1"
    apierrs "k8s.io/apimachinery/pkg/api/errors"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/runtime"
    ref "k8s.io/client-go/tools/reference"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"

    batch "tutorial.kubebuilder.io/project/api/v1"
)

Next, we’ll need a few more bits in our Reconciler:

We’ll need the Scheme, in order to call some helpers that set owner references, and we’ll need a Clock, which will allow us to fake timing in our tests.



// CronJobReconciler reconciles a CronJob object
type CronJobReconciler struct {
    client.Client
    Log    logr.Logger
    Scheme *runtime.Scheme
    Clock
}

Clock We’ll mock out the clock to make it easier to jump around in time while testing, the “real” clock just calls time.Now.


type realClock struct{}

func (_ realClock) Now() time.Time { return time.Now() }

// clock knows how to get the current time.
// It can be used to fake out timing for testing.
type Clock interface {
    Now() time.Time
}

ignoreNotFound We generally want to ignore (not requeue) NotFound errors, since we’ll get a reconciliation request once the object exists, and requeuing in the meantime won’t help.


func ignoreNotFound(err error) error {
    if apierrs.IsNotFound(err) {
        return nil
    }
    return err
}

Notice that we need a few more RBAC permissions -- since we’re creating and managing jobs now, we’ll need permissions for those.



// +kubebuilder:rbac:groups=batch.tutorial.kubebuilder.io,resources=cronjobs,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=batch.tutorial.kubebuilder.io,resources=cronjobs/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=batch,resources=jobs,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=batch,resources=jobs/status,verbs=get

Now, we get to the heart of the controller -- the reconciler logic.


var (
    scheduledTimeAnnotation = "batch.tutorial.kubebuilder.io/scheduled-at"
)

func (r *CronJobReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    ctx := context.Background()
    log := r.Log.WithValues("cronjob", req.NamespacedName)


1: Load the CronJob by name

We’ll fetch the CronJob using our client. All client methods take a context (to allow for cancellation) as their first argument, and the object in question as their last. Get is a bit special, in that it takes a NamespacedName as the middle argument (most don’t have a middle argument, as we’ll see below).

Many client methods also take variadic options at the end.


    var cronJob batch.CronJob
    if err := r.Get(ctx, req.NamespacedName, &cronJob); err != nil {
        log.Error(err, "unable to fetch CronJob")
        // we'll ignore not-found errors, since they can't be fixed by an immediate
        // requeue (we'll need to wait for a new notification), and we can get them
        // on deleted requests.
        return ctrl.Result{}, ignoreNotFound(err)
    }


2: List all active jobs, and update the status

To fully update our status, we’ll need to list all child jobs in this namespace that belong to this CronJob. Similarly to Get, we can use the List method to list the child jobs. Notice that we use variadic options to set the namespace and field match (which is actually an index lookup that we set up below).


    var childJobs kbatch.JobList
    if err := r.List(ctx, &childJobs, client.InNamespace(req.Namespace), client.MatchingField(jobOwnerKey, req.Name)); err != nil {
        log.Error(err, "unable to list child Jobs")
        return ctrl.Result{}, err
    }


Once we have all the jobs we own, we’ll split them into active, successful, and failed jobs, keeping track of the most recent run so that we can record it in status. Remember, status should be able to be reconstituted from the state of the world, so it’s generally not a good idea to read from the status of the root object. Instead, you should reconstruct it every run. That’s what we’ll do here.

We can check if a job is “finished” and whether it succeeded or failed using status conditions. We’ll put that logic in a helper to make our code cleaner.



    // find the active list of jobs
    var activeJobs []*kbatch.Job
    var successfulJobs []*kbatch.Job
    var failedJobs []*kbatch.Job
    var mostRecentTime *time.Time // find the last run so we can update the status


isJobFinished We consider a job “finished” if it has a “succeeded” or “failed” condition marked as true. Status conditions allow us to add extensible status information to our objects that other humans and controllers can examine to check things like completion and health.


    isJobFinished := func(job *kbatch.Job) (bool, kbatch.JobConditionType) {
        for _, c := range job.Status.Conditions {
            if (c.Type == kbatch.JobComplete || c.Type == kbatch.JobFailed) && c.Status == corev1.ConditionTrue {
                return true, c.Type
            }
        }

        return false, ""
    }

getScheduledTimeForJob We’ll use a helper to extract the scheduled time from the annotation that we added during job creation.


    getScheduledTimeForJob := func(job *kbatch.Job) (*time.Time, error) {
        timeRaw := job.Annotations[scheduledTimeAnnotation]
        if len(timeRaw) == 0 {
            return nil, nil
        }

        timeParsed, err := time.Parse(time.RFC3339, timeRaw)
        if err != nil {
            return nil, err
        }
        return &timeParsed, nil
    }



    for i, job := range childJobs.Items {
        _, finishedType := isJobFinished(&job)
        switch finishedType {
        case "": // ongoing
            activeJobs = append(activeJobs, &childJobs.Items[i])
        case kbatch.JobFailed:
            failedJobs = append(failedJobs, &childJobs.Items[i])
        case kbatch.JobComplete:
            successfulJobs = append(successfulJobs, &childJobs.Items[i])
        }

        // We'll store the launch time in an annotation, so we'll reconsitute that from
        // the active jobs themselves.
        scheduledTimeForJob, err := getScheduledTimeForJob(&job)
        if err != nil {
            log.Error(err, "unable to parse schedule time for child job", "job", &job)
            continue
        }
        if scheduledTimeForJob != nil {
            if mostRecentTime == nil {
                mostRecentTime = scheduledTimeForJob
            } else if mostRecentTime.Before(*scheduledTimeForJob) {
                mostRecentTime = scheduledTimeForJob
            }
        }
    }

    if mostRecentTime != nil {
        cronJob.Status.LastScheduleTime = &metav1.Time{Time: *mostRecentTime}
    } else {
        cronJob.Status.LastScheduleTime = nil
    }
    cronJob.Status.Active = nil
    for _, activeJob := range activeJobs {
        jobRef, err := ref.GetReference(r.Scheme, activeJob)
        if err != nil {
            log.Error(err, "unable to make reference to active job", "job", activeJob)
            continue
        }
        cronJob.Status.Active = append(cronJob.Status.Active, *jobRef)
    }


Here, we’ll log how many jobs we observed at a slightly higher logging level, for debugging. Notice how instead of using a format string, we use a fixed message, and attach key-value pairs with the extra information. This makes it easier to filter and query log lines.


    log.V(1).Info("job count", "active jobs", len(activeJobs), "successful jobs", len(successfulJobs), "failed jobs", len(failedJobs))


Using the date we’ve gathered, we’ll update the status of our CRD. Just like before, we use our client. To specifically update the status subresource, we’ll use the Status part of the client, with the Update method.

The status subresource ignores changes to spec, so it’s less likely to conflict with any other updates, and can have separate permissions.


    if err := r.Status().Update(ctx, &cronJob); err != nil {
        log.Error(err, "unable to update CronJob status")
        return ctrl.Result{}, err
    }


Once we’ve updated our status, we can move on to ensuring that the status of the world matches what we want in our spec.

3: Clean up old jobs according to the history limit

First, we’ll try to clean up old jobs, so that we don’t leave too many lying around.



    // NB: deleting these is "best effort" -- if we fail on a particular one,
    // we won't requeue just to finish the deleting.
    if cronJob.Spec.FailedJobsHistoryLimit != nil {
        sort.Slice(failedJobs, func(i, j int) bool {
            if failedJobs[i].Status.StartTime == nil {
                return failedJobs[j].Status.StartTime != nil
            }
            return failedJobs[i].Status.StartTime.Before(failedJobs[j].Status.StartTime)
        })
        for i, job := range failedJobs {
            if err := r.Delete(ctx, job); err != nil {
                log.Error(err, "unable to delete old failed job", "job", job)
            }
            if int32(i) >= *cronJob.Spec.FailedJobsHistoryLimit {
                break
            }
        }
    }

    if cronJob.Spec.SuccessfulJobsHistoryLimit != nil {
        sort.Slice(successfulJobs, func(i, j int) bool {
            if successfulJobs[i].Status.StartTime == nil {
                return successfulJobs[j].Status.StartTime != nil
            }
            return successfulJobs[i].Status.StartTime.Before(successfulJobs[j].Status.StartTime)
        })
        for i, job := range successfulJobs {
            if err := r.Delete(ctx, job); err != nil {
                log.Error(err, "unable to delete old successful job", "job", job)
            }
            if int32(i) >= *cronJob.Spec.SuccessfulJobsHistoryLimit {
                break
            }
        }
    }


4: Check if we’re suspended

If this object is suspended, we don’t want to run any jobs, so we’ll stop now. This is useful if something’s broken with the job we’re running and we want to pause runs to investigate or putz with the cluster, without deleting the object.



    if cronJob.Spec.Suspend != nil && *cronJob.Spec.Suspend {
        log.V(1).Info("cronjob suspended, skipping")
        return ctrl.Result{}, nil
    }


5: Get the next scheduled run

If we’re not paused, we’ll need to calculate the next scheduled run, and whether or not we’ve got a run that we haven’t processed yet.

getNextSchedule We’ll calculate the next scheduled time using our helpful cron library. We’ll start calculating appropriate times from our last run, or the creation of the CronJob if we can’t find a last run.

If there are too many missed runs and we don’t have any deadlines set, we’ll bail so that we don’t cause issues on controller restarts or wedges.

Otherwise, we’ll just return the missed runs (of which we’ll just use the latest), and the next run, so that we can know when it’s time to reconcile again.


    getNextSchedule := func(cronJob *batch.CronJob, now time.Time) (lastMissed *time.Time, next time.Time, err error) {
        sched, err := cron.ParseStandard(cronJob.Spec.Schedule)
        if err != nil {
            return nil, time.Time{}, fmt.Errorf("Unparseable schedule %q: %v", cronJob.Spec.Schedule, err)
        }

        // for optimization purposes, cheat a bit and start from our last observed run time
        // we could reconstitute this here, but there's not much point, since we've
        // just updated it.
        var earliestTime time.Time
        if cronJob.Status.LastScheduleTime != nil {
            earliestTime = cronJob.Status.LastScheduleTime.Time
        } else {
            earliestTime = cronJob.ObjectMeta.CreationTimestamp.Time
        }
        if cronJob.Spec.StartingDeadlineSeconds != nil {
            // controller is not going to schedule anything below this point
            schedulingDeadline := now.Add(-time.Second * time.Duration(*cronJob.Spec.StartingDeadlineSeconds))

            if schedulingDeadline.After(earliestTime) {
                earliestTime = schedulingDeadline
            }
        }
        if earliestTime.After(now) {
            return nil, sched.Next(now), nil
        }

        starts := 0
        for t := sched.Next(earliestTime); !t.After(now); t = sched.Next(t) {
            lastMissed = &t
            // An object might miss several starts. For example, if
            // controller gets wedged on Friday at 5:01pm when everyone has
            // gone home, and someone comes in on Tuesday AM and discovers
            // the problem and restarts the controller, then all the hourly
            // jobs, more than 80 of them for one hourly scheduledJob, should
            // all start running with no further intervention (if the scheduledJob
            // allows concurrency and late starts).
            //
            // However, if there is a bug somewhere, or incorrect clock
            // on controller's server or apiservers (for setting creationTimestamp)
            // then there could be so many missed start times (it could be off
            // by decades or more), that it would eat up all the CPU and memory
            // of this controller. In that case, we want to not try to list
            // all the missed start times.
            starts++
            if starts > 100 {
                // We can't get the most recent times so just return an empty slice
                return nil, time.Time{}, fmt.Errorf("Too many missed start times (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew.")
            }
        }
        return lastMissed, sched.Next(now), nil
    }



    // figure out the next times that we need to create
    // jobs at (or anything we missed).
    missedRun, nextRun, err := getNextSchedule(&cronJob, r.Now())
    if err != nil {
        log.Error(err, "unable to figure out CronJob schedule")
        // we don't really care about requeuing until we get an update that
        // fixes the schedule, so don't return an error
        return ctrl.Result{}, nil
    }


We’ll prep our eventual request to requeue until the next job, and then figure out if we actually need to run.


    scheduledResult := ctrl.Result{RequeueAfter: nextRun.Sub(r.Now())} // save this so we can re-use it elsewhere
    log = log.WithValues("now", r.Now(), "next run", nextRun)


6: Run a new job if it’s on schedule, not past the deadline, and not blocked by our concurrency policy

If we’ve missed a run, and we’re still within the deadline to start it, we’ll need to run a job.


    if missedRun == nil {
        log.V(1).Info("no upcoming scheduled times, sleeping until next")
        return scheduledResult, nil
    }

    // make sure we're not too late to start the run
    log = log.WithValues("current run", missedRun)
    tooLate := false
    if cronJob.Spec.StartingDeadlineSeconds != nil {
        tooLate = missedRun.Add(time.Duration(*cronJob.Spec.StartingDeadlineSeconds) * time.Second).Before(r.Now())
    }
    if tooLate {
        log.V(1).Info("missed starting deadline for last run, sleeping till next")
        // TODO(directxman12): events
        return scheduledResult, nil
    }


If we actually have to run a job, we’ll need to either wait till existing ones finish, replace the existing ones, or just add new ones. If our information is out of date due to cache delay, we’ll get a requeue when we get up-to-date information.


    // figure out how to run this job -- concurrency policy might forbid us from running
    // multiple at the same time...
    if cronJob.Spec.ConcurrencyPolicy == batch.ForbidConcurrent && len(activeJobs) > 0 {
        log.V(1).Info("concurrency policy blocks concurrent runs, skipping", "num active", len(activeJobs))
        return scheduledResult, nil
    }

    // ...or instruct us to replace existing ones...
    if cronJob.Spec.ConcurrencyPolicy == batch.ReplaceConcurrent {
        for _, activeJob := range activeJobs {
            // we don't care if the job was already deleted
            if err := r.Delete(ctx, activeJob); ignoreNotFound(err) != nil {
                log.Error(err, "unable to delete active job", "job", activeJob)
                return ctrl.Result{}, err
            }
        }
    }


Once we’ve figured out what to do with existing jobs, we’ll actually create our desired job

constructJobForCronJob We need to construct a job based on our CronJob’s template. We’ll copy over the spec from the template and copy some basic object meta.

Then, we’ll set the “scheduled time” annotation so that we can reconstitute our LastScheduleTime field each reconcile.

Finally, we’ll need to set an owner reference. This allows the Kubernetes garbage collector to clean up jobs when we delete the CronJob, and allows controller-runtime to figure out which cronjob needs to be reconciled when a given job changes (is added, deleted, completes, etc).


    constructJobForCronJob := func(cronJob *batch.CronJob, scheduledTime time.Time) (*kbatch.Job, error) {
        // We want job names for a given nominal start time to have a deterministic name to avoid the same job being created twice
        name := fmt.Sprintf("%s-%d", cronJob.Name, scheduledTime.Unix())

        job := &kbatch.Job{
            ObjectMeta: metav1.ObjectMeta{
                Labels:      make(map[string]string),
                Annotations: make(map[string]string),
                Name:        name,
                Namespace:   cronJob.Namespace,
            },
            Spec: *cronJob.Spec.JobTemplate.Spec.DeepCopy(),
        }
        for k, v := range cronJob.Spec.JobTemplate.Annotations {
            job.Annotations[k] = v
        }
        job.Annotations[scheduledTimeAnnotation] = scheduledTime.Format(time.RFC3339)
        for k, v := range cronJob.Spec.JobTemplate.Labels {
            job.Labels[k] = v
        }
        if err := ctrl.SetControllerReference(cronJob, job, r.Scheme); err != nil {
            return nil, err
        }

        return job, nil
    }



    // actually make the job...
    job, err := constructJobForCronJob(&cronJob, *missedRun)
    if err != nil {
        log.Error(err, "unable to construct job from template")
        // don't bother requeuing until we get a change to the spec
        return scheduledResult, nil
    }

    // ...and create it on the cluster
    if err := r.Create(ctx, job); err != nil {
        log.Error(err, "unable to create Job for CronJob", "job", job)
        return ctrl.Result{}, err
    }

    log.V(1).Info("created Job for CronJob run", "job", job)


7: Requeue when we either see a running job or it’s time for the next scheduled run

Finally, we’ll return the result that we prepped above, that says we want to requeue when our next run would need to occur. This is taken as a maximum deadline -- if something else changes in between, like our job starts or finishes, we get modified, etc, we might reconcile again sooner.


    // we'll requeue once we see the running job, and update our status
    return scheduledResult, nil
}

Setup

Finally, we’ll update our setup. In order to allow our reconciler to quickly look up Jobs by their owner, we’ll need an index. We declare an index key that we can later use with the client as a pseudo-field name, and then describe how to extract the indexed value from the Job object. The indexer will automatically take care of namespaces for us, so we just have to extract the owner name if the Job has a CronJob owner.

Additionally, we’ll inform the manager that this controller owns some Jobs, so that it will automatically call Reconcile on the underlying CronJob when a Job changes, is deleted, etc.


var (
    jobOwnerKey = ".metadata.controller"
    apiGVStr    = batch.GroupVersion.String()
)

func (r *CronJobReconciler) SetupWithManager(mgr ctrl.Manager) error {
    // set up a real clock, since we're not in a test
    if r.Clock == nil {
        r.Clock = realClock{}
    }

    if err := mgr.GetFieldIndexer().IndexField(&kbatch.Job{}, jobOwnerKey, func(rawObj runtime.Object) []string {
        // grab the job object, extract the owner...
        job := rawObj.(*kbatch.Job)
        owner := metav1.GetControllerOf(job)
        if owner == nil {
            return nil
        }
        // ...make sure it's a CronJob...
        if owner.APIVersion != apiGVStr || owner.Kind != "CronJob" {
            return nil
        }

        // ...and if so, return it
        return []string{owner.Name}
    }); err != nil {
        return err
    }

    return ctrl.NewControllerManagedBy(mgr).
        For(&batch.CronJob{}).
        Owns(&kbatch.Job{}).
        Complete(r)
}

That was a doozy, but now we’ve got a working controller. Let’s test against the cluster, then, if we don’t have any issues, deploy it!

You said something about main?

But first, remember how we said we’d come back to main.go again? Let’s take a look and see what’s changed, and what we need to add.

Apache License

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Imports



package main

import (
    "flag"
    "os"

    kbatchv1 "k8s.io/api/batch/v1"
    "k8s.io/apimachinery/pkg/runtime"
    clientgoscheme "k8s.io/client-go/kubernetes/scheme"
    _ "k8s.io/client-go/plugin/pkg/client/auth/gcp"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/log/zap"
    batchv1 "tutorial.kubebuilder.io/project/api/v1"
    "tutorial.kubebuilder.io/project/controllers"
    // +kubebuilder:scaffold:imports
)

The first difference to notice is that kubebuilder has added the new API group’s package (batchv1) to our scheme. This means that we can use those objects in our controller.

We’ll also need to add the kubernetes batch v1 (kbatchv1) scheme, since we’re creating and listing Jobs.


var (
    scheme   = runtime.NewScheme()
    setupLog = ctrl.Log.WithName("setup")
)

func init() {
    _ = clientgoscheme.AddToScheme(scheme)

    _ = kbatchv1.AddToScheme(scheme) // we've added this ourselves
    _ = batchv1.AddToScheme(scheme)
    // +kubebuilder:scaffold:scheme
}

The other thing that’s changed is that kubebuilder has added a block calling our CronJob controller’s SetupWithManager method. Since we now use a Scheme as well, we’ll need to pass that to the reconciler ourselves.



func main() {

old stuff


    var metricsAddr string
    var enableLeaderElection bool
    flag.StringVar(&metricsAddr, "metrics-addr", ":8080", "The address the metric endpoint binds to.")
    flag.BoolVar(&enableLeaderElection, "enable-leader-election", false,
        "Enable leader election for controller manager. Enabling this will ensure there is only one active controller manager.")
    flag.Parse()

    ctrl.SetLogger(zap.Logger(true))

    mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
        Scheme:             scheme,
        MetricsBindAddress: metricsAddr,
        LeaderElection:     enableLeaderElection,
    })
    if err != nil {
        setupLog.Error(err, "unable to start manager")
        os.Exit(1)
    }



    if err = (&controllers.CronJobReconciler{
        Client: mgr.GetClient(),
        Log:    ctrl.Log.WithName("controllers").WithName("Captain"),
        Scheme: mgr.GetScheme(), // we've added this ourselves
    }).SetupWithManager(mgr); err != nil {
        setupLog.Error(err, "unable to create controller", "controller", "Captain")
        os.Exit(1)
    }
    if err = (&batchv1.CronJob{}).SetupWebhookWithManager(mgr); err != nil {
        setupLog.Error(err, "unable to create webhook", "webhook", "Captain")
        os.Exit(1)
    }
    // +kubebuilder:scaffold:builder


old stuff



    setupLog.Info("starting manager")
    if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
        setupLog.Error(err, "problem running manager")
        os.Exit(1)
    }


}

Now we can implement our controller.

Implementing admission webhooks

If you want to implement admission webhooks for your CRD, the only thing you need to do is to implement the Defaulter and (or) the Validator interface.

Kubebuilder takes care of the rest for you, such as

  1. Creating the webhook server.
  2. Ensuring the server has been added in the manager.
  3. Creating handlers for your webhooks.
  4. Registering each handler with a path in your server.

Apache License

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Go imports



package v1

import (
    "github.com/robfig/cron"

    apierrors "k8s.io/apimachinery/pkg/api/errors"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/runtime/schema"
    validationutils "k8s.io/apimachinery/pkg/util/validation"
    "k8s.io/apimachinery/pkg/util/validation/field"

    ctrl "sigs.k8s.io/controller-runtime"
    logf "sigs.k8s.io/controller-runtime/pkg/runtime/log"
    "sigs.k8s.io/controller-runtime/pkg/webhook"
)

Next, we’ll setup a logger for the webhooks.



var cronjoblog = logf.Log.WithName("cronjob-resource")

Then, we set up the webhook with the manager.



func (r *CronJob) SetupWebhookWithManager(mgr ctrl.Manager) error {
    return ctrl.NewWebhookManagedBy(mgr).
        For(r).
        Complete()
}

Notice that we use kubebuilder markers to generate webhook manifests. This markers is responsible for generating a mutating webhook manifest.

The meaning of each marker can be found here.



// +kubebuilder:webhook:path=/mutate-batch-tutorial-kubebuilder-io-v1-cronjob,mutating=true,failurePolicy=fail,groups=batch.tutorial.kubebuilder.io,resources=cronjobs,verbs=create;update,versions=v1,name=mcronjob.kb.io

We use the webhook.Defaulter interface to set defaults to our CRD. A webhook will automatically be served that calls this defaulting.

The Default method is expected to mutate the receiver, setting the defaults.



var _ webhook.Defaulter = &CronJob{}

// Default implements webhook.Defaulter so a webhook will be registered for the type
func (r *CronJob) Default() {
    cronjoblog.Info("default", "name", r.Name)

    if r.Spec.ConcurrencyPolicy == "" {
        r.Spec.ConcurrencyPolicy = AllowConcurrent
    }
    if r.Spec.Suspend == nil {
        r.Spec.Suspend = new(bool)
    }
    if r.Spec.SuccessfulJobsHistoryLimit == nil {
        r.Spec.SuccessfulJobsHistoryLimit = new(int32)
        *r.Spec.SuccessfulJobsHistoryLimit = 3
    }
    if r.Spec.FailedJobsHistoryLimit == nil {
        r.Spec.FailedJobsHistoryLimit = new(int32)
        *r.Spec.FailedJobsHistoryLimit = 1
    }
}

Notice that we use kubebuilder markers to generate webhook manifests. This markers is responsible for generating a validating webhook manifest.

The meaning of each marker can be found here.



// +kubebuilder:webhook:path=/validate-batch-tutorial-kubebuilder-io-v1-cronjob,mutating=false,failurePolicy=fail,groups=batch.tutorial.kubebuilder.io,resources=cronjobs,verbs=create;update,versions=v1,name=vcronjob.kb.io

To validate our CRD beyond what’s possible with declarative validation. Generally, declarative validation should be sufficient, but sometimes more advanced use cases call for complex validation.

For instance, we’ll see below that we use this to validate a well-formed cron schedule without making up a long regular expression.

If webhook.Validator interface is implemented, a webhook will automatically be served that calls the validation.

The ValidateCreate and ValidateUpdate methods are expected to validate that its receiver upon creation and update respectively. We separate out ValidateCreate from ValidateUpdate to allow behavior like making certain fields immutable, so that they can only be set on creation. Here, however, we just use the same shared validation.



var _ webhook.Validator = &CronJob{}

// ValidateCreate implements webhook.Validator so a webhook will be registered for the type
func (r *CronJob) ValidateCreate() error {
    cronjoblog.Info("validate create", "name", r.Name)

    return r.validateCronJob()
}

// ValidateUpdate implements webhook.Validator so a webhook will be registered for the type
func (r *CronJob) ValidateUpdate(old runtime.Object) error {
    cronjoblog.Info("validate update", "name", r.Name)

    return r.validateCronJob()
}

We validate the name and the spec of the CronJob.



func (r *CronJob) validateCronJob() error {
    var allErrs field.ErrorList
    if err := r.validateCronJobName(); err != nil {
        allErrs = append(allErrs, err)
    }
    if err := r.validateCronJobSpec(); err != nil {
        allErrs = append(allErrs, err)
    }
    if len(allErrs) == 0 {
        return nil
    }

    return apierrors.NewInvalid(
        schema.GroupKind{Group: "batch.tutorial.kubebuilder.io", Kind: "CronJob"},
        r.Name, allErrs)
}

Some fields are declaratively validated by OpenAPI schema. You can find kubebuilder validation markers (prefixed with // +kubebuilder:validation) in the API You can find all of the kubebuilder supported markers for declaring validation by running controller-gen crd -w, or here.



func (r *CronJob) validateCronJobSpec() *field.Error {
    // The field helpers from the kubernetes API machinery help us return nicely
    // structured validation errors.
    return validateScheduleFormat(
        r.Spec.Schedule,
        field.NewPath("spec").Child("schedule"))
}

We’ll need to validate the cron schedule is well-formatted.



func validateScheduleFormat(schedule string, fldPath *field.Path) *field.Error {
    if _, err := cron.ParseStandard(schedule); err != nil {
        return field.Invalid(fldPath, schedule, err.Error())
    }
    return nil
}

Validate object name Validating the length of a string field can be done declaratively by the validation schema.

But the ObjectMeta.Name field is defined in a shared package under the apimachinery repo, so we can’t declaratively validate it using the validation schema.



func (r *CronJob) validateCronJobName() *field.Error {
    if len(r.ObjectMeta.Name) > validationutils.DNS1035LabelMaxLength-11 {
        // The job name length is 63 character like all Kubernetes objects
        // (which must fit in a DNS subdomain). The cronjob controller appends
        // a 11-character suffix to the cronjob (`-$TIMESTAMP`) when creating
        // a job. The job name length limit is 63 characters. Therefore cronjob
        // names must have length <= 63-11=52. If we don't validate this here,
        // then job creation will fail later.
        return field.Invalid(field.NewPath("metadata").Child("name"), r.Name, "must be no more than 52 characters")
    }
    return nil
}

Running and deploying the controller

To test out the controller, we can run it locally against the cluster. Before we do so, though, we’ll need to install our CRDs, as per the quick start. This will automatically update the YAML manifests using controller-tools, if needed:

make install

Now that we’ve installed our CRDs, we can run the controller against our cluster. This will use whatever credentials that we connect to the cluster with, so we don’t need to worry about RBAC just yet.

Note that if you have a webhook and want to deploy it locally, you need to ensure the certificates are in the right place.

In a separate terminal, run

make run

You should see logs from the controller about starting up, but it won’t do anything just yet.

At this point, we need a CronJob to test with. Let’s write a sample to config/samples/batch_v1_cronjob.yaml, and use that:

apiVersion: batch.tutorial.kubebuilder.io/v1
kind: CronJob
metadata:
  name: cronjob-sample
spec:
  schedule: "*/1 * * * *"
  startingDeadlineSeconds: 60
  concurrencyPolicy: Allow # explicitly specify, but Allow is also default.
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            args:
            - /bin/sh
            - -c
            - date; echo Hello from the Kubernetes cluster
          restartPolicy: OnFailure

kubectl create -f config/samples/batch_v1_cronjob.yaml

At this point, you should see a flurry of activity. If you watch the changes, you should see your cronjob running, and updating status:

kubectl get cronjob.batch.tutorial.kubebuilder.io -o yaml
kubectl get job

Now that we know it’s working, we can run it in the cluster. Stop the make run invocation, and run

make docker-build docker-push IMG=<some-registry>/controller
make deploy

If we list cronjobs again like we did before, we should see the controller functioning again!

Deploying the cert manager

We suggest using cert manager for provisioning the certificates for the webhook server. Other solutions should also work as long as they put the certificates in the desired location.

You can follow the cert manager document to install it.

Cert manager also has a component called CA injector, which is responsible for injecting the CA bundle into the Mutating|ValidatingWebhookConfiguration.

To accomplish that, you need to use an annotation with key certmanager.k8s.io/inject-ca-from in the Mutating|ValidatingWebhookConfiguration objects. The value of the annotation should point to an existing certificate CR instance in the format of <certificate-namespace>/<certificate-name>.

This is the kustomize patch we used for annotating the Mutating|ValidatingWebhookConfiguration objects.

# This patch add annotation to admission webhook config and
# the variables $(CERTIFICATE_NAMESPACE) and $(CERTIFICATE_NAME) will be substituted by kustomize.
apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingWebhookConfiguration
metadata:
  name: mutating-webhook-configuration
  annotations:
    certmanager.k8s.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
---
apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: validating-webhook-configuration
  annotations:
    certmanager.k8s.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)

Deploying Admission Webhooks

Kind Cluster

It is recommended to develop your webhook with a kind cluster for faster iteration. Why?

  • You can bring up a multi-node cluster locally within 1 minute.
  • You can tear it down in seconds.
  • You don’t need to push your images to remote registry.

Cert Manager

You need follow this to install the cert manager bundle.

Build your image

Run the following command to build your image locally.

make docker-build

You don’t need to push the image to a remote container registry if you are using a kind cluster. You can directly load your local image to your kind cluster:

kind load docker-image your-image-namge:your-tag

Deploy Webhooks

You need to enable the webhook and cert manager configuration through kustomize. config/default/kustomization.yaml should now look like the following:

# Adds namespace to all resources.
namespace: project-system

# Value of this field is prepended to the
# names of all resources, e.g. a deployment named
# "wordpress" becomes "alices-wordpress".
# Note that it should also match with the prefix (text before '-') of the namespace
# field above.
namePrefix: project-

# Labels to add to all resources and selectors.
#commonLabels:
#  someName: someValue

bases:
- ../crd
- ../rbac
- ../manager
# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in crd/kustomization.yaml
- ../webhook
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'. 'WEBHOOK' components are required.
- ../certmanager

patches:
- manager_image_patch.yaml
  # Protect the /metrics endpoint by putting it behind auth.
  # Only one of manager_auth_proxy_patch.yaml and
  # manager_prometheus_metrics_patch.yaml should be enabled.
- manager_auth_proxy_patch.yaml
  # If you want your controller-manager to expose the /metrics
  # endpoint w/o any authn/z, uncomment the following line and
  # comment manager_auth_proxy_patch.yaml.
  # Only one of manager_auth_proxy_patch.yaml and
  # manager_prometheus_metrics_patch.yaml should be enabled.
#- manager_prometheus_metrics_patch.yaml

# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in crd/kustomization.yaml
- manager_webhook_patch.yaml

# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'.
# Uncomment 'CERTMANAGER' sections in crd/kustomization.yaml to enable the CA injection in the admission webhooks.
# 'CERTMANAGER' needs to be enabled to use ca injection
- webhookcainjection_patch.yaml

# the following config is for teaching kustomize how to do var substitution
vars:
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER' prefix.
- name: CERTIFICATE_NAMESPACE # namespace of the certificate CR
  objref:
    kind: Certificate
    group: certmanager.k8s.io
    version: v1alpha1
    name: serving-cert # this name should match the one in certificate.yaml
  fieldref:
    fieldpath: metadata.namespace
- name: CERTIFICATE_NAME
  objref:
    kind: Certificate
    group: certmanager.k8s.io
    version: v1alpha1
    name: serving-cert # this name should match the one in certificate.yaml
- name: SERVICE_NAMESPACE # namespace of the service
  objref:
    kind: Service
    version: v1
    name: webhook-service
  fieldref:
    fieldpath: metadata.namespace
- name: SERVICE_NAME
  objref:
    kind: Service
    version: v1
    name: webhook-service

Now you can deploy it to your cluster by

make deploy

Wait a while til the webhook pod comes up and the certificates are provisioned. It usually completes within 1 minute.

Now you can create a valid CronJob to test your webhooks. The creation should successfully go through.

kubectl create -f config/samples/batch_v1_cronjob.yaml

You can also try to create an invalid CronJob (e.g. use an ill-formatted schedule field). You should see a creation failure with a validation error.

Note: If you are

Epilogue

Things left to do:

  • Write custom printer columns
  • Discuss webhooks
  • Use different watches

Implement multi-version API

This tutorial will teach you how to build an API with multiple versions. We have divided the tutorial in three sections:

Concepts

Let’s take a quick look at some of the basic concepts related to multiple versions in Kubernetes.

Hub/Spoke/Storage Versions

All versions of a Kubernetes object share the same underlying storage. So if you have versions v1, v2 and v3 of a kind, kubernetes will use one of the versions to persist the object of that Kind in Etcd. You can specify the version to be used for storage in the Custom Resource definition for that API.

One can think of storage version as the hub and other versions as spokes to visualize the relationship between storage and other versions (as shown below in the diagram). The important thing to note is that conversion between storage and other version should be lossless (round trippable). As shown in the diagram below, v1 is the storage/hub version and v2 and v3 spoke version. This tutorial uses the terms storage version and hub interchangeably.

conversion-image

So if each spoke version implements conversion functions to convert to/from a hub, then conversions betweek spokes can be derived. In the example shown in the above diagram, a v2 object can be converted to v3 object by first converting v2 to v1 and then converting v2 to v3. And same is true for converting v3 object to v2.

Conversion Webhook

API clients such as kubectl, controllers can request different versions of your API. So when a client requests for a version other than the storage version of your API, Kubernetes API server calls out to an HTTP endpoint to perform the conversion between the requested version and storage version. This HTTP endpoint is called Conversion Webhook and its discovery/connection parameters are configured in the CRD definition. With kubebuilder, you just need to implement conversion functions between different versions and it takes care of the rest of the work associated with running a webhook server, generating and plumbing the conversion webhook handler.

Prerequisites

CRD conversion webhook support was introduced as alpha feature in Kubernetes 1.13 release and has gone beta in Kubernetes 1.15 release. So ensure that you have a Kubernetes cluster that supports CRD conversion feature enabled. Refer to instructions to enable CRD conversion feature in your cluster. Refer to instructions to setup a local cluster with Kind.

What are we building ?

In this tutorial, we will implement a simple Disk API. Disk API has a field called price that represents price per GB. We will go through three iterations to evolve the price field specification.

  • In v1 version of Disk API, price field is string with “AMOUNT CURRENCY” format. Example values could be “10 USD”, “100 USD”.
  • In v2 version of Disk, price field is represented by a structure Price that has amount and currency as separate fields.
  • In v3 version of Disk, we rename the price field to pricePerGB to make it more explicit.

Here are some sample manifests of the three versions representing same Disk object.


apiVersion: infra.kubebuilder.io/v1
kind: Disk
metadata:
  name: disk-sample
spec:
  price: 10 USD <--- price as string
----------------------------------------

apiVersion: infra.kubebuilder.io/v2
kind: Disk
metadata:
  name: disk-sample
spec: 
  price:  <---- price as structured object
    amount: 10
    currency: USD
----------------------------------------

apiVersion: infra.kubebuilder.io/v3
kind: Disk
metadata:
  name: disk-sample
spec:
  pricePerGB: <--- price is renamed to pricePerGB
    amount: 10
    currency: USD

Tutorial

Now that we have covered the basics and the goal, we are all set to begin this tutorial. We will go through the following steps:

  • Project Setup
  • Adding API with versions v1, v2, v3 of Disk API
  • Setting up Webhooks
  • CRD Generation
  • Configuring Kustomization
  • Deploying and testing

Project Setup

Assuming you have created a new directory and cd in to it. Let’s initialize the project.


# Initialize Go module
go mod init infra.kubebuilder.io

# Initilize Kubebuilder project
kubebuilder init --domain kubebuilder.io

Version v1

Let’s create version v1 of our Disk API.


# create v1 version with resource and controller
kubebuilder create api --group infra --kind Disk --version v1
Create Resource [y/n]
y
Create Controller [y/n]
y

Let’s take a look at file api/v1/disk_types.go.

Apache License

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Imports



package v1

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.


// First version of our Spec has one field price of type string. We will evolve
// this field over the next two versions.

// DiskSpec defines the desired state of Disk
type DiskSpec struct {
    // Price represents price per GB for a Disk. It is specified in the
    // the format "<AMOUNT> <CURRENCY>". Example values will be "10 USD", "100 USD"
    Price string `json:"price"`
}

We need to specify the version that is being used as storage version. In this case, we decided to use v1 version for storage, so we use crd marker +kubebuilder:storageversion on v1.Disk type to indicate that.



// +kubebuilder:object:root=true

// +kubebuilder:storageversion
// Disk is the Schema for the disks API
type Disk struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   DiskSpec   `json:"spec,omitempty"`
    Status DiskStatus `json:"status,omitempty"`
}

We need to define a Hub type to faciliate conversion. Storage and hub version don’t have to be same, but to keep things simple, we will specify v1 to be the Hub for the Disk kind. A version needs to implement conversion.Hub interface to indicate that it is a Hub type. Given that v1 is a Hub version, it doesn’t need to implement any conversion functions.

Next we define Hub() method to indicate that v1 is the hub type


// implements conversion.Hub interface.
func (disk *Disk) Hub() {}

// DiskStatus defines the observed state of Disk
type DiskStatus struct {
    // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
    // Important: Run "make" to regenerate code after modifying this file
}

// +kubebuilder:object:root=true

// DiskList contains a list of Disk
type DiskList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []Disk `json:"items"`
}

func init() {
    SchemeBuilder.Register(&Disk{}, &DiskList{})
}

Version v2

Let’s add version v2 to the Disk API. We will not add any controller this time because we already have a controller for Disk API.

# create v2 version without controller
kubebuilder create api --group infra --kind Disk --version v2
Create Resource [y/n]
y
Create Controller [y/n]
n

Now, let’s take a look at file api/v2/disk_types.go.

Imports

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.



package v2

import (
    "fmt"
    "strconv"
    "strings"

    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "sigs.k8s.io/controller-runtime/pkg/conversion"

    diskapiv1 "infra.kubebuilder.io/api/v1"
)

// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.

We realized that the price field can be represented in a better way, so in next iteration v2 of our disk API, we changed the price to a structured field.



// DiskSpec defines the desired state of Disk
type DiskSpec struct {
    // Price represents price per GB for the Disk.
    Price Price `json:"price"`
}

// Price represents a generic price value that has amount and currency.
type Price struct {
    Amount   int64  `json:"amount"`
    Currency string `json:"currency"`
}

Type definitions



// DiskStatus defines the observed state of Disk
type DiskStatus struct {
    // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
    // Important: Run "make" to regenerate code after modifying this file
}

// +kubebuilder:object:root=true

// Disk is the Schema for the disks API
type Disk struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   DiskSpec   `json:"spec,omitempty"`
    Status DiskStatus `json:"status,omitempty"`
}

The new structure of price field can be converted to and from the price string field of the v1 API. Since v2 is a spoke version, v2 type is required to implement conversion.Convertible interface. Now, let’s take a look at the conversion methods.



// ConvertTo converts receiver (v2.Disk instance in this case) to provided Hub
// instance (v1.Disk in our case).
func (disk *Disk) ConvertTo(dst conversion.Hub) error {
    switch t := dst.(type) {
    case *diskapiv1.Disk:
        diskv1 := dst.(*diskapiv1.Disk)
        diskv1.ObjectMeta = disk.ObjectMeta
        // conversion implementation goes here
        // in our case, we convert the price in structured form to string form.
        diskv1.Spec.Price = fmt.Sprintf("%d %s",
            disk.Spec.Price.Amount, disk.Spec.Price.Currency)
        return nil
    default:
        return fmt.Errorf("unsupported type %v", t)
    }
}

// ConvertFrom converts provided Hub instance (v1.Disk in our case) to receiver
// (v2.Disk in our case).
func (disk *Disk) ConvertFrom(src conversion.Hub) error {
    switch t := src.(type) {
    case *diskapiv1.Disk:
        diskv1 := src.(*diskapiv1.Disk)
        disk.ObjectMeta = diskv1.ObjectMeta
        // conversion implementation goes here
        // We parse price amount and currency from the string form and
        // convert it in structured form.
        parts := strings.Fields(diskv1.Spec.Price)
        if len(parts) != 2 {
            return fmt.Errorf("invalid price")
        }
        amount, err := strconv.Atoi(parts[0])
        if err != nil {
            return err
        }
        disk.Spec.Price = Price{
            Amount:   int64(amount),
            Currency: parts[1],
        }
        return nil
    default:
        return fmt.Errorf("unsupported type %v", t)
    }
}

List definition



// +kubebuilder:object:root=true

// DiskList contains a list of Disk
type DiskList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []Disk `json:"items"`
}

func init() {
    SchemeBuilder.Register(&Disk{}, &DiskList{})
}

Version v3

Let’s add version v3 to the Disk API and once again, we will not add any controller since we already have controller for the Disk API.

# create v3 version without controller
kubebuilder create api --group infra --kind Disk --version v3
Create Resource [y/n]
y
Create Controller [y/n]
n

Imports

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.



package v3

import (
    "fmt"
    "strconv"
    "strings"

    diskapiv1 "infra.kubebuilder.io/api/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "sigs.k8s.io/controller-runtime/pkg/conversion"
)

// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.

In v2 iteration of our API, we evolved price field to a better structure. In v3 iteration we decided to rename price field to PricePerGB to make it more explicit since the price field represents price/per/GB.



// DiskSpec defines the desired state of Disk
type DiskSpec struct {
    // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster

    // PricePerGB represents price for the disk.
    PricePerGB Price `json:"pricePerGB"`
}

// Price represents a generic price value that has amount and currency.
type Price struct {
    Amount   int64  `json:"amount"`
    Currency string `json:"currency"`
}

Type definitions



// DiskStatus defines the observed state of Disk
type DiskStatus struct {
    // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
    // Important: Run "make" to regenerate code after modifying this file
}

// +kubebuilder:object:root=true

// Disk is the Schema for the disks API
type Disk struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   DiskSpec   `json:"spec,omitempty"`
    Status DiskStatus `json:"status,omitempty"`
}

The v3 version has two changes in price field

  • New structure of price field (same as v2 version)
  • Renaming of the field to PricePerGB

Since v3 is a spoke version, v3 type is required to implement conversion.Convertible interface. Now, let’s take a look at the conversion methods.



// ConvertTo converts receiver (v3.Disk instance in this case) to provided Hub
// instance (v1.Disk in our case).
func (disk *Disk) ConvertTo(dst conversion.Hub) error {
    switch t := dst.(type) {
    case *diskapiv1.Disk:
        diskv1 := dst.(*diskapiv1.Disk)
        diskv1.ObjectMeta = disk.ObjectMeta
        // conversion implementation
        // in our case, we convert the price in structured form to string form.
        // Note that we use the value from PricePerGB field.
        diskv1.Spec.Price = fmt.Sprintf("%d %s",
            disk.Spec.PricePerGB.Amount, disk.Spec.PricePerGB.Currency)
        return nil
    default:
        return fmt.Errorf("unsupported type %v", t)
    }
}

// ConvertFrom converts provided Hub instance (v1.Disk in our case) to receiver
// (v3.Disk in our case).
func (disk *Disk) ConvertFrom(src conversion.Hub) error {
    switch t := src.(type) {
    case *diskapiv1.Disk:
        diskv1 := src.(*diskapiv1.Disk)
        disk.ObjectMeta = diskv1.ObjectMeta
        // conversion implementation
        // Note that the conversion logic is same as we implement for v2 except
        // that we use PricePerGB instead of Price.
        parts := strings.Fields(diskv1.Spec.Price)
        if len(parts) != 2 {
            return fmt.Errorf("invalid price")
        }
        amount, err := strconv.Atoi(parts[0])
        if err != nil {
            return err
        }
        disk.Spec.PricePerGB = Price{
            Amount:   int64(amount),
            Currency: parts[1],
        }
        return nil
    default:
        return fmt.Errorf("unsupported type %v", t)
    }
}

List definition



// +kubebuilder:object:root=true

// DiskList contains a list of Disk
type DiskList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []Disk `json:"items"`
}

func init() {
    SchemeBuilder.Register(&Disk{}, &DiskList{})
}

Now that we have all the API implementations in place, let’s take a look at what is required to setup conversion webhook for our Disk API.

Setting up Webhooks

In 2.0.0+ release, Kubebuilder introduced new command create webhook to make it easy to setup admission and conversion webhooks. Run the following command to setup conversion webhook. Note that we can specify any version from v1, v2 or v3 in this command because there is single conversion webhook for a Kind.

kubebuilder create webhook --group infra --kind Disk --version v1 --conversion

Writing scaffold for you to edit...
api/v1/disk_webhook.go

Above commands does the following:

  • Scaffolds a new file api/v1/disk_webhook.go to implement webhook setup method.
  • Updates main.go to setup webhooks with the manager instance.

Let’s take a quick look at the api/v1/disk_webhook.go file.

Imports

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.



package v1

import (
    ctrl "sigs.k8s.io/controller-runtime"
    logf "sigs.k8s.io/controller-runtime/pkg/runtime/log"
)



// log is for logging in this package.
var disklog = logf.Log.WithName("disk-resource")

// SetupWebhookWithManager ensures that webhooks such as Admission or Conversion
// for the Disk API are registerd with the manager.
func (r *Disk) SetupWebhookWithManager(mgr ctrl.Manager) error {
    return ctrl.NewWebhookManagedBy(mgr).
        For(r).
        Complete()
}

If you look at main.go, you will notice the following snippet that invokes the SetupWebhook method.

    .....

    if err = (&infrav1.Disk{}).SetupWebhookWithManager(mgr); err != nil {
        setupLog.Error(err, "unable to create webhook", "webhook", "Disk")
        os.Exit(1)
    }

    ....

CRD Generation

The controller-gen tool that generates the CRD manifest takes a parameter to indicate if our API has multiple versions. We need to specify trivialVersions=false in CRD_OPTIONS in your project’s Makefile to enable multi-version.

...
CRD_OPTIONS ?= "crd:trivialVersions=false"
...

Run make manifests to ensure that CRD manifests gets generated under config/crd/bases/ directory. TODO embed a compressed form of the generated CRD testdata/project/config/crd

Manifests Generation

Kubebuilder generates Kubernetes manifests under ‘config’ directory with webhook bits disabled. Follow the steps below to enable conversion webhook in manifests generation.

  • Ensure that patches/webhook_in_<kind>.yaml and patches/cainjection_in_<kind>.yaml are enabled in config/crds/kustomization.yaml file.
  • Ensure that ../certmanager and ../webhook directories are enabled under bases section in config/default/kustomization.yaml file.
  • Ensure that manager_webhook_patch.yaml is enabled under patches section in config/default/kustomization.yaml file.
  • Enable all the vars under section CERTMANAGER in config/default/kustomization.yaml file.

Deployment

Now we have all our code changes and manifests in place, so let’s deploy it to the cluster and test it out.

Ensure that you have installed cert-manager 0.9.0+ version in your cluster. We have tested the instructions in this tutorial with 0.9.0-alpha.0 release.

Running make deploy will deploy the controller-manager in the cluster.

Testing

Now that we have deployed the controller-manager with conversion webhook enabled, let’s test out the version conversion feature. We will do the following to perform a simple version conversion test:

  • Create disk object named disk-sample using v1 specification
  • Get disk object disk-sample using v2 version
  • Get disk object disk-sample using v3 version

1. Create v1 disk object

apiVersion: infra.kubebuilder.io/v1
kind: Disk
metadata:
  name: disk-sample
spec:
  price: 10 USD

kubectl apply -f config/samples/infra_v1_disk.yaml

2. Get disk object using v2 version

kubectl get disks.v2.infra.kubebuilder.io/disk-sample -o yaml
apiVersion: infra.kubebuilder.io/v2 <-- note the v2 version
kind: Disk
metadata:
  name: disk-sample
  selfLink: /apis/infra.kubebuilder.io/v2/namespaces/default/disks/disk-sample
  uid: 0e9be0fd-a284-11e9-bbbe-42010a8001af
spec:
  price: <-- note the structured price object
    amount: 10
    currency: USD
status: {}

3. Get disk object using v3 version

kubectl get disks.v3.infra.kubebuilder.io/disk-sample -o yaml
apiVersion: infra.kubebuilder.io/v3 <-- note the v3 version
kind: Disk
metadata:
  name: disk-sample
  selfLink: /apis/infra.kubebuilder.io/v3/namespaces/default/disks/disk-sample
  uid: 0e9be0fd-a284-11e9-bbbe-42010a8001af <-- note the same uid as v2
  ....
spec:
  pricePerGB: <-- note the pricePerGB name of the field
    amount: 10
    currency: USD
status: {}

Troubleshooting

TODO(../TODO.md) steps for troubleshoting

Reference

  • Using Finalizers: Finalizers are a mechanism to execute any custom logic related to a resource before it gets deleted from Kubernetes cluster.

  • Webhooks: Webhooks are HTTP callbacks, there are 3 types of webhooks in k8s: 1) admission webhook 2) CRD conversion webhook 3) authorization webhook

    • Admission Webhook: Admission webhooks are HTTP callbacks for mutating or validating resources before the API server admit them.

Generating CRDs

Kubebuilder provides a tool named controller-gen to generate manifests for CustomResourceDefinitions. The tool resides in the controller-tools repository and is installed through a Makefile target called controller-gen.

If you examine the Makefile in your project, you will see a target named manifests for generating manifests. manifests target is also listed as prerequisite for other targets like run, tests, deploy etc to ensure CRD manifests are regenerated when needed.

# Generate manifests e.g. CRD, RBAC etc.
manifests: controller-gen
    $(CONTROLLER_GEN) $(CRD_OPTIONS) rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases

# find or download controller-gen
# download controller-gen if necessary
controller-gen:
ifeq (, $(shell which controller-gen))
    go get sigs.k8s.io/controller-tools/cmd/controller-gen@v0.2.0-beta.2
CONTROLLER_GEN=$(GOBIN)/controller-gen
else
CONTROLLER_GEN=$(shell which controller-gen)
endif

When you run make manifests, you should see generated CRDs are under config/crd/bases directory.

controller-gen generates manifests for RBAC as well, but this section covers the generation of CRD manifests.

controller-gen reads kubebuilder markers of the form // +kubebuilder:something... defined as Go comments in the <your-api-kind>_types.go file under apis/... to produce the CRD manifests. Sections below describe various supported annotations.

Validation

CRDs support validation by definining (OpenAPI v3 schema) in the validation section. To learn more about the validation feature, refer to the original docs here. One can specify validation for a field by annotating the field with kubebuilder marker which is of the form// +kubebuilder:validation:<key=value>. If you want to specify multiple validations for a field, you can add multiple such markers as demonstrated in the example below.

Currently, supporting keys are Maximum, Minimum, MaxLength, MinLength, MaxItems, MinItems, UniqueItems, Enum, Pattern, ExclusiveMaximum, ExclusiveMinimum, MultipleOf, Format. The // +kubebuilder:validation:Pattern=.+:.+ annotation specifies the Pattern validation requiring that the Image field match the regular expression .+:.+

Example:

type ToySpec struct {

    // +kubebuilder:validation:Maximum=100
    // +kubebuilder:validation:Minimum=1
    // +kubebuilder:validation:ExclusiveMinimum=true
    Power  float32 `json:"power,omitempty"`

    Bricks int32   `json:"bricks,omitempty"`
    // +kubebuilder:validation:MaxLength=15
    // +kubebuilder:validation:MinLength=1
    Name string `json:"name,omitempty"`

    // +kubebuilder:validation:MaxItems=500
    // +kubebuilder:validation:MinItems=1
    // +kubebuilder:validation:UniqueItems=false
    Knights []string `json:"knights,omitempty"`

    // +kubebuilder:validation:Enum=Lion;Wolf;Dragon
    Alias string `json:"alias,omitempty"`

    // +kubebuilder:validation:Enum=1;2;3
    Rank    int    `json:"rank"`
}

Additional printer columns

Starting with Kubernetes 1.11, kubectl uses server-side printing. The server decides which columns are shown by the kubectl get command. You can customize these columns using a CustomResourceDefinition. To add an additional column, add a comment with the following marker format just above the struct definition of the Kind.

Format: // +kubebuilder:printcolumn:name="Name",type="type",JSONPath="json-path",description="desc",priority="priority",format="format"

Note that description, priority and format are optional. Refer to the additonal printer columns docs to learn more about the values of name, type, JsonPath, description, priority and format.

The following example adds the Spec, Replicas, and Age columns.

// +kubebuilder:printcolumn:name="Spec",type="integer",JSONPath=".spec.cronSpec",description="status of the kind"
// +kubebuilder:printcolumn:name="Replicas",type="integer",JSONPath=".spec.replicas",description="The number of jobs launched by the CronJob"
// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
type CronTab struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   CronTabSpec   `json:"spec,omitempty"`
    Status CronTabStatus `json:"status,omitempty"`
}

Subresource

Custom resources support /status and /scale subresources as of kubernetes 1.13 release. You can learn more about the subresources here.

1. Status

To enable /status subresource, annotate the kind with // +kubebuilder:subresource:status marker.

2. Scale

To enable /scale subresource, annotate the kind with // +kubebuilder:subresource:scale:specpath=<jsonpath>,statuspath=<jsonpath>,selectorpath=<jsonpath> marker.

Scale subresource marker contains three fields: specpath, statuspath and selectorpath.

  • specpath refers to specReplicasPath attribute of Scale object, and value jsonpath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas. This is a required field.
  • statuspath refers to statusReplicasPath attribute of Scale object. and the jsonpath value of it defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas. This is a required field.
  • selectorpath refers to labelSelectorPath attribute of Scale object, and the value jsonpath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector. This is an optional field.

Example:

type ToySpec struct {
    Replicas *int32 `json:"replicas"` // Add this field in Toy Spec, so the jsonpath to this field is `.spec.replicas`
}

// ToyStatus defines the observed state of Toy
type ToyStatus struct {
    Replicas int32 `json:"replicas"` // Add this field in Toy Status, so the jsonpath to this field is `.status.replicas`
}


// Toy is the Schema for the toys API
// +kubebuilder:subresource:status
// +kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas
type Toy struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   ToySpec   `json:"spec,omitempty"`
    Status ToyStatus `json:"status,omitempty"`
}

In order to enable scale subresource in type definition file, you have to apply the scale subresource right before the kind struct definition, with correct jsonpath values according to the spec and status. And then make sure the jsonpaths are already defined in the Spec and Status struct. Finally, update the <kind>_types_test.go files according to the types Spec and Status changes.

In the above example for the type Toy, we added // +kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas comment before Toy struct definition. .spec.replicas refers to the josnpath of Spec struct field (ToySpec.Replicas). And jsonpath .status.healthyReplicas refers to Status struct field (ToyStatus.Replicas).

Multiple Versions

If you are defining multiple versions of a kind in your project, you need to do the following:

  • Set CRD_OPTIONS ?= "crd:trivialVersions=false" in the Makefile
  • Annotate the Go struct with marker // +kubebuilder:storageversion for the indicating the storage version.

Using Finalizers

Finalizers allow controllers to implement asynchronous pre-delete hooks. Let’s say you create an external resource (such as a storage bucket) for each object of your API type, and you want to delete the associated external resource on object’s deletion from Kubernetes, you can use a finalizer to do that.

You can read more about the finalizers in the Kubernetes reference docs. The section below demonstrates how to register and trigger pre-delete hooks in the Reconcile method of a controller.

The key point to note is that a finalizer causes “delete” on the object to become an “update” to set deletion timestamp. Presence of deletion timestamp on the object indicates that it is being deleted. Otherwise, without finalizers, a delete shows up as a reconcile where the object is missing from the cache.

Highlights:

  • If the object is not being deleted and does not have the finalizer registered, then add the finalizer and update the object in Kubernetes.
  • If object is being deleted and the finalizer is still present in finalizers list, then execute the pre-delete logic and remove the finalizer and update the object.
  • Ensure that the pre-delete logic is idempotent.

Apache License

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Imports First, we start out with some standard imports. As before, we need the core controller-runtime library, as well as the client package, and the package for our API types.


package controllers

import (
    "context"

    "github.com/go-logr/logr"
    ctrl "sigs.k8s.io/controller-runtime"
    "sigs.k8s.io/controller-runtime/pkg/client"

    batchv1 "tutorial.kubebuilder.io/project/api/v1"
)

The code snippet below shows skeleton code for implementing a finalizer.



func (r *CronJobReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
    ctx := context.Background()
    log := r.Log.WithValues("cronjob", req.NamespacedName)

    var cronJob batch.CronJob
    if err := r.Get(ctx, req.NamespacedName, &cronJob); err != nil {
        log.Error(err, "unable to fetch CronJob")
        // we'll ignore not-found errors, since they can't be fixed by an immediate
        // requeue (we'll need to wait for a new notification), and we can get them
        // on deleted requests.
        return ctrl.Result{}, ignoreNotFound(err)
    }

    // name of our custom finalizer
    myFinalizerName := "storage.finalizers.tutorial.kubebuilder.io"

    // examine DeletionTimestamp to determine if object is under deletion
    if cronJob.ObjectMeta.DeletionTimestamp.IsZero() {
        // The object is not being deleted, so if it does not have our finalizer,
        // then lets add the finalizer and update the object. This is equivalent
        // registering our finalizer.
        if !containsString(cronJob.ObjectMeta.Finalizers, myFinalizerName) {
            cronJob.ObjectMeta.Finalizers = append(cronJob.ObjectMeta.Finalizers, myFinalizerName)
            if err := r.Update(context.Background(), cronJob); err != nil {
                return ctrl.Result{}, err
            }
        }
    } else {
        // The object is being deleted
        if containsString(cronJob.ObjectMeta.Finalizers, myFinalizerName) {
            // our finalizer is present, so lets handle any external dependency
            if err := r.deleteExternalResources(cronJob); err != nil {
                // if fail to delete the external dependency here, return with error
                // so that it can be retried
                return ctrl.Result{}, err
            }

            // remove our finalizer from the list and update it.
            cronJob.ObjectMeta.Finalizers = removeString(cronJob.ObjectMeta.Finalizers, myFinalizerName)
            if err := r.Update(context.Background(), cronJob); err != nil {
                return ctrl.Result{}, err
            }
        }

        return ctrl.Result{}, err
    }

    // rest of the reconciler code
}

func (r *Reconciler) deleteExternalResources(cronJob *batch.CronJob) error {
    //
    // delete any external resources associated with the cronJob
    //
    // Ensure that delete implementation is idempotent and safe to invoke
    // multiple types for same object.
}

// Helper functions to check and remove string from a slice of strings.
func containsString(slice []string, s string) bool {
    for _, item := range slice {
        if item == s {
            return true
        }
    }
    return false
}

func removeString(slice []string, s string) (result []string) {
    for _, item := range slice {
        if item == s {
            continue
        }
        result = append(result, item)
    }
    return
}

Kind Cluster

This only cover the basics to use a kind cluster. You can find more details at kind documentation.

Installation

You can follow this to install kind.

Create a Cluster

You can simply create a kind cluster by

kind create cluster

To customize your cluster, you can provide additional configuration. For example, the following is a sample kind configuration.

kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
nodes:
  - role: control-plane
  - role: worker
  - role: worker
  - role: worker

Using the configuration above, run the following command will give you a k8s 1.14.2 cluster with 1 master and 3 workers.

kind create cluster --config hack/kind-config.yaml --image=kindest/node:v1.14.2

You can use --image flag to specify the cluster version you want, e.g. --image=kindest/node:v1.13.6, the supported version are listed here

Cheetsheet

kind load docker-image your-image-name:your-tag
  • Point kubectl to the kind cluster
export KUBECONFIG="$(kind get kubeconfig-path --name="kind")"
  • Delete a kind cluster
kind delete cluster

Webhook

Webhooks are requests for information sent in a blocking fashion. A web application implementing webhooks will send an HTTP request to other application when certain event happens.

In the kubernetes world, there are 3 kinds of webhooks: admission webhook, authorization webhook and CRD conversion webhook.

In controller-runtime libraries, we support admission webhooks and CRD conversion webhooks.

Kubernetes supports these dynamic admission webhooks as of version 1.9 (when the feature entered beta).

Kubernetes supports the conversion webhooks as of version 1.15 (when the feature entered beta).

Admission Webhooks

Admission webhooks are HTTP callbacks that receive admission requests, process them and return admission responses.

Kubernetes provides the following types of admission webhooks:

  • Mutating Admission Webhook: These can mutate the object while it’s being created or updated, before it gets stored. It can be used to default fields in a resource requests, e.g. fields in Deployment that are not specified by the user. It can be used to inject sidecar containers.

  • Validating Admission Webhook: These can validate the object while it’s being created or updated, before it gets stored. It allows more complex validation than pure schema-based validation. e.g. cross-field validation and pod image whitelisting.

The apiserver by default doesn’t authenticate itself to the webhooks. However, if you want to authenticate the clients, you can configure the apiserver to use basic auth, bearer token, or a cert to authenticate itself to the webhooks. You can find detailed steps here.

TODO

If you’re seeing this page, it’s probably because something’s not done in the book yet. Go see if anyone else has found this or bug the maintainers.