In an attempt to learn more about kubernetes controllers & resources, I decided to make a Custom Resource and a kubernetes custom controller to manage it.

Here are some of my notes:

What does a kubernetes controller do?

A controller is a loop, that compares the current state and the desired state of a resource, and attempts to bring the current state closer to the desired state

For example:

A ReplicaSet resource is managed by a replica_set controller, when the number of pods running is less than the desired number of replicas, the replica_set controller will create new pods to match the desired number of replicas

You can find the replica set code here

What this project should do

  • A simple custom resource with kind: Envoy and a controller to handle this custom resource

  • When created, this controller should deploy and manage a fleet of envoy proxy pods

  • It should generate a bootstrap envoy config, and mount it as configmap on these pods

  • It should allow setting an XDS server to allow dynamically configuring the envoy proxy pods using XDS protocol

Building the controller

  • I experimented with a few frameworks that help in building controllers, like Kubebuilder & Operator SDK, but decided against using them here.

  • First, I created a type, for this resource and generated the clientsets,
    this is how a resource of kind: Envoy would look:

    1
    apiVersion: example.com/v1
    2
    kind: Envoy
    3
    metadata:
    4
      name: edge-envoy
    5
    spec:
    6
      name: "webserver-beta"
    7
      configMapName: "envoy-cfg-1"
    8
      replicas: 3
    9
      xds:
    10
        name: "xds_cluster"
    11
        host: "xds-service.default"
    12
        port: 19000

  • Project structure

    1
    |
    2
    ├── crds
    3
    │   └── envoy-crd.yaml
    4
    ├── go.mod
    5
    ├── go.sum
    6
    ├── LICENSE
    7
    ├── main.go
    8
    ├── pkg
    9
    │   ├── api
    10
    │   │   └── example.com
    11
    │   │       └── v1
    12
    │   │           ├── doc.go
    13
    │   │           ├── register.go
    14
    │   │           ├── types.go
    15
    │   │           └── zz_generated.deepcopy.go
    16
    │   ├── client
    17
    │   │   ├── clientset
    18
    │   │   │   └── versioned
    19
    │   │   │       ├── clientset.go
    20
    │   │   │       ├── doc.go
    21
    │   │   │       ├── fake
    22
    │   │   │       ├── scheme
    23
    │   │   │       └── typed
    24
    │   │   ├── informers
    25
    │   │   │   └── externalversions
    26
    │   │   │       ├── example.com
    27
    │   │   │       ├── factory.go
    28
    │   │   │       ├── generic.go
    29
    │   │   │       └── internalinterfaces
    30
    │   │   └── listers
    31
    │   │       └── example.com
    32
    │   │           └── v1
    33
    │   └── envoy
    34
    │       ├── bootstrap.go
    35
    │       └── utils.go
    36
    ├── README.md
    37
    └── sample
    38
        └── envoy.yaml

  • A queue is created using workqueue from the client-go library, this queue has rate limiting and exponential backoff

  • I then create a SharedInformer for the envoys resource:

    1
    sharedFactory = factory.NewSharedInformerFactory(clientset, time.Second*30)
    2
    informer := sharedFactory.Example().V1().Envoys().Informer()

  • The informer contains a in-memory cache, a listerWatcher (functions that can list your custom resources, and watch the custom resource)

  • It also has a bunch of event handlers, that are called when a specific action(Add, Update, Delete) occurs on that resource

  • The informer periodically lists/watches for changed on the custom resource, and stores that info in its cache.

  • A SharedInformer is an informer where the underlying cache, is shared among controllers.

  • The event handlers for this informer are registered. On Add or Update, the key of the resource is enqueued to our workqueue

  • I then start the main controller loop:

    1
    func work() {
    2
      for {
    3
        key, shutdown := queue.Get()
    4
        if shutdown {
    5
          stopCh <- struct{}{}
    6
          return
    7
        }
    8
        var strKey string
    9
        var ok bool
    10
        if strKey, ok = key.(string); !ok {
    11
          log.Printf("\n Invalid key format %v", key)
    12
          return
    13
        }
    14
        processItem(strKey)
    15
      }
    16
    }

    This (very simple) function dequeues a key, and calls the processItem

  • The processItem retrieves the resource object using the key, and calls the reconcile function.
    It also tells the queue, to forget about this key (and not retry)

  • This reconcile function then deploys or updates the envoyproxy pods, services & configmap, based on this retrieved resource object.

  • The final state after reconciling is updated in the status field of the envoy resource

Testing it out

Installing

1
$ go get github.com/starizard/kube-envoy-controller
2
$ go build

Start the controller:

1
$ ./kube-envoy-controller

In a separate shell:

kubernetes envoy controller demo

The source for this project can be found on github

Future enhancements

  • Sidecar Injection (inject an envoy sidecar in every pod)
  • Implement XDS component
  • Ship envoy access log & expose prometheus metrics