In an attempt to learn more about kubernetes controllers & resources, I decided to make a Custom Resource and a kubernetes custom controller to manage it.

Here are some of my notes:

What does a kubernetes controller do?

A controller is a loop, that compares the current state and the desired state of a resource, and attempts to bring the current state closer to the desired state

For example:

A ReplicaSet resource is managed by a replica_set controller, when the number of pods running is less than the desired number of replicas, the replica_set controller will create new pods to match the desired number of replicas

You can find the replica set code here

What this project should do

  • A simple custom resource with kind: Envoy and a controller to handle this custom resource

  • When created, this controller should deploy and manage a fleet of envoy proxy pods

  • It should generate a bootstrap envoy config, and mount it as configmap on these pods

  • It should allow setting an XDS server to allow dynamically configuring the envoy proxy pods using XDS protocol

Building the controller

  • I experimented with a few frameworks that help in building controllers, like Kubebuilder & Operator SDK, but decided against using them here.

  • First, I created a type, for this resource and generated the clientsets,
    this is how a resource of kind: Envoy would look:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    apiVersion: example.com/v1
    kind: Envoy
    metadata:
      name: edge-envoy
    spec:
      name: "webserver-beta"
      configMapName: "envoy-cfg-1"
      replicas: 3
      xds:
        name: "xds_cluster"
        host: "xds-service.default"
        port: 19000

  • Project structure

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    |
    ├── crds
    │   └── envoy-crd.yaml
    ├── go.mod
    ├── go.sum
    ├── LICENSE
    ├── main.go
    ├── pkg
    │   ├── api
    │   │   └── example.com
    │   │       └── v1
    │   │           ├── doc.go
    │   │           ├── register.go
    │   │           ├── types.go
    │   │           └── zz_generated.deepcopy.go
    │   ├── client
    │   │   ├── clientset
    │   │   │   └── versioned
    │   │   │       ├── clientset.go
    │   │   │       ├── doc.go
    │   │   │       ├── fake
    │   │   │       ├── scheme
    │   │   │       └── typed
    │   │   ├── informers
    │   │   │   └── externalversions
    │   │   │       ├── example.com
    │   │   │       ├── factory.go
    │   │   │       ├── generic.go
    │   │   │       └── internalinterfaces
    │   │   └── listers
    │   │       └── example.com
    │   │           └── v1
    │   └── envoy
    │       ├── bootstrap.go
    │       └── utils.go
    ├── README.md
    └── sample
        └── envoy.yaml

  • A queue is created using workqueue from the client-go library, this queue has rate limiting and exponential backoff

  • I then create a SharedInformer for the envoys resource:

    1
    2
    sharedFactory = factory.NewSharedInformerFactory(clientset, time.Second*30)
    informer := sharedFactory.Example().V1().Envoys().Informer()

  • The informer contains a in-memory cache, a listerWatcher (functions that can list your custom resources, and watch the custom resource)

  • It also has a bunch of event handlers, that are called when a specific action(Add, Update, Delete) occurs on that resource

  • The informer periodically lists/watches for changed on the custom resource, and stores that info in its cache.

  • A SharedInformer is an informer where the underlying cache, is shared among controllers.

  • The event handlers for this informer are registered. On Add or Update, the key of the resource is enqueued to our workqueue

  • I then start the main controller loop:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    func work() {
      for {
        key, shutdown := queue.Get()
        if shutdown {
          stopCh <- struct{}{}
          return
        }
        var strKey string
        var ok bool
        if strKey, ok = key.(string); !ok {
          log.Printf("\n Invalid key format %v", key)
          return
        }
        processItem(strKey)
      }
    }

    This (very simple) function dequeues a key, and calls the processItem

  • The processItem retrieves the resource object using the key, and calls the reconcile function.
    It also tells the queue, to forget about this key (and not retry)

  • This reconcile function then deploys or updates the envoyproxy pods, services & configmap, based on this retrieved resource object.

  • The final state after reconciling is updated in the status field of the envoy resource

Testing it out

Installing

1
2
$ go get github.com/starizard/kube-envoy-controller
$ go build

Start the controller:

1
$ ./kube-envoy-controller

In a separate shell:

kubernetes envoy controller demo

The source for this project can be found on github

Future enhancements

  • Sidecar Injection (inject an envoy sidecar in every pod)
  • Implement XDS component
  • Ship envoy access log & expose prometheus metrics