Kubernetes CRDs in Action

In this post, I’ll show a working example of using Kubernetes custom resources and events to capture and use stateful data for service communication and monitoring. The goal here is to help you understand the basics of custom resource definitions (CRDs) and events in Kubernetes so you can create, update, and read Kubernetes resources in a Node application on your own.

The image below shows the architecture of the technical exercise I’ll be demonstrating. You can also follow the technical exercise on Github.

The architecture of the technical exercise that follows.

Kubernetes Custom Resource Definitions and Event Basics

Everything in Kubernetes is a resource, including configMaps, pods, or secrets. You can make calls to the Kubernetes API to create resources, modify resources, retrieve information, and delete resources. You can modify resources by extending the default Kubernetes API with custom resource definitions, which enable non-direct communication between services and tailor those services for use by other resources.

Resources also have events, which help monitor state. Events are objects that are stored and can be accessed on the Kubernetes cluster. Objects contain information such as decisions made by the scheduler or reasons why a pod was evicted. As part of our example, we’ll use events to monitor the state of our resources in our cluster to determine if any changes occurred. We’ll use Kubernetes custom resource definitions with services to demonstrate the modularity of Kubernetes, focusing particularly on how we can easily interchange parts such as the type of custom resource we’re tracking or the data aggregation tool we utilize in our example.

In our example, we’ll be utilizing custom resource definitions to store information from our webhook service and feed data into our data aggregation service. We’ll map out all of the components (webhook listener, Prometheus agent, and resource specs) of the example, as well as map out how they communicate with each other. These services are interchangeable, so we can simply plug and play applications into our architecture as necessary.

Custom Resources

Custom resources are extensions of the Kubernetes API. Below, we’ll create a branch custom resource and a pull request custom resource. These two resources map to our resources in our Git repo, enabling us to register events that will translate into data we can monitor.

Pull Request CRD

Pull-request-crd.yaml

The code below shows a Kubernetes CRD for storing information about pull requests. This resource uses four parts to identify and track each pull request: user, branch, status, and id.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
	name: pullrequests.stable.liatr.io
  labels:
  	app.kubernetes.io/managed-by: helm
  spec:
  	group: stable.liatr.io
    versions:
    	- name: v1
      served: true
      storage: true
    version: v1
    scope: Namespaced
    names:
    	plural: pullrequests
      singular: pullrequest
      kind: PullRequest
    validation:
    	openAPIV3Schema:
      	properties:
        	spec:
          	required: ["user", "branch", "status",  ”id”]
            properties:
            	user:
              	type: string
              branch:
              	type: string
              status:
              	type: string
              id:
              	type: string

Here is sample output from Kubernetes when a pull request CRD is created:

apiVersion: stable.liatr.io/v1
kind: PullRequest
metadata:
	creationTimestamp: “2019-09-19T23:23:12Z”
  generation: 1
  name: feature-x-1568935392031
  namespace: default
spec:
	user: tnishida1
  status: open
  branch: feature-x
  id: 4

Branch CRD

Branch-crd.yaml

Similar to how the pull request CRD is formatted, the code below shows how the branch CRD is structured. It contains two parts, a state determining if a branch was deleted and the name of the branch.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
	name: branches.stable.liatr.io
  labels:
  	app.kubernetes.io/managed-by: helm
  spec:  group: stable.liatr.io
  versions:
  	- name: v1
    	served: true
      storage: true
    version: v1
    scope: Namespaced
    names:
    	plural: branches
      singular: branch
      kind: Branch
    validation:
    	openAPIV3Schema:
      	properties:
        	spec:
          	required: ["state", "name" ]
            properties:
            	state:
              	type: string
              name:
              	type: string

Here is sample output from Kubernetes when the branch CRD is created:

apiVersion: stable.liatr.io/v1
kind: PullRequest
metadata:
	creationTimestamp: “2019-09-19T23:23:12Z”
	generation: 1
  name: feature-x-1568935392031
  namespace: default
spec:
	state: created
  name: feature-x

Services

Webhook Listener

GitHub webhook listener listens for repo pushes and pull requests from GitHub. It links a webhook in GitHub to your resources to reflect the state of the branch or the pull request, as well as creates events based on changes.

Use this Express web endpoint to receive Github webhook requests:

const run = async () => {
	app.post('/webhook', (req, res) => {
  	getRequestType(req.body).then((resp) => {
    	res.status(200).send({
      	success: 'true',
      });
    });
  });
  app.listen(port, () => console.log(`webhook-listener listening on port ${port}`));
}

When a webhook request comes in, we want to create both a Custom Resource, in this case a Pull Request Custom Resource, and a corresponding linked Kubernetes Event. The following is the code to create a Pull Request Custom Resource.

const createPRCRD = async(request) => {
	if (request.pull_request ? request.pull_request:false) {
  	const pullrequest = await client.apis['stable.liatr.io'].v1.namespaces(namespace).pullrequests.post({
    	body: {
      	apiVersion: 'stable.liatr.io/v1',
        kind: 'PullRequest',
        metadata: {
        	name: `${pull_request.head.ref}-${Date.now()}`,
        },
        spec: {
        	user: request.pull_request.user.login,
          branch: request.pull_request.head.ref,
          status: request.pull_request.state,
          id: request.pull_request.number,
        }
      },
    });
  }
}

And this code creates a Kubernetes event with information from the Github webhook:

const createPREvent = async (body) => {
	const timestamp = new Date().toISOString();
  const body = {
  	metadata: {
    	name: `${body.metadata.name}`,
    },
    reason: 'pull_requests',
    message: 'Pull Request',
    type: 'Normal',
    reportingComponent: 'sdm.lead.liatrio/operator-jenkins',
    source: {
    	component: 'sdm.lead.liatrio/operator-jenkins',
    },
    involvedObject: {
    	...pullrequest.metadata,
      apiVersion: pullrequest.apiVersion,
      kind: pullrequest.kind,
    },
    count: 1,
    firstTimestamp: timestamp,
    lastTimestamp: timestamp,
  };
  try {
  	const response = await client.api.v1.namespaces(namespace).events.post({
    	body: body
    });
    setTimeout(createPREvent, Math.random() * 3600000);
    return response.body;
  } catch (e) {
  	console.log(e);
    throw new Error('Error in createPREvent');
  }
  setTimeout(createEvent, Math.random() * 3600000);
};

Prometheus Agent

The Prometheus Agent aggregates data from custom resource events and supplies metrics to Prometheus. The agent watches for new Kubernetes events related to our custom resources and increments a counter when one is created. It also listens for requests from the Prometheus service and responds with the metrics aggregated in the counters.

The code below sets up the counters and data to keep track of the pull request information to be displayed on Prometheus’s view.

const pullRequestCounter = new promClient.Counter({
	name: 'pull_requests',
  help: 'Pull Requests'
});...
	} else {
  	console.log('Watch stream updated');
    pullRequestCounter.inc();
  }
...

Use this code to update the Prometheus endpoint with data from the pull request and branch custom resource definitions:

app.get('/metrics', async (req, res) => {
	res.set('Content-Type', promClient.register.contentType);
  res.end(promClient.register.metrics());}
});

In the example below, we use the JavaScript Kubernetes client to watch for new Kubernetes events and increment our Prometheus counter:

const watch = async () => {
	do {
  	const stream = client.api.v1.watch.namespaces(namespace).events.getStream();
    const jsonStream = new JSONStream();
    stream.pipe(jsonStream);
    await readStream(jsonStream);
    stream.destroy();
    jsonStream.destroy();
  } while (true);
};

const readStream = (jsonStream) => new Promise((resolve, reject) => {
	let skipInitial = true;
  let initialTimeout = setTimeout(() => { skipInitial = false; }, 500);
  jsonStream
  	.on('data', (object) => {
    	if (object.type === 'ADDED'&& skipInitial) {
      	clearTimeout(initialTimeout);
        initialTimeout = setTimeout(() => { skipInitial = false; }, 500);
      } else {
      	console.log('Watch stream updated');
        pullRequestCounter.inc();
      }
    })
    .on('end', (object) => {
    	console.log('Watch stream ended');
      resolve();
    });
  });

Reach Out!

After completing this exercise, you should be able to understand what custom resources are, how to leverage events to represent the state of custom resources, and what options are available for using custom resources for communication between services.

While being able to set up communication in this way is useful, it’s particularly valuable to be able to swap out services, as desired, to make the process more flexible and modular.

Don’t hesitate to reach out if you have any questions or comments!