A blog icon.

Kubernetes CRDs in Action

I’ll show a working example that highlights the basics of custom resources and events in Kubernetes.
USB phone cables arranged to look like a small robot.

In this post, I’ll show a working example of using Kubernetes custom resources and events to capture and use stateful data for service communication and monitoring. The goal here is to help you understand the basics of custom resource definitions (CRDs) and events in Kubernetes so you can create, update, and read Kubernetes resources in a Node application on your own.

The image below shows the architecture of the technical exercise I’ll be demonstrating. You can also follow the technical exercise on Github.

The architecture of the technical exercise that follows.

Kubernetes Custom Resource Definitions and Event Basics

Everything in Kubernetes is a resource, including configMaps, pods, or secrets. You can make calls to the Kubernetes API to create resources, modify resources, retrieve information, and delete resources. You can modify resources by extending the default Kubernetes API with custom resource definitions, which enable non-direct communication between services and tailor those services for use by other resources.

Resources also have events, which help monitor state. Events are objects that are stored and can be accessed on the Kubernetes cluster. Objects contain information such as decisions made by the scheduler or reasons why a pod was evicted. As part of our example, we’ll use events to monitor the state of our resources in our cluster to determine if any changes occurred. We’ll use Kubernetes custom resource definitions with services to demonstrate the modularity of Kubernetes, focusing particularly on how we can easily interchange parts such as the type of custom resource we’re tracking or the data aggregation tool we utilize in our example.

In our example, we’ll be utilizing custom resource definitions to store information from our webhook service and feed data into our data aggregation service. We’ll map out all of the components (webhook listener, Prometheus agent, and resource specs) of the example, as well as map out how they communicate with each other. These services are interchangeable, so we can simply plug and play applications into our architecture as necessary.

Custom Resources

Custom resources are extensions of the Kubernetes API. Below, we’ll create a branch custom resource and a pull request custom resource. These two resources map to our resources in our Git repo, enabling us to register events that will translate into data we can monitor.

Pull Request CRD

Pull-request-crd.yaml

The code below shows a Kubernetes CRD for storing information about pull requests. This resource uses four parts to identify and track each pull request: user, branch, status, and id.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
	name: pullrequests.stable.liatr.io
  labels:
  	app.kubernetes.io/managed-by: helm
  spec:
  	group: stable.liatr.io
    versions:
    	- name: v1
      served: true
      storage: true
    version: v1
    scope: Namespaced
    names:
    	plural: pullrequests
      singular: pullrequest
      kind: PullRequest
    validation:
    	openAPIV3Schema:
      	properties:
        	spec:
          	required: ["user", "branch", "status",  ”id”]
            properties:
            	user:
              	type: string
              branch:
              	type: string
              status:
              	type: string
              id:
              	type: string

Here is sample output from Kubernetes when a pull request CRD is created:

apiVersion: stable.liatr.io/v1
kind: PullRequest
metadata:
	creationTimestamp: “2019-09-19T23:23:12Z”
  generation: 1
  name: feature-x-1568935392031
  namespace: default
spec:
	user: tnishida1
  status: open
  branch: feature-x
  id: 4

Branch CRD

Branch-crd.yaml

Similar to how the pull request CRD is formatted, the code below shows how the branch CRD is structured. It contains two parts, a state determining if a branch was deleted and the name of the branch.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
	name: branches.stable.liatr.io
  labels:
  	app.kubernetes.io/managed-by: helm
  spec:  group: stable.liatr.io
  versions:
  	- name: v1
    	served: true
      storage: true
    version: v1
    scope: Namespaced
    names:
    	plural: branches
      singular: branch
      kind: Branch
    validation:
    	openAPIV3Schema:
      	properties:
        	spec:
          	required: ["state", "name" ]
            properties:
            	state:
              	type: string
              name:
              	type: string

Here is sample output from Kubernetes when the branch CRD is created:

apiVersion: stable.liatr.io/v1
kind: PullRequest
metadata:
	creationTimestamp: “2019-09-19T23:23:12Z”
	generation: 1
  name: feature-x-1568935392031
  namespace: default
spec:
	state: created
  name: feature-x

Services

Webhook Listener

GitHub webhook listener listens for repo pushes and pull requests from GitHub. It links a webhook in GitHub to your resources to reflect the state of the branch or the pull request, as well as creates events based on changes.

Use this Express web endpoint to receive Github webhook requests:

const run = async () => {
	app.post('/webhook', (req, res) => {
  	getRequestType(req.body).then((resp) => {
    	res.status(200).send({
      	success: 'true',
      });
    });
  });
  app.listen(port, () => console.log(`webhook-listener listening on port ${port}`));
}

When a webhook request comes in, we want to create both a Custom Resource, in this case a Pull Request Custom Resource, and a corresponding linked Kubernetes Event. The following is the code to create a Pull Request Custom Resource.

const createPRCRD = async(request) => {
	if (request.pull_request ? request.pull_request:false) {
  	const pullrequest = await client.apis['stable.liatr.io'].v1.namespaces(namespace).pullrequests.post({
    	body: {
      	apiVersion: 'stable.liatr.io/v1',
        kind: 'PullRequest',
        metadata: {
        	name: `${pull_request.head.ref}-${Date.now()}`,
        },
        spec: {
        	user: request.pull_request.user.login,
          branch: request.pull_request.head.ref,
          status: request.pull_request.state,
          id: request.pull_request.number,
        }
      },
    });
  }
}

And this code creates a Kubernetes event with information from the Github webhook:

const createPREvent = async (body) => {
	const timestamp = new Date().toISOString();
  const body = {
  	metadata: {
    	name: `${body.metadata.name}`,
    },
    reason: 'pull_requests',
    message: 'Pull Request',
    type: 'Normal',
    reportingComponent: 'sdm.lead.liatrio/operator-jenkins',
    source: {
    	component: 'sdm.lead.liatrio/operator-jenkins',
    },
    involvedObject: {
    	...pullrequest.metadata,
      apiVersion: pullrequest.apiVersion,
      kind: pullrequest.kind,
    },
    count: 1,
    firstTimestamp: timestamp,
    lastTimestamp: timestamp,
  };
  try {
  	const response = await client.api.v1.namespaces(namespace).events.post({
    	body: body
    });
    setTimeout(createPREvent, Math.random() * 3600000);
    return response.body;
  } catch (e) {
  	console.log(e);
    throw new Error('Error in createPREvent');
  }
  setTimeout(createEvent, Math.random() * 3600000);
};

Prometheus Agent

The Prometheus Agent aggregates data from custom resource events and supplies metrics to Prometheus. The agent watches for new Kubernetes events related to our custom resources and increments a counter when one is created. It also listens for requests from the Prometheus service and responds with the metrics aggregated in the counters.

The code below sets up the counters and data to keep track of the pull request information to be displayed on Prometheus’s view.

const pullRequestCounter = new promClient.Counter({
	name: 'pull_requests',
  help: 'Pull Requests'
});...
	} else {
  	console.log('Watch stream updated');
    pullRequestCounter.inc();
  }
...

Use this code to update the Prometheus endpoint with data from the pull request and branch custom resource definitions:

app.get('/metrics', async (req, res) => {
	res.set('Content-Type', promClient.register.contentType);
  res.end(promClient.register.metrics());}
});

In the example below, we use the JavaScript Kubernetes client to watch for new Kubernetes events and increment our Prometheus counter:

const watch = async () => {
	do {
  	const stream = client.api.v1.watch.namespaces(namespace).events.getStream();
    const jsonStream = new JSONStream();
    stream.pipe(jsonStream);
    await readStream(jsonStream);
    stream.destroy();
    jsonStream.destroy();
  } while (true);
};

const readStream = (jsonStream) => new Promise((resolve, reject) => {
	let skipInitial = true;
  let initialTimeout = setTimeout(() => { skipInitial = false; }, 500);
  jsonStream
  	.on('data', (object) => {
    	if (object.type === 'ADDED'&& skipInitial) {
      	clearTimeout(initialTimeout);
        initialTimeout = setTimeout(() => { skipInitial = false; }, 500);
      } else {
      	console.log('Watch stream updated');
        pullRequestCounter.inc();
      }
    })
    .on('end', (object) => {
    	console.log('Watch stream ended');
      resolve();
    });
  });

Reach Out!

After completing this exercise, you should be able to understand what custom resources are, how to leverage events to represent the state of custom resources, and what options are available for using custom resources for communication between services.

While being able to set up communication in this way is useful, it’s particularly valuable to be able to swap out services, as desired, to make the process more flexible and modular.

Don’t hesitate to reach out if you have any questions or comments!


Share This Article
Have a question or comment?
Contact uS

Related Posts

Terraform 6 ways
What Is Terraform Used for? 6 Ways to Use Terraform

Terraform is an open-source infrastructure as code (IaC) tool that enables users to define and manage their cloud infrastructure in a declarative and reproducible manner.

Markdown examples floating on top of a laptop computer.
Better Markdown Means Better DevOps

Your Github READMEs and Pull Requests don't have to be boring. Here are 5 markdown features to help level-up repo docs and dev workflow.

Github Actions workflow diagrams
Github Actions For Everything: It Does More Than Build Code

Github Actions has a large set of “workflow triggers” that can be used to kick off new pipeline runs. Used in tandem with the Github Actions and APIs, workflows can be used to automate many parts of the SDLC beyond building and deploying code.

An illustration of a tree with a large trunk and numerous small branches, against a green gradient background.
GitOps: Defining the Best Infrastructure Pattern For You

A trunk-based GitOps approach enables users to deliver software more quickly and efficiently.

The Liatrio logo mark.
About Liatrio

Liatrio is a collaborative, end-to-end Enterprise Delivery Acceleration consulting firm that helps enterprises transform the way they work. We work as boots-on-the-ground change agents, helping our clients improve their development practices, react more quickly to market shifts, and get better at delivering value from conception to deployment.