A blog icon.

Kubernetes CRDs in Action

I’ll show a working example that highlights the basics of custom resources and events in Kubernetes.
USB phone cables arranged to look like a small robot.

In this post, I’ll show a working example of using Kubernetes custom resources and events to capture and use stateful data for service communication and monitoring. The goal here is to help you understand the basics of custom resource definitions (CRDs) and events in Kubernetes so you can create, update, and read Kubernetes resources in a Node application on your own.

The image below shows the architecture of the technical exercise I’ll be demonstrating. You can also follow the technical exercise on Github.

The architecture of the technical exercise that follows.

Kubernetes Custom Resource Definitions and Event Basics

Everything in Kubernetes is a resource, including configMaps, pods, or secrets. You can make calls to the Kubernetes API to create resources, modify resources, retrieve information, and delete resources. You can modify resources by extending the default Kubernetes API with custom resource definitions, which enable non-direct communication between services and tailor those services for use by other resources.

Resources also have events, which help monitor state. Events are objects that are stored and can be accessed on the Kubernetes cluster. Objects contain information such as decisions made by the scheduler or reasons why a pod was evicted. As part of our example, we’ll use events to monitor the state of our resources in our cluster to determine if any changes occurred. We’ll use Kubernetes custom resource definitions with services to demonstrate the modularity of Kubernetes, focusing particularly on how we can easily interchange parts such as the type of custom resource we’re tracking or the data aggregation tool we utilize in our example.

In our example, we’ll be utilizing custom resource definitions to store information from our webhook service and feed data into our data aggregation service. We’ll map out all of the components (webhook listener, Prometheus agent, and resource specs) of the example, as well as map out how they communicate with each other. These services are interchangeable, so we can simply plug and play applications into our architecture as necessary.

Custom Resources

Custom resources are extensions of the Kubernetes API. Below, we’ll create a branch custom resource and a pull request custom resource. These two resources map to our resources in our Git repo, enabling us to register events that will translate into data we can monitor.

Pull Request CRD

Pull-request-crd.yaml

The code below shows a Kubernetes CRD for storing information about pull requests. This resource uses four parts to identify and track each pull request: user, branch, status, and id.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
	name: pullrequests.stable.liatr.io
  labels:
  	app.kubernetes.io/managed-by: helm
  spec:
  	group: stable.liatr.io
    versions:
    	- name: v1
      served: true
      storage: true
    version: v1
    scope: Namespaced
    names:
    	plural: pullrequests
      singular: pullrequest
      kind: PullRequest
    validation:
    	openAPIV3Schema:
      	properties:
        	spec:
          	required: ["user", "branch", "status",  ”id”]
            properties:
            	user:
              	type: string
              branch:
              	type: string
              status:
              	type: string
              id:
              	type: string

Here is sample output from Kubernetes when a pull request CRD is created:

apiVersion: stable.liatr.io/v1
kind: PullRequest
metadata:
	creationTimestamp: “2019-09-19T23:23:12Z”
  generation: 1
  name: feature-x-1568935392031
  namespace: default
spec:
	user: tnishida1
  status: open
  branch: feature-x
  id: 4

Branch CRD

Branch-crd.yaml

Similar to how the pull request CRD is formatted, the code below shows how the branch CRD is structured. It contains two parts, a state determining if a branch was deleted and the name of the branch.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
	name: branches.stable.liatr.io
  labels:
  	app.kubernetes.io/managed-by: helm
  spec:  group: stable.liatr.io
  versions:
  	- name: v1
    	served: true
      storage: true
    version: v1
    scope: Namespaced
    names:
    	plural: branches
      singular: branch
      kind: Branch
    validation:
    	openAPIV3Schema:
      	properties:
        	spec:
          	required: ["state", "name" ]
            properties:
            	state:
              	type: string
              name:
              	type: string

Here is sample output from Kubernetes when the branch CRD is created:

apiVersion: stable.liatr.io/v1
kind: PullRequest
metadata:
	creationTimestamp: “2019-09-19T23:23:12Z”
	generation: 1
  name: feature-x-1568935392031
  namespace: default
spec:
	state: created
  name: feature-x

Services

Webhook Listener

GitHub webhook listener listens for repo pushes and pull requests from GitHub. It links a webhook in GitHub to your resources to reflect the state of the branch or the pull request, as well as creates events based on changes.

Use this Express web endpoint to receive Github webhook requests:

const run = async () => {
	app.post('/webhook', (req, res) => {
  	getRequestType(req.body).then((resp) => {
    	res.status(200).send({
      	success: 'true',
      });
    });
  });
  app.listen(port, () => console.log(`webhook-listener listening on port ${port}`));
}

When a webhook request comes in, we want to create both a Custom Resource, in this case a Pull Request Custom Resource, and a corresponding linked Kubernetes Event. The following is the code to create a Pull Request Custom Resource.

const createPRCRD = async(request) => {
	if (request.pull_request ? request.pull_request:false) {
  	const pullrequest = await client.apis['stable.liatr.io'].v1.namespaces(namespace).pullrequests.post({
    	body: {
      	apiVersion: 'stable.liatr.io/v1',
        kind: 'PullRequest',
        metadata: {
        	name: `${pull_request.head.ref}-${Date.now()}`,
        },
        spec: {
        	user: request.pull_request.user.login,
          branch: request.pull_request.head.ref,
          status: request.pull_request.state,
          id: request.pull_request.number,
        }
      },
    });
  }
}

And this code creates a Kubernetes event with information from the Github webhook:

const createPREvent = async (body) => {
	const timestamp = new Date().toISOString();
  const body = {
  	metadata: {
    	name: `${body.metadata.name}`,
    },
    reason: 'pull_requests',
    message: 'Pull Request',
    type: 'Normal',
    reportingComponent: 'sdm.lead.liatrio/operator-jenkins',
    source: {
    	component: 'sdm.lead.liatrio/operator-jenkins',
    },
    involvedObject: {
    	...pullrequest.metadata,
      apiVersion: pullrequest.apiVersion,
      kind: pullrequest.kind,
    },
    count: 1,
    firstTimestamp: timestamp,
    lastTimestamp: timestamp,
  };
  try {
  	const response = await client.api.v1.namespaces(namespace).events.post({
    	body: body
    });
    setTimeout(createPREvent, Math.random() * 3600000);
    return response.body;
  } catch (e) {
  	console.log(e);
    throw new Error('Error in createPREvent');
  }
  setTimeout(createEvent, Math.random() * 3600000);
};

Prometheus Agent

The Prometheus Agent aggregates data from custom resource events and supplies metrics to Prometheus. The agent watches for new Kubernetes events related to our custom resources and increments a counter when one is created. It also listens for requests from the Prometheus service and responds with the metrics aggregated in the counters.

The code below sets up the counters and data to keep track of the pull request information to be displayed on Prometheus’s view.

const pullRequestCounter = new promClient.Counter({
	name: 'pull_requests',
  help: 'Pull Requests'
});...
	} else {
  	console.log('Watch stream updated');
    pullRequestCounter.inc();
  }
...

Use this code to update the Prometheus endpoint with data from the pull request and branch custom resource definitions:

app.get('/metrics', async (req, res) => {
	res.set('Content-Type', promClient.register.contentType);
  res.end(promClient.register.metrics());}
});

In the example below, we use the JavaScript Kubernetes client to watch for new Kubernetes events and increment our Prometheus counter:

const watch = async () => {
	do {
  	const stream = client.api.v1.watch.namespaces(namespace).events.getStream();
    const jsonStream = new JSONStream();
    stream.pipe(jsonStream);
    await readStream(jsonStream);
    stream.destroy();
    jsonStream.destroy();
  } while (true);
};

const readStream = (jsonStream) => new Promise((resolve, reject) => {
	let skipInitial = true;
  let initialTimeout = setTimeout(() => { skipInitial = false; }, 500);
  jsonStream
  	.on('data', (object) => {
    	if (object.type === 'ADDED'&& skipInitial) {
      	clearTimeout(initialTimeout);
        initialTimeout = setTimeout(() => { skipInitial = false; }, 500);
      } else {
      	console.log('Watch stream updated');
        pullRequestCounter.inc();
      }
    })
    .on('end', (object) => {
    	console.log('Watch stream ended');
      resolve();
    });
  });

Reach Out!

After completing this exercise, you should be able to understand what custom resources are, how to leverage events to represent the state of custom resources, and what options are available for using custom resources for communication between services.

While being able to set up communication in this way is useful, it’s particularly valuable to be able to swap out services, as desired, to make the process more flexible and modular.

Don’t hesitate to reach out if you have any questions or comments!


Share This Article
Have a question or comment?
Contact uS

Related Posts

An image of a coding icon.
Navigating Your Path to the Ideal IDP: Backstage vs. Getport.io
DevOps Tools

This blog explores the vital role of internal development portals (IDPs) like Backstage and Getport.io in enhancing development workflows.

Jon Rudy
2/22/2024
The Github logo, which looks like the outline of a cartoon cat sitting down, surrounded by the words "automate everything".
GitHub API — Automate Everything!
DevOps Tools

This blog explores the diverse capabilities of the GitHub API beyond version control, detailing how it can enhance CICD environments. From cleanup tasks and data funneling to enforcing standardization and enabling self-service jobs, the API's versatility is showcased.

Devin Leaman
2/18/2024
An icon of a software developer in front of a desktop.
ServiceNow is NOT a Developer Platform (iDP)
DevOps Tools

Exploring the limitations of ServiceNow in developer environments, this blog contrasts its rigid, ticket-based system with more agile developer portals, emphasizing the need for collaboration, rapid iteration, and developer autonomy in software development.

Mitchell Phillips
2/14/2024
The Liatrio logo mark.
About Liatrio

Liatrio is a collaborative, end-to-end Enterprise Delivery Acceleration consulting firm that helps enterprises transform the way they work. We work as boots-on-the-ground change agents, helping our clients improve their development practices, react more quickly to market shifts, and get better at delivering value from conception to deployment.