Implementing Kubernetes Operators with Python

Over the past few years, Kubernetes has become a default choice for container orchestration. The features and ease of use make it widely popular amongst the developers. Kubernetes API is one such feature that lets you query and manipulates the state of objects in Kubernetes. And to enhance the functionalities of K8s APIs, you can use a Kubernetes Operator. An Operator is an application-specific controller used to pack, deploy, and manage a Kubernetes application. It can replace the human Operator, someone with technical details of the application, understanding the business requirements, and working accordingly. For example, consider monitoring of the Kubernetes cluster using Prometheus Operator. If you do not use Operator then you need to setup Prometheus Kubernetes. It is a Kubernetes service discovery config and alert manager config for basic alerting rules like installing Grafana for visualization and many more. But when you use Prometheus Operator, everything is taken care of by the Operator. If you want some specific custom configurations or alerts, you need to pass your configuration to the Operator, and bingo, Operator makes it that simple! Just deploy an Operator, and you will find the complete monitoring stack implemented and provided with cluster insights. In simple words, an Operator is a process that runs in a pod and uses custom Kubernetes resources that do not exist in Kubernetes by default. And most importantly, the Operator communicates with the API server in automating the workflow of complicated applications. An Operator is basically a package of two components viz. Custom Resource Definition (CRD) and Controller and works on the principle of control loop Observe -> Diff -> Act. Observe the cluster, find out the difference between the current and desired state, and then act on it to bring it to the desired state. Your controller takes care of your custom resource defined in the cluster. It has the logic built inside it to handle the custom resource created by you. Let me explain this with an example. I will build an Operator in this article to understand the Operators in the easiest possible way. Here, I will create a Grafana operator that will create a Grafana instance and expose the Grafana dashboard to the end-user's port. For this, the end-user needs to create a Grafana object into the cluster and pass the desired nodeport where he wants to access the dashboard. Rest all the logic of creating a pod and nodeport service will be taken care by controller of your Operator. Let's see the actual process -

Writing Operator in Python:

You should use the Kubernetes Operator Pythonic Framework (Kopf) to build a simple Operator. Kopf is a framework used to build Kubernetes Operators in Python language. Just like any framework, Kopf provides you with both outer toolkit and inner libraries. The outer toolkit is used to run the Operator, to connect to the Kubernetes, and to collect the Kubernetes events into the pure Python functions of the Kopf-based Operator. The inner libraries assist with a limited set of common tasks of manipulating the Kubernetes objects. Let's check out the pre-requisites -

Working Kubernetes cluster
IDE for Python
Docker Hub account

Once you have all these, here is how you can create an Operator -

Create a Custom Resource Definition (CRD) An operator defines its custom resource definition. For our example, I will create a custom resource with the name Grafana with the Kubernetes CustomResourceDefinition object's help.

$ cat < custom_resource_definition.yml 
apiVersion: apiextensions.k8s.io/v1beta1 
kind: CustomResourceDefinition 
metadata: 
  name: grafana.opcito.org 
spec: 
  scope: Namespaced 
  group: opcito.org 
  versions: 
    - name: v1 
      served: true 
      storage: true 
  names: 
    kind: Grafana 
    plural: grafana 
    singular: grafana 
    shortNames: 
      - gf 
      - gfn 
EOF 
$ kubectl apply -f custom_resource_definition.yml

This will create a new object Grafana in the cluster, but it will need a controller to manage this object, which is our second step.

Create an Operator handler or controller to manage the created CRD object The script given below will take care of the creation and deletion of your object:

$ vi operator_handler.py 
import kopf 
import kubernetes 
import yaml 
@kopf.on.create('opcito.org', 'v1', 'grafana') 
def create_fn(body, spec, **kwargs): 

    # Get info from grafana object 
    name = body['metadata']['name'] 
    namespace = body['metadata']['namespace'] 
    nodeport = spec['nodeport'] 
    image = 'grafana/grafana' 
    port = 3000 
    if not nodeport: 
        raise kopf.HandlerFatalError(f"Nodeport must be set. Got {nodeport}.") 

      # Pod template 
    pod = {'apiVersion': 'v1', 'metadata': {'name' : name, 'labels': {'app': 'grafana'}},'spec': {'containers': [ { 'image': image, 'name': name }]}} 
 
    # Service template 
    svc = {'apiVersion': 'v1', 'metadata': {'name' : name}, 'spec': { 'selector': {'app': 'grafana'}, 'type': 'NodePort', 'ports': [{ 'port': port, 'targetPort': port,  'nodePort': nodeport }]}} 

    # Make the Pod and Service the children of the grafana object 
    kopf.adopt(pod, owner=body) 
    kopf.adopt(svc, owner=body) 
  
    # Object used to communicate with the API Server 
    api = kubernetes.client.CoreV1Api() 

    # Create Pod 
    obj = api.create_namespaced_pod(namespace, pod) 
    print(f"Pod {obj.metadata.name} created") 

    # Create Service 
    obj = api.create_namespaced_service(namespace, svc) 
    print(f"NodePort Service {obj.metadata.name} created, exposing on port {obj.spec.ports[0].node_port}") 

    # Update status 
    msg = f"Pod and Service created for grafana object {name}" 
    return {'message': msg}
@kopf.on.delete('opcito.org', 'v1', 'grafana') 
def delete(body, **kwargs): 
    msg = f"Grafana {body['metadata']['name']} and its Pod / Service children deleted" 
    return {'message': msg}

The above script has comments mentioned on each line, which describes the task it is performing.

Build Operator handler image The operator controller runs in a pod as a process, and hence we need to create a docker image for it using the below-mentioned dockerfile.

$ vi Dockerfile 

FROM python:3.7 
RUN pip install kopf && pip install kubernetes 
COPY operator_handler.py /operator_handler.py 
CMD kopf run --standalone /operator_handler.py

Build the image and push it to the Docker Hub using the commands mentioned below:

$ docker image build -t sanket07/operator-grafana:latest . 
$ docker image push sanket07/operator-grafana:latest

Create a service account and role binding An operator needs permission to create resources in the cluster. I will assign a service account to the operator pod with permission to create resources in our cluster.

$ cat < service_account.yml 
apiVersion: v1 
kind: ServiceAccount 
metadata: 
  name: grafana-operator 
EOF 

$ kubectl apply -f service_account.yml 
  
$ cat < service_account_binding.yml 
apiVersion: rbac.authorization.k8s.io/v1 
kind: ClusterRoleBinding 
metadata: 
  name: grafana-operator 
roleRef: 
  apiGroup: rbac.authorization.k8s.io 
  kind: ClusterRole 
  name: cluster-admin 
subjects: 
  - kind: ServiceAccount 
    name: grafana-operator 
    namespace: default 

EOF 

$ kubectl apply -f service_account_binding.yml

Create deployment for operator in the cluster

$ cat < grafana_operator.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: grafana-operator
spec:
selector:
matchLabels:
app: grafana-operator
template:
metadata:
labels:
app: grafana-operator
spec:
serviceAccountName: grafana-operator
containers:
- image: sanket07/operator-grafana
name: grafana-operator
EOF

$ kubectl apply -f grafana_operator.yml

Verify if deployment for the operator is successfully running using the following command:

 $ Kubectl get pods

Check for Grafana-operator pod:

Picture1-2

Now that the Operator is successfully deployed, you should test it by creating a Grafana object in the cluster and then try to access the Grafana dashboard using node IP. You can specify the nodeport in the following object definition:

$ cat <<EOF > grafana.yml 
apiVersion: opcito.org/v1 
kind: Grafana 
metadata: 
  name: grafana-testing 
spec: 
  nodeport: 30087 
EOF 
$ kubectl apply -f grafana.yml

You can check the created pod and the service for your Grafana object using the following command:

$ kubectl get pod,svc

Verify pods and service presented with the object name:

Open the browser, visit the nodeIP and nodeport to check whether Grafana instance is available. You can log in to Grafana with admin:admin credentials. Now, this is an elementary example of creating a Grafana operator using Python. But the same process can be used to develop an operator to manage your complex applications and reduce the human intervention required for successful execution. So, try this method, and do not forget to share your experiences and queries in the comments section. Till then, stay safe and happy coding!