Remoting over Apache Kafka plugin with Kafka launcher in Kubernetes

I am Long Nguyen from FPT University, Vietnam. My project for Google Summer of Code 2019 is Remoting over Apache Kafka with Kubernetes features. This is the first time I have contributed for Jenkins and I am very excited to announce the features that have been done in Phase 1.

Project Introduction

Current version of Remoting over Apache Kafka plugin requires users to manually configure the entire system which includes Zookeeper, Kafka and remoting agents. It also doesn’t support dynamic agent provisioning so scalability is harder to achieve. My project aims to solve two problems:

  1. Out-of-the-box solution to provision Apache Kafka cluster.
  2. Dynamic agent provisioning in a Kubernetes cluster.

Current State

  • Kubernetes connector with credentials supported.
  • Apache Kafka provisioning in Kubernetes feature is fully implemented.
  • Helm chart is partially implemented.

Apache Kafka provisioning in Kubernetes

This feature is part of 2.0 version so it is not yet released officially. You can try out the feature by using the Experimental Update Center to update to 2.0.0-alpha version or building directly from master branch:

git clone
cd remoting-kafka-plugin/plugin
mvn hpi:run

On the Global Configuration page, users can input Kubernetes server information and credentials. Then they can start Apache Kafka with only one button click.

Kafka provisioning in Kubernetes UI

When users click Start Kafka on Kubernetes button, Jenkins will create a Kubernetes client from the information and then apply Zookeeper and Kafka YAML specification files from resources.

Kafka provisioning in Kubernetes architecture

Helm Chart

Helm chart for Remoting over Apache Kafka plugin is based on stable/jenkins chart and incubator/kafka chart. As of now, the chart is still a Work in Progress because it is still waiting for Cloud API implementation in Phase 2. However, you can check out the demo chart with a single standalone Remoting Kafka Agent:

git clone -b demo-helm-phase-1
cd remoting-kafka-plugin
K8S_NODE= ./helm/jenkins-remoting-kafka/ start

The command start will do the following steps:

  • Install the chart (with Jenkins and Kafka).
  • Launch a Kafka computer on Jenkins master by applying the following JCasC.
    - permanent:
        name: "test"
        remoteFS: "/home/jenkins"
          kafka: {}
  • Launch a single Remoting Kafka Agent pod.

You can check the chart state by running kubectl, for example:

$ kubectl get all -n demo-helm
NAME                                    READY   STATUS    RESTARTS   AGE
pod/demo-jenkins-998bcdfd4-tjmjs        2/2     Running   0          6m30s
pod/demo-jenkins-remoting-kafka-agent   1/1     Running   0          4m10s
pod/demo-kafka-0                        1/1     Running   0          6m30s
pod/demo-zookeeper-0                    1/1     Running   0          6m30s

NAME                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/demo-0-external           NodePort           19092:31090/TCP              6m30s
service/demo-jenkins              NodePort             8080:31465/TCP               6m31s
service/demo-jenkins-agent        ClusterIP             50000/TCP                    6m31s
service/demo-kafka                ClusterIP            9092/TCP                     6m30s
service/demo-kafka-headless       ClusterIP   None                     9092/TCP                     6m30s
service/demo-zookeeper            ClusterIP            2181/TCP                     6m30s
service/demo-zookeeper-headless   ClusterIP   None                     2181/TCP,3888/TCP,2888/TCP   6m31s

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/demo-jenkins   1/1     1            1           6m30s

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/demo-jenkins-998bcdfd4   1         1         1       6m30s

NAME                              READY   AGE
statefulset.apps/demo-kafka       1/1     6m30s
statefulset.apps/demo-zookeeper   1/1     6m30s

Next Phase Plan

  • Implement Cloud API to provision Remoting Kafka Agent. (JENKINS-57668)
  • Integrate Cloud API implementation with Helm chart. (JENKINS-58288)
  • Unit tests and integration tests.
  • Release version 2.0 and address feedbacks. (JENKINS-58289)

Originally posted on Jenkins Blog
Author: longnguyen

Deja una respuesta

Tu dirección de correo electrónico no será publicada. Los campos obligatorios están marcados con *