K8s部署Jenkins主从节点并与Gitblit集成

一、制作Jenkins-master镜像

拉取base镜像

docker pull jenkinsci/jenkins:lts
运行base镜像，为便于构建，这里打算手动运行和安装好镜像内容后进行docker commit出一个新镜像。

docker run -tid -p 8080:8080 –name jenkins-master jenkinsci/jenkins:lts
访问jenkins-master镜像暴露的url: http://<ip>:8080
根据提示在容器内获取admin的初始秘钥。
安装推荐的插件（Suggested plugins），如果没有安装Kubernetes插件，则在

系统管理–>管理插件–>可选插件–>输入 Kubernetes
选择 Kubernetes plugin–>直接安装
复制kubectl命令到容器中

docker cp kubectl jenkins-master:/usr/bin
安装 libltdl7

docker exec -ti -u root jenkins-master /bin/sh
apt-get update
apt-get install libltdl7
apt-get clean
把Jenkins Home的文件拷贝出来，作为在k8s上持久化存放（Gluster或者NFS文件系统）。注意：主机上 .kube/config 文件也要放到这个目录下以供jenkins的k8s插件与kubernetes交互。

docker cp jenkins-master:/var/jenkins_home ./config/jenkins_home
mkdir -p ./config/jenkins_home/.kube && cp /root/.kube/config ./config/jenkins_home.kube/
如果后续需要在master上构建工程，安装所需要的组件（如nodejs、maven、JDK等）。然后提交镜像。

docker commit jenkins-master jenkins-k8s-master:v1

二、部署Jenkins-master

环境中master节点用Rancher部署，这里截取部分yaml内容以供参考：

jenkins-deploy.yaml

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "3"
  labels:
    workload.user.cattle.io/workloadselector: deployment-ci-cd-jenkins-master
  name: jenkins-master
  namespace: ci-cd
spec:
  progressDeadlineSeconds: 
  replicas: 
  revisionHistoryLimit: 
  selector:
    matchLabels:
      workload.user.cattle.io/workloadselector: deployment-ci-cd-jenkins-master
  strategy:
    rollingUpdate:
      maxSurge: 
      maxUnavailable: 
    type: RollingUpdate
  template:
    metadata:
      annotations:
      labels:
        workload.user.cattle.io/workloadselector: deployment-ci-cd-jenkins-master
    spec:
      affinity: {}
      containers:
      - image: jenkins-k8s-master:v1
        imagePullPolicy: IfNotPresent
        name: jenkins-master
        ports:
        - containerPort: 
          hostPort: 
          protocol: TCP
        - containerPort: 
          hostPort: 
          protocol: TCP
        resources: {}
        volumeMounts:
        - mountPath: /var/run/docker.sock
          name: dockersock
        - mountPath: /var/jenkins_home
          name: jenkinshome
        - mountPath: /root/.kube
          name: kube-config
        - mountPath: /usr/bin/docker
          name: docker-binary
      dnsPolicy: ClusterFirst
      nodeName: izwz9ghgtf18pi19ytou0vz
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 
      volumes:
      - hostPath:
          path: /var/run/docker.sock
          type: ""
        name: dockersock
      - hostPath:
          path: /mnt/jenkins/jenkins_home
          type: ""
        name: jenkinshome
      - hostPath:
          path: /root/.kube
          type: ""
        name: kube-config
      - hostPath:
          path: /usr/bin/docker
          type: ""
        name: docker-binary

jenkins-svc.yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
    field.cattle.io/targetWorkloadIds: '["deployment:ci-cd:jenkins-master"]'
  name: jenkins-master
  namespace: ci-cd
spec:
  ports:
  - name: tcp01-jenkins-master
    port: 
    protocol: TCP
    targetPort: 
  - name: tcp01-jenkins-master
    port: 
    protocol: TCP
    targetPort: 
  selector:
    workload.user.cattle.io/workloadselector: deployment-ci-cd-jenkins-master
  sessionAffinity: None
  type: ClusterIP

三、制作Jenkins-slave镜像

在Dockerfile所在目录准备好以下内容

maven仓库的位置epel-apache-maven.repo

wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo
kubectl可执行文件
jenkins-slave可执行文件
需要特殊配置的hosts文件
其它构建工具如node等
下载base镜像centos

docker pull centos

vi Dockerfile

FROM  centos
ARG AGENT_WORKDIR=/root/agent
RUN curl --create-dirs -sSLo /usr/share/jenkins/slave.jar https://repo.jenkins-ci.org/public/org/jenkins-ci/main/remoting/3.16/remoting-3.16.jar \
  && chmod  /usr/share/jenkins

COPY epel-apache-maven.repo /etc/yum.repos.d/epel-apache-maven.repo
RUN yum -y install git libtool-ltdl java--openjdk apache-maven
RUN rm -rf /var/cache/yum/x86_64//*

ENV AGENT_WORKDIR=${AGENT_WORKDIR}
COPY node /usr/local/
COPY kubectl /usr/bin/kubectl

COPY jenkins-slave /usr/local/bin/jenkins-slave
COPY .kube /root/.kube
COPY hosts /tmp/hosts
COPY ojdbc6.jar /root/ojdbc6.jar
RUN mkdir -p -- /lib-override && cp /usr/lib64/libnss_files.so.2 /lib-override
RUN sed -i 's:/etc/hosts:/tmp/hosts:g' /lib-override/libnss_files.so.2
ENV LD_LIBRARY_PATH /lib-override

RUN mkdir /root/.jenkins && mkdir -p ${AGENT_WORKDIR}
ENV JAVA_HOME=/usr/lib/jvm/java-1.8.0
WORKDIR /root
ENTRYPOINT ["jenkins-slave"]

构建slave镜像

docker build -t jenkins-slave:v1 .

四、配置Jenkins与K8s协作

在Jenkins-master的UI界面上进入

系统管理 > 系统设置 > （最下方）新增一个云 > 选择Kubernetes

参考下图的配置

Jenkins配置K8s插件

添加Pod template

Jenkins配置K8s Pod模板

五、配置Gitblit

Gitblit（这里使用版本是1.8.0）作为K8s外部系统，这里不考虑其部署。但为了和Jenkins交互，需要修改一些配置：

data目录下新建 gitblit.properties

include = defaults.properties

#
# Define your overrides or custom settings below
#
# web.canonicalUrl为gitblit自身的url，用于生成工程的访问路径等。
web.canonicalUrl = http://172.18.109.238:3000

# groovy.jenkinsServer为Jenkins-master的访问路径，Gitblit会通过该url调用Jenkins-master上的hook来触发构建
groovy.jenkinsServer = http://172.18.63.25:8080

新建maven项目如下图所示

六、配置Jenkins-master的工程和代码

在Jenkins-master UI中，新建一个任务，配置：

构建触发器：轮询SCM，日程表留空
流水线：定义Pipeline Script from SCM
SCM选择Git，Repositories写Gitblit的代码库url和登陆用户
选择要build的代码线（branch）
脚本路径选择默认的Jenkinsfile

代码中新增Jenkinsfile
vi jenkinsfile

node('jnlp-slave') {
  stage('git clone'){
    checkout scm
  }

  stage 'build'
    sh "mvn install:install-file -DgroupId=com.oracle -DartifactId=ojdbc6 -Dversion=11.2.0.1.0 -Dpackaging=jar -Dfile=/root/ojdbc6.jar"
    sh "mvn package"
    sh "echo done.."
}

在git中提交代码