K8S下redis哨兵集群使用secret隐藏configmap内明文密码方案详解

#作者：朱雷

文章目录

一、背景环境及方案说明
- 1.1、环境说明
- 1.2、方案一：使用配置文件设置密码
- 1.3、方案二：使用args 的命令行传参设置密码
二、redis secret configmap deployment参考
- 2.1 创建secret-redis.yaml参考
- 2.2 修改configmap配置参考
- - 2.2.1 哨兵节点修改（每个节点都修改）
  - 2.2.2 主从节点配置修改
  - 2.2.3 使用命令行参数指定密码（本小节与上面两小节互斥）
- 2.3 修改deployment配置参考
- - 2.3.1 master&slaves节点的deployment的yaml修改
  - 2.3.2 哨兵节点的deployment的yaml修改
  - 2.3.4 镜像环境变量参考
三、测试是否生效
四、注意事项

一、背景环境及方案说明

Redis 哨兵configmap里包含明文密码，需要处理不在configmap内显示明文密码。

1.1、环境说明

方案基于Redis-5.0.14 哨兵版本（5.x，6.x版本兼容）
方案基于redis-sentinel-exporter-5.0.8版本
方案基于容器环境变量
使用两种方案的其中任意一种均可实现

1.2、方案一：使用配置文件设置密码

参照 2.2.1 和2.2.2 修改configmap
参照 2.3.1 和2.3.2 修改deployment

1.3、方案二：使用args 的命令行传参设置密码

参照 2.2.3.1 和2.2.3.2 修改configmap
参照 2.3.3.1 和2.3.3.2 修改deployment

redis_secret_configmap_deployment_16">二、redis secret configmap deployment参考

redisyaml_17">2.1 创建secret-redis.yaml参考

${}内为redis密码的base64的编码，若认证和同步密码不一致分别定义即可
apiVersion: v1
data:
  password: ${aGFyYm9yMjM0NSM=}
kind: Secret
metadata:
  name: redis-auth-secret
  namespace: paas-middleware

2.2 修改configmap配置参考

2.2.1 哨兵节点修改（每个节点都修改）

apiVersion: v1
data:
  redis-docker-entrypoint.sh: |
    #!/bin/bash
    if [ ! -f "/redis-conf/redis.conf" ]; then
        cp /etc/redis/redis.conf /redis-conf/redis.conf
        **echo -e "sentinel auth-pass mymaster ${REDIS_PASSWORD}" >> /redis-conf/redis.conf
    fi**
    redis-sentinel /redis-conf/redis.conf $@
  redis.conf: |
    port 26379
    protected-mode no
    daemonize no
    sentinel monitor mymaster 169.169.164.253 6379 2
    sentinel down-after-milliseconds mymaster 15000
    sentinel failover-timeout mymaster 60000
    sentinel deny-scripts-reconfig yes
    sentinel parallel-syncs mymaster 2
    sentinel auth-pass mymaster somepassword   # 删除这行配置
kind: ConfigMap
metadata:
  labels:
    app: redis-base-1
    type: redis
  name: redis-base-1-sentinel-1
  namespace: paas-middleware
  每个哨兵的configmap 都修改下， 有****不带删除线的为新增行

2.2.2 主从节点配置修改

apiVersion: v1
data:
  redis-docker-entrypoint.sh: |
    #!/bin/bash
    if [ ! -f "/redis-conf/redis.conf" ]; then
        cp /etc/redis/redis.conf /redis-conf/redis.conf
        **echo -e "masterauth ${REDIS_MASTER_PASSWORD}" >> /redis-conf/redis.conf
        echo -e "requirepass ${REDIS_PASSWORD}" >> /redis-conf/redis.conf**
    fi
    redis-server /redis-conf/redis.conf $@
  redis.conf: |
    bind 0.0.0.0 ::
    port 6379
    daemonize no
    protected-mode no
    timeout 300
    tcp-keepalive 300
    replica-read-only yes
    replica-serve-stale-data yes
    maxclients 20000
    maxmemory 0
    maxmemory-policy noeviction
    masterauth somepassword  # 删除此行配置
    requirepass somepassword  # 删除此行配置
    rename-command FLUSHALL ""
    dir "/data/"
    pidfile "/data/redis.pid"
    logfile "/data/redis.log"
kind: ConfigMap
metadata:
  labels:
    app: redis-base-1
    type: redis
  name: redis-base-1-master
  namespace: paas-middleware
 所有主从configmap配置文件都修改， 有****不带删除线的为新增行

2.2.3 使用命令行参数指定密码（本小节与上面两小节互斥）

以下为哨兵节点configmap 修改

apiVersion: v1
data:
  redis-docker-entrypoint.sh: |
    #!/bin/bash
    if [ ! -f "/redis-conf/redis.conf" ]; then
        cp /etc/redis/redis.conf /redis-conf/redis.conf
    fi
    redis-sentinel /redis-conf/redis.conf $@
  redis.conf: |
    port 26379
    protected-mode no
    daemonize no
    sentinel monitor mymaster 169.169.164.253 6379 2
    sentinel down-after-milliseconds mymaster 15000
    sentinel failover-timeout mymaster 60000
    sentinel deny-scripts-reconfig yes
    sentinel parallel-syncs mymaster 2
    sentinel auth-pass mymaster somepassword   # 删除这行配置
kind: ConfigMap
metadata:
  labels:
    app: redis-base-1
    type: redis
  name: redis-base-1-sentinel-1
  namespace: paas-middleware
  每个哨兵的configmap 都修改下， 有****不带删除线的为新增行

以下为主从节点configmap 修改


下面为主从节点实例configmap修改，有****不带删除线为新增行
apiVersion: v1
data:
redis-docker-entrypoint.sh: |
  #!/bin/bash
  if [ ! -f "/redis-conf/redis.conf" ]; then
      cp /etc/redis/redis.conf /redis-conf/redis.conf
  fi
  redis-server /redis-conf/redis.conf $@
redis.conf: |
  bind 0.0.0.0 ::
  port 6379
  daemonize no
  protected-mode no
  timeout 300
  tcp-keepalive 300
  replica-read-only yes
  replica-serve-stale-data yes
  maxclients 20000
  maxmemory 0
  maxmemory-policy noeviction
  masterauth somepassword  # 删除此行配置
  requirepass somepassword  # 删除此行配置
  rename-command FLUSHALL ""
  dir "/data/"
  pidfile "/data/redis.pid"
  logfile "/data/redis.log"
kind: ConfigMap
metadata:
labels:
  app: redis-base-1
  type: redis
name: redis-base-1-master
namespace: paas-middleware

2.3 修改deployment配置参考

2.3.1 master&slaves节点的deployment的yaml修改

所有主从节点配置文件都修改，  有**xxx**为新增行
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
deployment.kubernetes.io/revision: "1"
  labels:
    app: redis-base-1
    type: redis
  name: redis-base-1-master
  namespace: paas-middleware
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: redis-base-1
      name: redis-base-1-master
      servicename: redis-base-1
      type: redis
      withexporter: "yes"
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io/port: "9121"
        prometheus.io/scrape: "true"
      labels:
        app: redis-base-1
        name: redis-base-1-master
        servicename: redis-base-1
        type: redis
        withexporter: "yes"
    spec:
      containers:
      - args:
        - --replica-announce-ip
        - 169.169.164.253
        - --replica-announce-port
        - "6379"
        command:
        - /etc/redis/redis-docker-entrypoint.sh
        image: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latest
        imagePullPolicy: Always
        name: redis
        **env:
         - name: REDIS_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password
         - name: REDIS_MASTER_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password**

        ports:
        - containerPort: 6379
          name: client
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/redis/
          name: config
        - mountPath: /data
          name: data
        - mountPath: /redis-conf
          name: actual-config
      - args:
        - --redis.addr
        - redis://localhost:6379
        - --redis.password
        - somepassword  #密码替换成 $(REDIS_PASSWORD) 变量
        - **$(REDIS_PASSWORD)**
        - --web.listen-address
        - 0.0.0.0:9121
        image: harbor.somedomain/paas_middleware/redis-sentinel-exporter-5.0.8:latest
        imagePullPolicy: Always
        name: redis-exporter
		**env:
         - name: REDIS_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password**
        ports:
        - containerPort: 9121
          name: redis-exporter
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/hostname: 10.179.75.111
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 509
          name: redis-base-1-master
        name: config
      - hostPath:
          path: /data/redis/redis-base-1-master/data
          type: ""
        name: data
      - hostPath:
          path: /data/redis/redis-base-1-master/redis-conf
          type: ""
        name: actual-config

2.3.2 哨兵节点的deployment的yaml修改


所有哨兵节点配置文件都修改， 有****为新增行
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  labels:
    app: redis-base-1
    type: redis
  name: redis-base-1-sentinel-1
  namespace: paas-middleware
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: redis-base-1
      name: redis-base-1-sentinel-1
      role: sentinel
      type: redis
      withexporter: "no"
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: redis-base-1
        name: redis-base-1-sentinel-1
        role: sentinel
        type: redis
        withexporter: "no"
    spec:
      containers:
      - args:
        - --sentinel
        - announce-ip
        - 169.169.196.242
        - --replica-announce-port
        - "26379"
        command:
        - /etc/redis/redis-docker-entrypoint.sh
        image: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latest
        imagePullPolicy: Always
        name: redis
        **env:
         - name: REDIS_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password
         - name: REDIS_MASTER_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password**

        ports:
        - containerPort: 26379
          name: client
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/redis/
          name: config
        - mountPath: /data
          name: data
        - mountPath: /redis-conf
          name: actual-config
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/hostname: 10.179.75.111
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 509
          name: redis-base-1-sentinel-1
        name: config
      - hostPath:
          path: /data/redis/redis-base-1-sentinel-1/data
          type: ""
        name: data
      - hostPath:
          path: /data/redis/redis-base-1-sentinel-1/redis-conf
          type: ""
        name: actual-config
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2023-11-09T03:25:41Z"
    lastUpdateTime: "2023-11-09T03:25:43Z"
    message: ReplicaSet "redis-base-1-sentinel-1-668c76f9bc" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2024-07-21T16:48:34Z"
    lastUpdateTime: "2024-07-21T16:48:34Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 3
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

2.3.3 使用命令行参数指定密码（本小节与上面两小节互斥）

哨兵节点deployment 修改
所有哨兵节点配置文件都修改，有****不带删除线的为新增行

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  labels:
    app: redis-base-1
    type: redis
  name: redis-base-1-sentinel-1
  namespace: paas-middleware
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: redis-base-1
      name: redis-base-1-sentinel-1
      role: sentinel
      type: redis
      withexporter: "no"
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: redis-base-1
        name: redis-base-1-sentinel-1
        role: sentinel
        type: redis
        withexporter: "no"
    spec:
      containers:
      - args:
        - --sentinel
        - announce-ip
        - 169.169.196.242
        - --replica-announce-port
        - "26379"
        - --sentinel
        - auth-pass
        - mymaster
        - $(REDIS_PASSWORD)
        command:
        - /etc/redis/redis-docker-entrypoint.sh
        image: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latest
        imagePullPolicy: Always
        name: redis
        env:
         - name: REDIS_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password
         - name: REDIS_MASTER_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password

        ports:
        - containerPort: 26379
          name: client
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/redis/
          name: config
        - mountPath: /data
          name: data
        - mountPath: /redis-conf
          name: actual-config
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/hostname: 10.179.75.111
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 509
          name: redis-base-1-sentinel-1
        name: config
      - hostPath:
          path: /data/redis/redis-base-1-sentinel-1/data
          type: ""
        name: data
      - hostPath:
          path: /data/redis/redis-base-1-sentinel-1/redis-conf
          type: ""
        name: actual-config
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2023-11-09T03:25:41Z"
    lastUpdateTime: "2023-11-09T03:25:43Z"
    message: ReplicaSet "redis-base-1-sentinel-1-668c76f9bc" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2024-07-21T16:48:34Z"
    lastUpdateTime: "2024-07-21T16:48:34Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 3
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

2、以下为主从实例deployment 配置修改
所有主从节点配置文件都修改，有****不带删除线的为新增行

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
deployment.kubernetes.io/revision: "1"
  labels:
    app: redis-base-1
    type: redis
  name: redis-base-1-master
  namespace: paas-middleware
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: redis-base-1
      name: redis-base-1-master
      servicename: redis-base-1
      type: redis
      withexporter: "yes"
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io/port: "9121"
        prometheus.io/scrape: "true"
      labels:
        app: redis-base-1
        name: redis-base-1-master
        servicename: redis-base-1
        type: redis
        withexporter: "yes"
    spec:
      containers:
      - args:
        - --replica-announce-ip
        - 169.169.164.253
        - --replica-announce-port
        - "6379"
        - --**requirepass
        - $(REDIS_PASSWORD)
        - -- masterauth
        - $(REDIS_MASTER_PASSWORD)**
        command:
        - /etc/redis/redis-docker-entrypoint.sh
        image: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latest
        imagePullPolicy: Always
        name: redis
        **env:
         - name: REDIS_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password
         - name: REDIS_MASTER_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password**

        ports:
        - containerPort: 6379
          name: client
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/redis/
          name: config
        - mountPath: /data
          name: data
        - mountPath: /redis-conf
          name: actual-config
      - args:
        - --redis.addr
        - redis://localhost:6379
        - --redis.password
        - somepassword  #密码替换成 $(REDIS_PASSWORD) 变量
        - **$(REDIS_PASSWORD)**
        - --web.listen-address
        - 0.0.0.0:9121
        image: harbor.somedomain/paas_middleware/redis-sentinel-exporter-5.0.8:latest
        imagePullPolicy: Always
        name: redis-exporter
		**env:
         - name: REDIS_PASSWORD
           valueFrom:
             secretKeyRef:
               name: redis-auth-secret
               key: password**
        ports:
        - containerPort: 9121
          name: redis-exporter
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/hostname: 10.179.75.111
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 509
          name: redis-base-1-master
        name: config
      - hostPath:
          path: /data/redis/redis-base-1-master/data
          type: ""
        name: data
      - hostPath:
          path: /data/redis/redis-base-1-master/redis-conf
          type: ""
        name: actual-config

2.3.4 镜像环境变量参考

https://hub.docker.com/r/bitnami/redis#configuration
https://github.com/oliver006/redis_exporter#flags

三、测试是否生效

Master节点
在这里插入图片描述
Slave节点

哨兵节点

测试redis-sentinel-exporter 指标抓取

四、注意事项

所有节点configmap和deployment yaml 配置文件都按照上面修改别遗漏
修改完先在测试环境验证没有问题，再连接到连接哨兵集群进行读写测试