静态加密 Secret 数据

因为 secret 默认存储为 base64 编码的非加密字符串，所以启用静态加密来增加安全性。

1 创建配置文件 encrypt.conf

配置文件样例如下:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: <BASE 64 ENCODED SECRET>
    - identity: {}

// resources.resources 里的内容就是要加密的对象，此处为 secrets;
// providers 里是加解密的实际提供者 —— identity, aescbc, secretbox, aesgcm, kms. 综合对比，aescec 最合适。
// 加密时会使用 providers 里的第一个来加密，解密时会从头到尾尝试直至成功解密。
// identity 就是明文，其余都为加密算法。
// 此处的配置文件最后是 identity，是为了服务在此之前已经存储了的明文 secrets 。

需生成随机密钥放入 secret 字段中，直接进行如下操作：

[centos@ml-k8s-1 ~]$ head -c 32 /dev/urandom | base64
7unrE0P6q9DmlqGVeZ+k02d2eNaAgveFC01V+gV3VYc=

// 生成了一个 32 字节的随机密钥并进行了 base64 编码

配置文件命名为 encrypt.conf , 并存放在 /etc/kubernetes/pki/

[centos@ml-k8s-1 ~]$ cd /etc/kubernetes/pki/
[centos@ml-k8s-1 pki]$ ls
apiserver-etcd-client.crt  apiserver-kubelet-client.crt  apiserver.crt  ca.crt  encrypt.conf  front-proxy-ca.crt  front-proxy-client.crt  sa.key
apiserver-etcd-client.key  apiserver-kubelet-client.key  apiserver.key  ca.key  etcd          front-proxy-ca.key  front-proxy-client.key  sa.pub

2 设置 kube-apiserver

增加 –encryption-provider-config ，指向 encrypt.conf

[centos@ml-k8s-1 ~]$ cd /etc/kubernetes/manifests
[centos@ml-k8s-1 manifests]$ ls
etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml
[centos@ml-k8s-1 manifests]$ sudo vim kube-apiserver.yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.20.9.60:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=10.20.9.60
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --encryption-provider-config=/etc/kubernetes/pki/encrypt.conf

// 若使用 kubeadm 搭建的集群，此时会自动重启 API server , 否则需要手动重启。

3 验证数据已被加密

重新启动 kube-apiserver 后，任何新创建或更新的 secret 在存储时都应该被加密

在 default 命名空间里创建一个名为 secret1 的 secret ：

[centos@ml-k8s-1 ~]$ kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
secret/secret1 created
[centos@ml-k8s-1 ~]$ kubectl get secret
NAME                  TYPE                                  DATA   AGE
default-token-44r9r   kubernetes.io/service-account-token   3      18h
secret1               Opaque                                1      23s

使用 etcdctl 命令行，从 etcd 中读取 secret ：

[centos@ml-k8s-1 etcd]$ sudo ETCDCTL_API=3 ./etcdctl  --cacert=/etc/kubernetes/pki/etcd/ca.crt  --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt   --key=/etc/kubernetes/pki/apiserver-etcd-client.key get /registry/secrets/default/secret1 | hexdump -C
00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 64 65 66 61 75 6c  74 2f 73 65 63 72 65 74  |s/default/secret|
00000020  31 0a 6b 38 73 3a 65 6e  63 3a 61 65 73 63 62 63  |1.k8s:enc:aescbc|
00000030  3a 76 31 3a 6b 65 79 31  3a cb d8 67 c6 67 21 07  |:v1:key1:..g.g!.|
00000040  e6 e7 f8 97 01 4a 76 24  2c bb 23 3f 9c 6f 4c 10  |.....Jv$,.#?.oL.|
00000050  44 ce a9 d9 a4 db 1d b3  76 6c d2 a2 dc 41 7e 55  |D.......vl...A~U|
00000060  b2 da fc c0 6d fb 0b 79  0c 3e 87 70 6e 5b 31 5d  |....m..y.>.pn[1]|
00000070  f7 8d 7f de f8 ec 4d 2d  ec eb c5 a2 3a 80 95 4f  |......M-....:..O|
00000080  b0 de 3b e1 83 f4 76 32  f5 d4 19 d2 96 6a 46 df  |..;...v2.....jF.|
00000090  03 ca d4 e1 3d 9d 48 6e  81 61 ec 18 04 b2 73 d2  |....=.Hn.a....s.|
000000a0  cd f8 1b e5 bd 2f cc 0e  68 b5 24 d7 7a 44 d2 8e  |...../..h.$.zD..|
000000b0  6c 2c fb 3a 43 e8 7c 37  c8 bd e6 06 dd 29 04 0d  |l,.:C.|7.....)..|
000000c0  aa 71 c1 75 8d 3d de 39  d4 15 26 c5 e9 af a9 c4  |.q.u.=.9..&.....|
000000d0  2d c6 4d 1a 36 7c 4d ec  10 50 20 69 dc 28 49 d2  |-.M.6|M..P i.(I.|
000000e0  af e4 10 44 f2 9e ad 98  b9 f8 43 d3 74 4f 1a 05  |...D......C.tO..|
000000f0  13 d0 72 cf f4 60 12 26  af 47 ee a7 de 1d 9f e2  |..r..`.&.G......|
00000100  41 ba 33 2c 45 3c 98 f9  0c d8 3d 3f 90 bd 06 be  |A.3,E<....=?....|
00000110  64 9a f4 d9 48 94 93 21  f1 ed 54 e7 be 59 31 d8  |d...H..!..T..Y1.|
00000120  1b 40 e0 9b 51 14 6b 9a  96 ed 7a 42 95 20 b6 b5  |.@..Q.k...zB. ..|
00000130  5b 02 99 8d 3a 66 c1 1d  19 0a                    |[...:f....|
0000013a

//  cacert, cert, key 是用来连接 etcd 服务的证书
// hexdump 命令一般用来查看“二进制”文件的十六进制编码
// 数据头部出现 k8s:enc:aescbc:v1: ,说明数据已经被正确加密，且使用的是 aescbc 算法，密钥为 key1

验证 secret 是否被正确解密：

[centos@ml-k8s-1 etcd]$  kubectl get secrets secret1 -o yaml
apiVersion: v1
data:
  mykey: bXlkYXRh
kind: Secret
metadata:
  creationTimestamp: "2021-05-06T03:09:25Z"
  name: secret1
  namespace: default
  resourceVersion: "89523"
  uid: dc8e6c8d-fb3c-47b3-b2bb-ed5014fb3779
type: Opaque

// mykey: bXlkYXRh ,数据已被加密
// 加密插件只是加密了 etcd 中保存的数据,执行这类 kubectl get secrets mysecret -o yaml 命令时，kube-apiserver 在从  etcd 中取出数据的时候已经自动解密了
// 此时用 decode 即可得到数据

[centos@ml-k8s-1 etcd]$ echo -n "bXlkYXRh" | base64 --decode
mydata

4 确保所有 secret 都被加密

读取所有 Secret，然后通过更新来实现全部加密

[centos@ml-k8s-1 etcd]$ kubectl get secrets --all-namespaces -o json | kubectl replace -f -
secret/default-token-44r9r replaced
secret/secret1 replaced
secret/default-token-d49sg replaced
secret/default-token-dhpj7 replaced
secret/attachdetach-controller-token-xfvx9 replaced
secret/bootstrap-signer-token-4z576 replaced
secret/bootstrap-token-rliyei replaced
secret/certificate-controller-token-6g4hj replaced
secret/clusterrole-aggregation-controller-token-vzn58 replaced
secret/coredns-token-bkmbn replaced
secret/cronjob-controller-token-x45gt replaced
secret/daemon-set-controller-token-skxtk replaced
secret/default-token-plx8s replaced
secret/deployment-controller-token-77g4q replaced
secret/disruption-controller-token-mw4lq replaced
secret/endpoint-controller-token-b5dqn replaced
secret/endpointslice-controller-token-26rcq replaced
secret/endpointslicemirroring-controller-token-zgjgs replaced
secret/ephemeral-volume-controller-token-f9njw replaced
secret/expand-controller-token-2k6v2 replaced
secret/fabric-etcd-secrets replaced
secret/fabric-node-serviceaccount-token-7b6hj replaced
secret/generic-garbage-collector-token-48bvm replaced
secret/horizontal-pod-autoscaler-token-ls2v4 replaced
secret/job-controller-token-nl87g replaced
secret/kube-proxy-token-xp7gj replaced
secret/namespace-controller-token-wdrs5 replaced
secret/node-controller-token-9gs5c replaced
secret/persistent-volume-binder-token-qnclk replaced
secret/pod-garbage-collector-token-c2k7v replaced
secret/pv-protection-controller-token-slkm5 replaced
secret/pvc-protection-controller-token-wnxbw replaced
secret/replicaset-controller-token-4pkz4 replaced
secret/replication-controller-token-vw268 replaced
secret/resourcequota-controller-token-8g9ql replaced
secret/root-ca-cert-publisher-token-fkzln replaced
secret/service-account-controller-token-9686g replaced
secret/service-controller-token-wc49g replaced
secret/statefulset-controller-token-jxcp8 replaced
secret/token-cleaner-token-rp76n replaced
secret/ttl-after-finished-controller-token-9rgmc replaced
secret/ttl-controller-token-59wtj replaced
secret/webhook-server-cert replaced
[centos@ml-k8s-1 etcd]$
secret/attachdetach-controller-token-xfvx9 replaced
secret/bootstrap-signer-token-4z576 replaced
secret/bootstrap-token-rliyei replaced
secret/certificate-controller-token-6g4hj replaced
secret/clusterrole-aggregation-controller-token-vzn58 replaced
secret/coredns-token-bkmbn replaced
secret/cronjob-controller-token-x45gt replaced
secret/daemon-set-controller-token-skxtk replaced
secret/default-token-plx8s replaced
secret/deployment-controller-token-77g4q replaced
secret/disruption-controller-token-mw4lq replaced
secret/endpoint-controller-token-b5dqn replaced
secret/endpointslice-controller-token-26rcq replaced
secret/endpointslicemirroring-controller-token-zgjgs replaced
secret/ephemeral-volume-controller-token-f9njw replaced
secret/expand-controller-token-2k6v2 replaced
secret/fabric-etcd-secrets replaced
secret/fabric-node-serviceaccount-token-7b6hj replaced
secret/generic-garbage-collector-token-48bvm replaced
secret/horizontal-pod-autoscaler-token-ls2v4 replaced
secret/job-controller-token-nl87g replaced
secret/kube-proxy-token-xp7gj replaced
secret/namespace-controller-token-wdrs5 replaced
secret/node-controller-token-9gs5c replaced
secret/persistent-volume-binder-token-qnclk replaced
secret/pod-garbage-collector-token-c2k7v replaced
secret/pv-protection-controller-token-slkm5 replaced
secret/pvc-protection-controller-token-wnxbw replaced
secret/replicaset-controller-token-4pkz4 replaced
secret/replication-controller-token-vw268 replaced
secret/resourcequota-controller-token-8g9ql replaced
secret/root-ca-cert-publisher-token-fkzln replaced
secret/service-account-controller-token-9686g replaced
secret/service-controller-token-wc49g replaced
secret/statefulset-controller-token-jxcp8 replaced
secret/token-cleaner-token-rp76n replaced
secret/ttl-after-finished-controller-token-9rgmc replaced
secret/ttl-controller-token-59wtj replaced
secret/webhook-server-cert replaced

5 解密所有 secrets

修改配置文件 encrypt.conf ，把 provider 里的 identity 修改为第一项

[centos@ml-k8s-1 pki]$ sudo vim encrypt.conf
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - identity: {}
    - aescbc:
        keys:
        - name: key1
          secret: 7unrE0P6q9DmlqGVeZ+k02d2eNaAgveFC01V+gV3VYc=

再通过更新来解密

[centos@ml-k8s-1 pki]$ kubectl get secrets --all-namespaces -o json | kubectl replace -f -
secret/default-token-44r9r replaced
secret/secret1 replaced
secret/default-token-d49sg replaced
secret/default-token-dhpj7 replaced
secret/attachdetach-controller-token-xfvx9 replaced
secret/bootstrap-signer-token-4z576 replaced
secret/bootstrap-token-rliyei replaced
secret/certificate-controller-token-6g4hj replaced
secret/clusterrole-aggregation-controller-token-vzn58 replaced
secret/coredns-token-bkmbn replaced
secret/cronjob-controller-token-x45gt replaced
secret/daemon-set-controller-token-skxtk replaced
secret/default-token-plx8s replaced
secret/deployment-controller-token-77g4q replaced
secret/disruption-controller-token-mw4lq replaced
secret/endpoint-controller-token-b5dqn replaced
secret/endpointslice-controller-token-26rcq replaced
secret/endpointslicemirroring-controller-token-zgjgs replaced
secret/ephemeral-volume-controller-token-f9njw replaced
secret/expand-controller-token-2k6v2 replaced
secret/fabric-etcd-secrets replaced
secret/fabric-node-serviceaccount-token-7b6hj replaced
secret/generic-garbage-collector-token-48bvm replaced
secret/horizontal-pod-autoscaler-token-ls2v4 replaced
secret/job-controller-token-nl87g replaced
secret/kube-proxy-token-xp7gj replaced
secret/namespace-controller-token-wdrs5 replaced
secret/node-controller-token-9gs5c replaced
secret/persistent-volume-binder-token-qnclk replaced
secret/pod-garbage-collector-token-c2k7v replaced
secret/pv-protection-controller-token-slkm5 replaced
secret/pvc-protection-controller-token-wnxbw replaced
secret/replicaset-controller-token-4pkz4 replaced
secret/replication-controller-token-vw268 replaced
secret/resourcequota-controller-token-8g9ql replaced
secret/root-ca-cert-publisher-token-fkzln replaced
secret/service-account-controller-token-9686g replaced
secret/service-controller-token-wc49g replaced
secret/statefulset-controller-token-jxcp8 replaced
secret/token-cleaner-token-rp76n replaced
secret/ttl-after-finished-controller-token-9rgmc replaced
secret/ttl-controller-token-59wtj replaced
secret/webhook-server-cert replaced