GCDW运行在k8s上，本文介绍docker运行环境的安装，特别是containerd服务的配置，与harbor镜像集群的交互证书的配置等。
目录导航
上一步
安装docker
如果没有docker yum源，可以参考上一步的操作系统环境准备文章。
yum install docker-ce docker-ce-cli containerd.io docker-compose-plugin
[root@vm246 172.16.3.246]# yum install docker-ce docker-ce-cli containerd.io docker-compose-plugin
Loaded plugins: fastestmirror, langpacks
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Loading mirror speeds from cached hostfile
* base: mirrors.bfsu.edu.cn
* extras: mirrors.bfsu.edu.cn
* updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package containerd.io.x86_64 0:1.6.21-3.1.el7 will be installed
--> Processing Dependency: container-selinux >= 2:2.74 for package: containerd.io-1.6.21-3.1.el7.x86_64
---> Package docker-ce.x86_64 3:24.0.2-1.el7 will be installed
--> Processing Dependency: docker-ce-rootless-extras for package: 3:docker-ce-24.0.2-1.el7.x86_64
---> Package docker-ce-cli.x86_64 1:24.0.2-1.el7 will be installed
--> Processing Dependency: docker-buildx-plugin for package: 1:docker-ce-cli-24.0.2-1.el7.x86_64
---> Package docker-compose-plugin.x86_64 0:2.18.1-1.el7 will be installed
--> Running transaction check
---> Package container-selinux.noarch 2:2.119.2-1.911c772.el7_8 will be installed
---> Package docker-buildx-plugin.x86_64 0:0.10.5-1.el7 will be installed
---> Package docker-ce-rootless-extras.x86_64 0:24.0.2-1.el7 will be installed
--> Processing Dependency: fuse-overlayfs >= 0.7 for package: docker-ce-rootless-extras-24.0.2-1.el7.x86_64
--> Processing Dependency: slirp4netns >= 0.4 for package: docker-ce-rootless-extras-24.0.2-1.el7.x86_64
--> Running transaction check
---> Package fuse-overlayfs.x86_64 0:0.7.2-6.el7_8 will be installed
--> Processing Dependency: libfuse3.so.3(FUSE_3.2)(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
--> Processing Dependency: libfuse3.so.3(FUSE_3.0)(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
--> Processing Dependency: libfuse3.so.3()(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
---> Package slirp4netns.x86_64 0:0.4.3-4.el7_8 will be installed
--> Running transaction check
---> Package fuse3-libs.x86_64 0:3.6.1-4.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=====================================================================================================================================================================
Package Arch Version Repository Size
=====================================================================================================================================================================
Installing:
containerd.io x86_64 1.6.21-3.1.el7 docker-ce-stable 34 M
docker-ce x86_64 3:24.0.2-1.el7 docker-ce-stable 24 M
docker-ce-cli x86_64 1:24.0.2-1.el7 docker-ce-stable 13 M
docker-compose-plugin x86_64 2.18.1-1.el7 docker-ce-stable 12 M
Installing for dependencies:
container-selinux noarch 2:2.119.2-1.911c772.el7_8 extras 40 k
docker-buildx-plugin x86_64 0.10.5-1.el7 docker-ce-stable 12 M
docker-ce-rootless-extras x86_64 24.0.2-1.el7 docker-ce-stable 9.1 M
fuse-overlayfs x86_64 0.7.2-6.el7_8 extras 54 k
fuse3-libs x86_64 3.6.1-4.el7 extras 82 k
slirp4netns x86_64 0.4.3-4.el7_8 extras 81 k
Transaction Summary
=====================================================================================================================================================================
Install 4 Packages (+6 Dependent packages)
Total download size: 105 M
Installed size: 372 M
Is this ok [y/d/N]: y
Downloading packages:
(1/10): container-selinux-2.119.2-1.911c772.el7_8.noarch.rpm | 40 kB 00:00:00
warning: /var/cache/yum/x86_64/7/docker-ce-stable/packages/docker-buildx-plugin-0.10.5-1.el7.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY6 ETA
Public key for docker-buildx-plugin-0.10.5-1.el7.x86_64.rpm is not installed
(2/10): docker-buildx-plugin-0.10.5-1.el7.x86_64.rpm | 12 MB 00:00:07
(3/10): containerd.io-1.6.21-3.1.el7.x86_64.rpm | 34 MB 00:00:18
(4/10): docker-ce-24.0.2-1.el7.x86_64.rpm | 24 MB 00:00:13
(5/10): docker-ce-rootless-extras-24.0.2-1.el7.x86_64.rpm | 9.1 MB 00:00:04
(6/10): fuse3-libs-3.6.1-4.el7.x86_64.rpm | 82 kB 00:00:00
(7/10): fuse-overlayfs-0.7.2-6.el7_8.x86_64.rpm | 54 kB 00:00:00
(8/10): slirp4netns-0.4.3-4.el7_8.x86_64.rpm | 81 kB 00:00:00
(9/10): docker-ce-cli-24.0.2-1.el7.x86_64.rpm | 13 MB 00:00:07
(10/10): docker-compose-plugin-2.18.1-1.el7.x86_64.rpm | 12 MB 00:00:05
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 3.4 MB/s | 105 MB 00:00:31
Retrieving key from https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
Importing GPG key 0x621E9F35:
Userid : "Docker Release (CE rpm) <docker@docker.com>"
Fingerprint: 060a 61c5 1b55 8a7f 742b 77aa c52f eb6b 621e 9f35
From : https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : 2:container-selinux-2.119.2-1.911c772.el7_8.noarch 1/10
Installing : containerd.io-1.6.21-3.1.el7.x86_64 2/10
Installing : docker-buildx-plugin-0.10.5-1.el7.x86_64 3/10
Installing : slirp4netns-0.4.3-4.el7_8.x86_64 4/10
Installing : fuse3-libs-3.6.1-4.el7.x86_64 5/10
Installing : fuse-overlayfs-0.7.2-6.el7_8.x86_64 6/10
Installing : docker-compose-plugin-2.18.1-1.el7.x86_64 7/10
Installing : 1:docker-ce-cli-24.0.2-1.el7.x86_64 8/10
Installing : docker-ce-rootless-extras-24.0.2-1.el7.x86_64 9/10
Installing : 3:docker-ce-24.0.2-1.el7.x86_64 10/10
Verifying : 3:docker-ce-24.0.2-1.el7.x86_64 1/10
Verifying : docker-compose-plugin-2.18.1-1.el7.x86_64 2/10
Verifying : fuse3-libs-3.6.1-4.el7.x86_64 3/10
Verifying : fuse-overlayfs-0.7.2-6.el7_8.x86_64 4/10
Verifying : containerd.io-1.6.21-3.1.el7.x86_64 5/10
Verifying : slirp4netns-0.4.3-4.el7_8.x86_64 6/10
Verifying : 2:container-selinux-2.119.2-1.911c772.el7_8.noarch 7/10
Verifying : 1:docker-ce-cli-24.0.2-1.el7.x86_64 8/10
Verifying : docker-ce-rootless-extras-24.0.2-1.el7.x86_64 9/10
Verifying : docker-buildx-plugin-0.10.5-1.el7.x86_64 10/10
Installed:
containerd.io.x86_64 0:1.6.21-3.1.el7 docker-ce.x86_64 3:24.0.2-1.el7 docker-ce-cli.x86_64 1:24.0.2-1.el7 docker-compose-plugin.x86_64 0:2.18.1-1.el7
Dependency Installed:
container-selinux.noarch 2:2.119.2-1.911c772.el7_8 docker-buildx-plugin.x86_64 0:0.10.5-1.el7 docker-ce-rootless-extras.x86_64 0:24.0.2-1.el7
fuse-overlayfs.x86_64 0:0.7.2-6.el7_8 fuse3-libs.x86_64 0:3.6.1-4.el7 slirp4netns.x86_64 0:0.4.3-4.el7_8
Complete!
[root@vm246 172.16.3.246]#
安装docker-compose
下载docker-compose-linux-x86_64 并改名mv到/usr/local/bin/docker-compose
注意下版本。我这里都是v2.18.1，和docker的版本一致
文件50M,网速不行的，可以只在1个节点下载，然后分发到别的。
[root@vm246 172.16.3.246]# wget https://github.com/docker/compose/releases/download/v2.18.1/docker-compose-linux-x86_64
--2023-06-15 18:26:54-- https://github.com/docker/compose/releases/download/v2.18.1/docker-compose-linux-x86_64
Resolving github.com (github.com)... 20.205.243.166
Connecting to github.com (github.com)|20.205.243.166|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/15045751/ebe621cd-2d6b-4306-b81c-eedc1b74e4da?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230616%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230616T012655Z&X-Amz-Expires=300&X-Amz-Signature=bbb81932fba33ad38be588df03770e70da5aff4ea444931750dd8320c591717d&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=15045751&response-content-disposition=attachment%3B%20filename%3Ddocker-compose-linux-x86_64&response-content-type=application%2Foctet-stream [following]
--2023-06-15 18:26:55-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/15045751/ebe621cd-2d6b-4306-b81c-eedc1b74e4da?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230616%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230616T012655Z&X-Amz-Expires=300&X-Amz-Signature=bbb81932fba33ad38be588df03770e70da5aff4ea444931750dd8320c591717d&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=15045751&response-content-disposition=attachment%3B%20filename%3Ddocker-compose-linux-x86_64&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.110.133, 185.199.109.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 54537935 (52M) [application/octet-stream]
Saving to: ‘docker-compose-linux-x86_64’
100%[===========================================================================================================================>] 54,537,935 13.0KB/s in 35m 22s
2023-06-15 19:02:18 (25.1 KB/s) - ‘docker-compose-linux-x86_64’ saved [54537935/54537935]
[root@vm246 172.16.3.246]#
[root@vm246 172.16.3.246]# chmod a+x docker-compose-linux-x86_64
[root@vm246 172.16.3.246]# ./docker-compose-linux-x86_64 -v
Docker Compose version v2.18.1
[root@vm246 172.16.3.246]#
[root@vm246 172.16.3.246]# mv docker-compose-linux-x86_64 /usr/local/bin/docker-compose
[root@vm246 172.16.3.246]# docker-compose -v
Docker Compose version v2.18.1
[root@vm246 172.16.3.246]#
查看版本
[root@vm246 ~]# docker compose version
Docker Compose version v2.18.1
[root@vm246 ~]#修改docker的镜像服务器配置
其中https://172.16.3.249:8443是harbor镜像服务器的地址。
[root@k8s-81 ~]# cat /etc/docker/daemon.json
{
"registry-mirrors": [
"https://registry.docker-cn.com",
"http://hub-mirror.c.163.com",
"https://docker.mirrors.ustc.edu.cn",
"https://172.16.3.249:8443"
],
"insecure-registries": [
],
"log-opts": {
"max-size": "10m"
}
}
将harbor的证书复制过来
mkdir -p /etc/docker/certs.d
scp -r 172.16.3.249:/etc/docker/certs.d/172.16.3.249\:8443 /etc/docker/certs.d/
[root@vm246 172.16.3.246]# vi /etc/docker/daemon.json
[root@vm246 172.16.3.246]# mkdir -p /etc/docker/certs.d
[root@vm246 172.16.3.246]# scp -r 172.16.3.249:/etc/docker/certs.d/172.16.3.249\:8443 /etc/docker/certs.d/
The authenticity of host '172.16.3.249 (172.16.3.249)' can't be established.
ECDSA key fingerprint is SHA256:Xs1gi6NKPEsAxLRIL2NHIv7jG1vt68oBlWZ0YUe/Swk.
ECDSA key fingerprint is MD5:b4:9c:dd:e1:3c:42:28:8d:db:c5:a0:73:30:2f:60:78.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.3.249' (ECDSA) to the list of known hosts.
root@172.16.3.249's password:
172.16.3.249.cert 100% 2053 1.4MB/s 00:00
172.16.3.249.crt 100% 2053 2.1MB/s 00:00
172.16.3.249.key 100% 3247 3.8MB/s 00:00
ca.crt 100% 2029 2.2MB/s 00:00
[root@vm246 172.16.3.246]
[root@vm246 ~]# ll /etc/docker/
total 4
drwxr-xr-x. 3 root root 31 Jun 15 19:09 certs.d
-rw-r--r--. 1 root root 274 Jun 15 19:07 daemon.json
[root@vm246 ~]# ll /etc/docker/certs.d/
total 0
drwxr-xr-x. 2 root root 93 Jun 15 19:09 172.16.3.249:8443
[root@vm246 ~]# ll /etc/docker/certs.d/172.16.3.249\:8443/
total 16
-rw-r--r--. 1 root root 2053 Jun 15 19:09 172.16.3.249.cert
-rw-r--r--. 1 root root 2053 Jun 15 19:09 172.16.3.249.crt
-rw-r--r--. 1 root root 3247 Jun 15 19:09 172.16.3.249.key
-rw-r--r--. 1 root root 2029 Jun 15 19:09 ca.crt
[root@vm246 ~]#
修改containerd的配置
将docker访问harbor的证书复制一份
cp -r /etc/docker/certs.d/172.16.3.249\:8443/ /etc/containerd/
[root@vm246 ~]# ll /etc/containerd/
total 4
-rw-r--r--. 1 root root 886 May 5 13:20 config.toml
[root@vm246 ~]# cp -r /etc/docker/certs.d/172.16.3.249\:8443/ /etc/containerd/
[root@vm246 ~]# ll /etc/containerd/
total 4
drwxr-xr-x. 2 root root 93 Jun 15 19:11 172.16.3.249:8443
-rw-r--r--. 1 root root 886 May 5 13:20 config.toml
[root@vm246 ~]# ll /etc/containerd/172.16.3.249\:8443/
total 16
-rw-r--r--. 1 root root 2053 Jun 15 19:11 172.16.3.249.cert
-rw-r--r--. 1 root root 2053 Jun 15 19:11 172.16.3.249.crt
-rw-r--r--. 1 root root 3247 Jun 15 19:11 172.16.3.249.key
-rw-r--r--. 1 root root 2029 Jun 15 19:11 ca.crt
[root@vm246 ~]#
生成默认配置文件
containerd config default > /etc/containerd/config.toml修改配置文件
将containerd配置文件里面的pause:2.6的镜像，改成能用的，比如. 注意项目名称(如下例子是library）。镜像服务器证书配置看后面。
以前的
sandbox_image = "registry.k8s.io/pause:3.6"
替换成
sandbox_image = "172.16.3.249:8443/library/pause:3.6"
如上的pause:3.6如果不想自己上传到harbor,也可以从其它仓库下载，比如
[root@vm248 ~]# ctr -n k8s.io i pull registry.aliyuncs.com/google_containers/pause:3.6
registry.aliyuncs.com/google_containers/pause:3.6: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:c2280d2f5f56cf9c9a01bb64b2db4651e35efd6d62a54dcfc12049fe6449c5e4: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:fbe1a72f5dcd08ba4ca3ce3468c742786c1f6578c1f6bb401be1c4620d6ff705: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 1.6 s total: 5.1 Ki (3.2 KiB/s)
unpacking linux/amd64 sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db...
done: 61.92735ms
[root@vm248 ~]# ctr -n k8s.io i tag registry.aliyuncs.com/google_containers/pause:3.6 registry.k8s.io/pause:3.6
registry.k8s.io/pause:3.6修改服务IP
以前的
stream_server_address = "127.0.0.1"
改成， 具体IP已每个节点的实际IP为准
stream_server_address = "10.0.2.81"下图中为修改后示意，具体以实际IP为准
配置镜像服务器证书，包括如下2个部分，分别是镜像地址和证书文件位置
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = ""
[plugins."io.containerd.grpc.v1.cri".registry.auths]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."172.16.3.249:8443".tls]
ca_file = "/etc/containerd/172.16.3.249:8443/ca.crt"
[plugins."io.containerd.grpc.v1.cri".registry.headers]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."172.16.3.249:8443"]
endpoint = ["https://172.16.3.249:8443"]systemd配置,从false改成true
plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options
...
SystemdCgroup = true完整的配置文件例子，红色是我改动过的部分
[root@k8s-82 containerd]# cat config.toml
disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "/var/lib/containerd"
state = "/run/containerd"
temp = ""
version = 2
[cgroup]
path = ""
[debug]
address = ""
format = ""
gid = 0
level = ""
uid = 0
[grpc]
address = "/run/containerd/containerd.sock"
gid = 0
max_recv_message_size = 16777216
max_send_message_size = 16777216
tcp_address = ""
tcp_tls_ca = ""
tcp_tls_cert = ""
tcp_tls_key = ""
uid = 0
[metrics]
address = ""
grpc_histogram = false
[plugins]
[plugins."io.containerd.gc.v1.scheduler"]
deletion_threshold = 0
mutation_threshold = 100
pause_threshold = 0.02
schedule_delay = "0s"
startup_delay = "100ms"
[plugins."io.containerd.grpc.v1.cri"]
device_ownership_from_security_context = false
disable_apparmor = false
disable_cgroup = false
disable_hugetlb_controller = true
disable_proc_mount = false
disable_tcp_service = true
enable_selinux = false
enable_tls_streaming = false
enable_unprivileged_icmp = false
enable_unprivileged_ports = false
ignore_image_defined_volumes = false
max_concurrent_downloads = 3
max_container_log_line_size = 16384
netns_mounts_under_state_dir = false
restrict_oom_score_adj = false
sandbox_image = "172.16.3.249:8443/library/pause:3.6"
selinux_category_range = 1024
stats_collect_period = 10
stream_idle_timeout = "4h0m0s"
stream_server_address = "10.0.2.82"
stream_server_port = "0"
systemd_cgroup = false
tolerate_missing_hugetlb_controller = true
unset_seccomp_profile = ""
[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"
conf_template = ""
ip_pref = ""
max_conf_num = 1
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
disable_snapshot_annotations = true
discard_unpacked_layers = false
ignore_rdt_not_enabled_errors = false
no_pivot = false
snapshotter = "overlayfs"
[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
base_runtime_spec = ""
cni_conf_dir = ""
cni_max_conf_num = 0
container_annotations = []
pod_annotations = []
privileged_without_host_devices = false
runtime_engine = ""
runtime_path = ""
runtime_root = ""
runtime_type = ""
[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
base_runtime_spec = ""
cni_conf_dir = ""
cni_max_conf_num = 0
container_annotations = []
pod_annotations = []
privileged_without_host_devices = false
runtime_engine = ""
runtime_path = ""
runtime_root = ""
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
BinaryName = ""
CriuImagePath = ""
CriuPath = ""
CriuWorkPath = ""
IoGid = 0
IoUid = 0
NoNewKeyring = false
NoPivotRoot = false
Root = ""
ShimCgroup = ""
SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
base_runtime_spec = ""
cni_conf_dir = ""
cni_max_conf_num = 0
container_annotations = []
pod_annotations = []
privileged_without_host_devices = false
runtime_engine = ""
runtime_path = ""
runtime_root = ""
runtime_type = ""
[plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]
[plugins."io.containerd.grpc.v1.cri".image_decryption]
key_model = "node"
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = ""
[plugins."io.containerd.grpc.v1.cri".registry.auths]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."172.16.3.249:8443".tls]
ca_file = "/etc/containerd/172.16.3.249:8443/ca.crt"
[plugins."io.containerd.grpc.v1.cri".registry.headers]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."172.16.3.249:8443"]
endpoint = ["https://172.16.3.249:8443"]
[plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
tls_cert_file = ""
tls_key_file = ""
[plugins."io.containerd.internal.v1.opt"]
path = "/opt/containerd"
[plugins."io.containerd.internal.v1.restart"]
interval = "10s"
[plugins."io.containerd.internal.v1.tracing"]
sampling_ratio = 1.0
service_name = "containerd"
[plugins."io.containerd.metadata.v1.bolt"]
content_sharing_policy = "shared"
[plugins."io.containerd.monitor.v1.cgroups"]
no_prometheus = false
[plugins."io.containerd.runtime.v1.linux"]
no_shim = false
runtime = "runc"
runtime_root = ""
shim = "containerd-shim"
shim_debug = false
[plugins."io.containerd.runtime.v2.task"]
platforms = ["linux/amd64"]
sched_core = false
[plugins."io.containerd.service.v1.diff-service"]
default = ["walking"]
[plugins."io.containerd.service.v1.tasks-service"]
rdt_config_file = ""
[plugins."io.containerd.snapshotter.v1.aufs"]
root_path = ""
[plugins."io.containerd.snapshotter.v1.btrfs"]
root_path = ""
[plugins."io.containerd.snapshotter.v1.devmapper"]
async_remove = false
base_image_size = ""
discard_blocks = false
fs_options = ""
fs_type = ""
pool_name = ""
root_path = ""
[plugins."io.containerd.snapshotter.v1.native"]
root_path = ""
[plugins."io.containerd.snapshotter.v1.overlayfs"]
root_path = ""
upperdir_label = false
[plugins."io.containerd.snapshotter.v1.zfs"]
root_path = ""
[plugins."io.containerd.tracing.processor.v1.otlp"]
endpoint = ""
insecure = false
protocol = ""
[proxy_plugins]
[stream_processors]
[stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
path = "ctd-decoder"
returns = "application/vnd.oci.image.layer.v1.tar"
[stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
path = "ctd-decoder"
returns = "application/vnd.oci.image.layer.v1.tar+gzip"
[timeouts]
"io.containerd.timeout.bolt.open" = "0s"
"io.containerd.timeout.shim.cleanup" = "5s"
"io.containerd.timeout.shim.load" = "5s"
"io.containerd.timeout.shim.shutdown" = "3s"
"io.containerd.timeout.task.state" = "2s"
[ttrpc]
address = ""
gid = 0
uid = 0
[root@k8s-82 containerd]#生成 CNI配置文件/etc/cni/net.d/10-containerd-net.conflist
cat << EOF | tee /etc/cni/net.d/10-containerd-net.conflist
{
"cniVersion": "1.0.0",
"name": "containerd-net",
"plugins": [
{
"type": "bridge",
"bridge": "cni0",
"isGateway": true,
"ipMasq": true,
"promiscMode": true,
"ipam": {
"type": "host-local",
"ranges": [
[{
"subnet": "10.88.0.0/16"
}],
[{
"subnet": "2001:db8:4860::/64"
}]
],
"routes": [
{ "dst": "0.0.0.0/0" },
{ "dst": "::/0" }
]
}
},
{
"type": "portmap",
"capabilities": {"portMappings": true},
"externalSetMarkChain": "KUBE-MARK-MASQ"
}
]
}
EOF配置开机自启动
systemctl enable containerd
systemctl enable docker
[root@vm246 ~]# systemctl enable containerd
Created symlink from /etc/systemd/system/multi-user.target.wants/containerd.service to /usr/lib/systemd/system/containerd.service.
[root@vm246 ~]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
[root@vm246 ~]#
重启服务
systemctl daemon-reload
systemctl restart containerd
systemctl restart docker检查各个服务的status,能正常启动，且没有报错为准。 特别是镜像拉取部分。一般包含关键字images
重点是containerd服务。
systemctl status containerd -l
systemctl status docker -l成功标记
[root@vm246 ~]# systemctl status containerd -l
● containerd.service - containerd container runtime
Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2023-06-15 19:23:29 PDT; 22s ago
Docs: https://containerd.io
Process: 122442 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Main PID: 122447 (containerd)
Tasks: 9
Memory: 12.5M
CGroup: /system.slice/containerd.service
└─122447 /usr/bin/containerd
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.875312841-07:00" level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.876437032-07:00" level=info msg="Start subscribing containerd event"
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.876571680-07:00" level=info msg="Start recovering state"
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.876731048-07:00" level=info msg="Start event monitor"
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.876776079-07:00" level=info msg="Start snapshots syncer"
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.876804634-07:00" level=info msg="Start cni network conf syncer for default"
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.876824365-07:00" level=info msg="Start streaming server"
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.880223372-07:00" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.880340135-07:00" level=info msg=serving... address=/run/containerd/containerd.sock
Jun 15 19:23:29 vm246 containerd[122447]: time="2023-06-15T19:23:29.880507655-07:00" level=info msg="containerd successfully booted in 0.045285s"
[root@vm246 ~]# systemctl status docker -l
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2023-06-15 19:23:31 PDT; 47s ago
Docs: https://docs.docker.com
Main PID: 122475 (dockerd)
Tasks: 8
Memory: 23.9M
CGroup: /system.slice/docker.service
└─122475 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
Jun 15 19:23:31 vm246 systemd[1]: Starting Docker Application Container Engine...
Jun 15 19:23:31 vm246 dockerd[122475]: time="2023-06-15T19:23:31.090256047-07:00" level=info msg="Starting up"
Jun 15 19:23:31 vm246 dockerd[122475]: time="2023-06-15T19:23:31.128884422-07:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Jun 15 19:23:31 vm246 dockerd[122475]: time="2023-06-15T19:23:31.129218516-07:00" level=info msg="Loading containers: start."
Jun 15 19:23:31 vm246 dockerd[122475]: time="2023-06-15T19:23:31.294765086-07:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Jun 15 19:23:31 vm246 dockerd[122475]: time="2023-06-15T19:23:31.358672366-07:00" level=info msg="Loading containers: done."
Jun 15 19:23:31 vm246 dockerd[122475]: time="2023-06-15T19:23:31.381460228-07:00" level=info msg="Docker daemon" commit=659604f graphdriver=overlay2 version=24.0.2
Jun 15 19:23:31 vm246 dockerd[122475]: time="2023-06-15T19:23:31.381616108-07:00" level=info msg="Daemon has completed initialization"
Jun 15 19:23:31 vm246 dockerd[122475]: time="2023-06-15T19:23:31.418241862-07:00" level=info msg="API listen on /run/docker.sock"
Jun 15 19:23:31 vm246 systemd[1]: Started Docker Application Container Engine.
[root@vm246 ~]#
查看服务日志
journalctl -u docker -f -n 200
-- Logs begin at Fri 2023-06-16 15:32:21 CST. --
Jun 16 16:23:29 vm246 systemd[1]: Starting Docker Application Container Engine...
Jun 16 16:23:29 vm246 dockerd[14222]: time="2023-06-16T01:23:29.903461113-07:00" level=info msg="Starting up"
Jun 16 16:23:29 vm246 dockerd[14222]: time="2023-06-16T01:23:29.975099107-07:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Jun 16 16:23:29 vm246 dockerd[14222]: time="2023-06-16T01:23:29.976284285-07:00" level=info msg="Loading containers: start."
Jun 16 16:23:30 vm246 dockerd[14222]: time="2023-06-16T01:23:30.269691401-07:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Jun 16 16:23:30 vm246 dockerd[14222]: time="2023-06-16T01:23:30.337563745-07:00" level=info msg="Loading containers: done."
Jun 16 16:23:30 vm246 dockerd[14222]: time="2023-06-16T01:23:30.362873451-07:00" level=info msg="Docker daemon" commit=659604f graphdriver=overlay2 version=24.0.2
Jun 16 16:23:30 vm246 dockerd[14222]: time="2023-06-16T01:23:30.363081732-07:00" level=info msg="Daemon has completed initialization"
Jun 16 16:23:30 vm246 dockerd[14222]: time="2023-06-16T01:23:30.403721175-07:00" level=info msg="API listen on /run/docker.sock"
Jun 16 16:23:30 vm246 systemd[1]: Started Docker Application Container Engine.
Jun 16 16:34:48 vm246 systemd[1]: Stopping Docker Application Container Engine...
Jun 16 16:34:48 vm246 dockerd[14222]: time="2023-06-16T01:34:48.779962200-07:00" level=info msg="Processing signal 'terminated'"
Jun 16 16:34:48 vm246 dockerd[14222]: time="2023-06-16T01:34:48.784279762-07:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Jun 16 16:34:48 vm246 dockerd[14222]: time="2023-06-16T01:34:48.785022718-07:00" level=info msg="Daemon shutdown complete"
Jun 16 16:34:48 vm246 systemd[1]: Stopped Docker Application Container Engine.
Jun 16 16:42:28 vm246 systemd[1]: Starting Docker Application Container Engine...
Jun 16 16:42:28 vm246 dockerd[16056]: time="2023-06-16T01:42:28.776106585-07:00" level=info msg="Starting up"
Jun 16 16:42:28 vm246 dockerd[16056]: time="2023-06-16T01:42:28.807766565-07:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Jun 16 16:42:28 vm246 dockerd[16056]: time="2023-06-16T01:42:28.808052262-07:00" level=info msg="Loading containers: start."
Jun 16 16:42:28 vm246 dockerd[16056]: time="2023-06-16T01:42:28.982890894-07:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Jun 16 16:42:29 vm246 dockerd[16056]: time="2023-06-16T01:42:29.043206290-07:00" level=info msg="Loading containers: done."
Jun 16 16:42:29 vm246 dockerd[16056]: time="2023-06-16T01:42:29.065665867-07:00" level=info msg="Docker daemon" commit=659604f graphdriver=overlay2 version=24.0.2
Jun 16 16:42:29 vm246 dockerd[16056]: time="2023-06-16T01:42:29.065775807-07:00" level=info msg="Daemon has completed initialization"
Jun 16 16:42:29 vm246 dockerd[16056]: time="2023-06-16T01:42:29.109587449-07:00" level=info msg="API listen on /run/docker.sock"
Jun 16 16:42:29 vm246 systemd[1]: Started Docker Application Container Engine.