南大通用GCDW技术栈,镜像仓库harbor安装,包括https 服务CA证书的生成(openssl)

GCDW是GBase 的云数仓版本(GBase Cloud Database Warehouse), 其运行在k8s上,镜像要先上传到镜像仓库,在安装部署时各个节点再从镜像仓库同时拉取。本文介绍harbor安装配置方法,特别是启用https服务的方法。

harbor需要容器环境

yum install docker-ce docker-ce-cli containerd.io docker-compose-plugin

还需要docker-compose,这个直接下载,就是个可执行文件,改名到/usr/local/bin下就可以了

[root@vm249 harbor]# ll /usr/local/bin/docker-compose
-rwxr-xr-x. 1 root root 44953600 Feb 21 00:58 /usr/local/bin/docker-compose
[root@vm249 harbor]#

生成CA证书

如果不开通https服务,可以直接跳到后面部分

生成CA私钥

[root@vm249 ssl]# openssl genrsa -out ca.key 4096
Generating RSA private key, 4096 bit long modulus
.........................................++
........................................................................................................................................++
e is 65537 (0x10001)
[root@vm249 ssl]# ll
total 4
-rw-r--r--. 1 root root 3247 Feb 21 17:23 ca.key
[root@vm249 ssl]#

生成CA证书 Generate the CA certificate.

openssl req -x509 -new -nodes -sha512 -days 3650 \
 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=yourdomain.com" \
 -key ca.key \
 -out ca.crt

其中

  • -subj 指定组织名称等,如果使用FQDN连接Harbor主机,则必须将其指定为通用名称(CN)属性,比如你的主机名或者域名等。
  • -key 是前面生成的CA私钥
  • -out 是生成的CA证书名字
openssl req -x509 -new -nodes -sha512 -days 3650 \
 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=172.16.3.249" \
 -key ca.key \
 -out ca.crt

生成服务器端证书Generate a Server Certificate

生成私钥

openssl genrsa -out yourdomain.com.key 4096

其中域名是yourdomain.com, 下面你是用IP的例子

[root@vm249 ssl]# openssl genrsa -out 172.16.3.249.key 4096
Generating RSA private key, 4096 bit long modulus
................................................++
.++
e is 65537 (0x10001)
[root@vm249 ssl]# ll
total 12
-rw-r--r--. 1 root root 3247 Feb 21 17:49 172.16.3.249.key
-rw-r--r--. 1 root root 2029 Feb 21 17:33 ca.crt
-rw-r--r--. 1 root root 3247 Feb 21 17:23 ca.key
[root@vm249 ssl]#

生成CSR Generate a certificate signing request (CSR).

openssl req -sha512 -new \
    -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=yourdomain.com" \
    -key yourdomain.com.key \
    -out yourdomain.com.csr

如下是用IP作为主机名

[root@vm249 ssl]# openssl req -sha512 -new \
>     -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=172.16.3.249" \
>     -key 172.16.3.249.key \
>     -out 172.16.3.249.csr
[root@vm249 ssl]# ll
total 16
-rw-r--r--. 1 root root 1704 Feb 21 18:10 172.16.3.249.csr
-rw-r--r--. 1 root root 3247 Feb 21 17:49 172.16.3.249.key
-rw-r--r--. 1 root root 2029 Feb 21 17:33 ca.crt
-rw-r--r--. 1 root root 3247 Feb 21 17:23 ca.key
[root@vm249 ssl]#

生成X509 v3的扩展文件 Generate an x509 v3 extension file.

包括域名和IP方式的主机名,用于harbor编译时指定了SAN和X509 v3(Harbor host that complies with the Subject Alternative Name (SAN) and x509 v3 extension requirements.)

cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1=yourdomain.com
DNS.2=yourdomain
DNS.3=hostname
EOF

如果用IP,则需要在subjectAltName部分直接写IP

[root@vm249 ssl]# cat > v3.ext <<-EOF
> authorityKeyIdentifier=keyid,issuer
> basicConstraints=CA:FALSE
> keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
> extendedKeyUsage = serverAuth
> subjectAltName = IP:172.16.3.249
> EOF
[root@vm249 ssl]# ll
total 20
-rw-r--r--. 1 root root 1704 Feb 21 18:10 172.16.3.249.csr
-rw-r--r--. 1 root root 3247 Feb 21 17:49 172.16.3.249.key
-rw-r--r--. 1 root root 2029 Feb 21 17:33 ca.crt
-rw-r--r--. 1 root root 3247 Feb 21 17:23 ca.key
-rw-r--r--. 1 root root  231 Feb 21 18:32 v3.ext
[root@vm249 ssl]# cat v3.ext
authorityKeyIdentifier=keyid,issuer
[root@vm249 ssl]# cat v3.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = IP:172.16.3.249

[root@vm249 ssl]#

用v3.ext生成harbor主机的证书 Use the v3.ext file to generate a certificate for your Harbor host.

openssl x509 -req -sha512 -days 3650 \
    -extfile v3.ext \
    -CA ca.crt -CAkey ca.key -CAcreateserial \
    -in yourdomain.com.csr \
    -out yourdomain.com.crt

如下是使用IP的例子

openssl x509 -req -sha512 -days 3650 \
    -extfile v3.ext \
    -CA ca.crt -CAkey ca.key -CAcreateserial \
    -in 172.16.3.249.csr \
    -out 172.16.3.249.crt

[root@vm249 ssl]# openssl x509 -req -sha512 -days 3650 \
>     -extfile v3.ext \
>     -CA ca.crt -CAkey ca.key -CAcreateserial \
>     -in 172.16.3.249.csr \
>     -out 172.16.3.249.crt
Signature ok
subject=/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=172.16.3.249
Getting CA Private Key
[root@vm249 ssl]#
[root@vm249 ssl]# ll
total 28
-rw-r--r--. 1 root root 2061 Feb 21 18:34 172.16.3.249.crt
-rw-r--r--. 1 root root 1704 Feb 21 18:10 172.16.3.249.csr
-rw-r--r--. 1 root root 3247 Feb 21 17:49 172.16.3.249.key
-rw-r--r--. 1 root root 2029 Feb 21 17:33 ca.crt
-rw-r--r--. 1 root root 3247 Feb 21 17:23 ca.key
-rw-r--r--. 1 root root   17 Feb 21 18:34 ca.srl
-rw-r--r--. 1 root root  231 Feb 21 18:32 v3.ext
[root@vm249 ssl]#

将证书发给容器和harbor

将服务器端证书和私钥复制到harbor主机

Copy the server certificate and key into the certficates folder on your Harbor host.

cp yourdomain.com.crt /data/cert/
cp yourdomain.com.key /data/cert/

如下是IP的操作记录。 目录如果不存在,可以创建上。

[root@vm249 ssl]# mkdir -p /data/cert
[root@vm249 ssl]# cp 172.16.3.249.crt  /data/cert/
[root@vm249 ssl]# cp 172.16.3.249.key  /data/cert/
[root@vm249 ssl]# ll /data/cert/
total 8
-rw-r--r--. 1 root root 2061 Feb 21 19:13 172.16.3.249.crt
-rw-r--r--. 1 root root 3247 Feb 21 19:13 172.16.3.249.key
[root@vm249 ssl]#

将crt转化为cert,提供给docker使用

Convert yourdomain.com.crt to yourdomain.com.cert, for use by Docker.

The Docker daemon interprets .crt files as CA certificates and .cert files as client certificates.

docker进程将crt作为CA证书, cert作为客户端证书。

openssl x509 -inform PEM -in yourdomain.com.crt -out yourdomain.com.cert

运行记录

[root@vm249 ssl]# openssl x509 -inform PEM -in 172.16.3.249.crt -out 172.16.3.249.cert
[root@vm249 ssl]# ll
total 32
-rw-r--r--. 1 root root 2061 Feb 21 19:17 172.16.3.249.cert
-rw-r--r--. 1 root root 2061 Feb 21 18:34 172.16.3.249.crt
-rw-r--r--. 1 root root 1704 Feb 21 18:10 172.16.3.249.csr
-rw-r--r--. 1 root root 3247 Feb 21 17:49 172.16.3.249.key
-rw-r--r--. 1 root root 2029 Feb 21 17:33 ca.crt
-rw-r--r--. 1 root root 3247 Feb 21 17:23 ca.key
-rw-r--r--. 1 root root   17 Feb 21 18:34 ca.srl
-rw-r--r--. 1 root root  231 Feb 21 18:32 v3.ext
[root@vm249 ssl]#

将服务器端证书,私钥和CA文件,复制到docker的harbor主机目录下。

Copy the server certificate, key and CA files into the Docker certificates folder on the Harbor host. You must create the appropriate folders first.

注意,默认是443端口,如果不是默认值,则后面创建的目录,要带上端口号。 比如有从172.16.3.249,变成172.16.3.249:8443

cp yourdomain.com.cert /etc/docker/certs.d/yourdomain.com/
cp yourdomain.com.key /etc/docker/certs.d/yourdomain.com/
cp ca.crt /etc/docker/certs.d/yourdomain.com/

运行记录

[root@vm249 ssl]# mkdir -p /etc/docker/certs.d/172.16.3.249/
[root@vm249 ssl]# cp 172.16.3.249.cert /etc/docker/certs.d/172.16.3.249/
[root@vm249 ssl]# cp 172.16.3.249.key /etc/docker/certs.d/172.16.3.249/
[root@vm249 ssl]# cp ca.crt /etc/docker/certs.d/172.16.3.249/
[root@vm249 ssl]# ll /etc/docker/certs.d/172.16.3.249/
total 12
-rw-r--r--. 1 root root 2061 Feb 21 19:20 172.16.3.249.cert
-rw-r--r--. 1 root root 3247 Feb 21 19:20 172.16.3.249.key
-rw-r--r--. 1 root root 2029 Feb 21 19:20 ca.crt
[root@vm249 ssl]#

如果修改了SSL端口

如果不是默认的443,则前面创建的目录,要加上端口,比如。 并且所有的CA文件也是要放到这个目录下。

[root@vm249 ssl]# mkdir /etc/docker/certs.d/172.16.3.249:8443

重启容器docker

systemctl restart docker

下载harbor

https://goharbor.io/

如果网速快,可以选择offline的版本。 我这里用的是online的。

修改harbor支持https

解压缩后,将harbor目录下的harbor.yml.tmp复制一份为harbor.yml

修改配置文件

包括 hostname 和 https 部分,其中证书和私钥正确配置。 如果不开通https服务,可以注释掉相关部分。 注意配置文件缩进格式,不要出现无效的额外空格等,否则会造成解析错误。

[root@vm249 ssl]# cat ../harbor.yml
# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: 172.16.3.249

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 8088

# https related config
https:
  # https port for harbor, default is 443
  port: 8443
  # The path of cert and key files for nginx
  certificate: /data/cert/172.16.3.249.crt
  private_key: /data/cert/172.16.3.249.key
。。。。。。

启动服务

如果没有运行在http模式下,可以直接install.sh安装

./install.sh

如果已经部署了http模式,可以重新部署

./prepare
docker-compose down -v
docker-compose up -d

执行记录

[root@vm249 harbor]# ./prepare
prepare base dir is set to /root/harbor
Clearing the configuration file: /config/portal/nginx.conf
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/core/env
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/registry/passwd
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/db/env
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/jobservice/config.yml
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /data/secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir
[root@vm249 harbor]# ll -rt
total 56
-rwxr-xr-x. 1 root root  1881 Dec 14 20:29 prepare
-rwxr-xr-x. 1 root root  3171 Dec 14 20:29 install.sh
-rw-r--r--. 1 root root 11567 Dec 14 20:29 harbor.yml.tmpl
-rw-r--r--. 1 root root  3639 Dec 14 20:29 common.sh
-rw-r--r--. 1 root root 11347 Dec 14 20:29 LICENSE
drwxr-xr-x. 3 root root    20 Feb 21 00:33 common
drwxr-xr-x. 2 root root   159 Feb 21 19:17 ssl
-rw-r--r--. 1 root root 11576 Feb 21 21:42 harbor.yml
-rw-r--r--. 1 root root  5947 Feb 21 21:47 docker-compose.yml
[root@vm249 harbor]# systemctl daemon-reload
[root@vm249 harbor]# docker-compose down -v^C
[root@vm249 harbor]# systemctl restart docker
[root@vm249 harbor]# docker-compose down -v
[+] Running 10/10
 ⠿ Container harbor-jobservice  Removed                                                                                                                        11.3s
 ⠿ Container registryctl        Removed                                                                                                                        10.4s
 ⠿ Container nginx              Removed                                                                                                                         1.3s
 ⠿ Container harbor-portal      Removed                                                                                                                         0.3s
 ⠿ Container harbor-core        Removed                                                                                                                         3.4s
 ⠿ Container harbor-db          Removed                                                                                                                         1.0s
 ⠿ Container registry           Removed                                                                                                                         0.9s
 ⠿ Container redis              Removed                                                                                                                         1.0s
 ⠿ Container harbor-log         Removed                                                                                                                        10.3s
 ⠿ Network harbor_harbor        Removed                                                                                                                         0.2s
[root@vm249 harbor]# docker-compose down -v^C
[root@vm249 harbor]# docker-compose up -d
[+] Running 10/10
 ⠿ Network harbor_harbor        Created                                                                                                                         0.3s
 ⠿ Container harbor-log         Started                                                                                                                         2.2s
 ⠿ Container registry           Started                                                                                                                         5.4s
 ⠿ Container harbor-db          Started                                                                                                                         6.2s
 ⠿ Container registryctl        Started                                                                                                                         5.9s
 ⠿ Container harbor-portal      Started                                                                                                                         6.1s
 ⠿ Container redis              Started                                                                                                                         5.8s
 ⠿ Container harbor-core        Started                                                                                                                         6.8s
 ⠿ Container nginx              Started                                                                                                                         9.2s
 ⠿ Container harbor-jobservice  Started                                                                                                                         9.1s
[root@vm249 harbor]#

测试效果

如果访问的是8088端口,也会自动转到8443的https。

默认访问密码,可以从harbor.yml里看到,搜索: harbor_admin_password

增加https的docker daemon.json配置

[root@vm249 harbor]# vi /etc/docker/daemon.json
[root@vm249 harbor]# cat /etc/docker/daemon.json
{
 "registry-mirrors": [
    "https://registry.docker-cn.com",
    "http://hub-mirror.c.163.com",
    "https://docker.mirrors.ustc.edu.cn",
    "https://172.16.3.249:8443"
  ],
  "insecure-registries": [   
  ],

  "log-opts": {
              "max-size": "10m"
            }
}

[root@vm249 harbor]#
[root@vm249 harbor]# systemctl daemon-reload
[root@vm249 harbor]# systemctl restart docker
[root@vm249 harbor]# systemctl restart harbor
[root@vm249 harbor]# docker login 172.16.3.249:8443
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[root@vm249 harbor]#

其它节点远程访问harbor的方法

需要将所需的/etc/docker/cert.d下对应的地址的证书复制过去。 其中的daemon.json,根据需要进行修改。 如果目录不存在,可以提前创建。

docker login

docker访问harbor需要先login,输入用户名和密码。 默认会将登录信息保存到/root/.docker/config.json.

[root@k8s-81 ~]# docker login https://172.16.3.249:8443
Username: admin
Password:
Error response from daemon: Get "https://172.16.3.249:8443/v2/": x509: certificate signed by unknown authority
[root@k8s-81 ~]#

[root@k8s-81 ~]# docker login https://172.16.3.249:8443
Username: admin
Password:
Error response from daemon: Get "https://172.16.3.249:8443/v2/": x509: certificate signed by unknown authority
[root@k8s-81 ~]# cd /etc/docker/
[root@k8s-81 docker]# ll
total 0
[root@k8s-81 docker]# scp -r 172.16.3.249:/etc/docker/cert.d/172.16.3.249:8443 ./certd.d/
The authenticity of host '172.16.3.249 (172.16.3.249)' can't be established.
ECDSA key fingerprint is SHA256:Xs1gi6NKPEsAxLRIL2NHIv7jG1vt68oBlWZ0YUe/Swk.
ECDSA key fingerprint is MD5:b4:9c:dd:e1:3c:42:28:8d:db:c5:a0:73:30:2f:60:78.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.3.249' (ECDSA) to the list of known hosts.
root@172.16.3.249's password:
Permission denied, please try again.
root@172.16.3.249's password:
172.16.3.249.cert                                                                               100% 2053    36.5KB/s   00:00
172.16.3.249.crt                                                                                100% 2053    44.7KB/s   00:00
172.16.3.249.key                                                                                100% 3247   597.2KB/s   00:00
ca.crt                                                                                          100% 2029    29.0KB/s   00:00
daemon.json                                                                                     100%  276    42.9KB/s   00:00
[root@k8s-81 docker]# ll
total 4
drwxr-xr-x. 3 root root  31 Feb 23 08:59 certs.d
-rw-r--r--. 1 root root 276 Feb 23 08:59 daemon.json
[root@k8s-81 docker]# cat daemon.json
{
 "registry-mirrors": [
    "https://registry.docker-cn.com",
    "http://hub-mirror.c.163.com",
    "https://docker.mirrors.ustc.edu.cn",
    "https://172.16.3.249:8443"
  ],
  "insecure-registries": [
  ],

  "log-opts": {
              "max-size": "10m"
            }
}

[root@k8s-81 docker]# ll
total 4
drwxr-xr-x. 3 root root  31 Feb 23 08:59 certs.d
-rw-r--r--. 1 root root 276 Feb 23 08:59 daemon.json
[root@k8s-81 docker]# cd certs.d/
[root@k8s-81 certs.d]# ll
total 0
drwxr-xr-x. 2 root root 93 Feb 23 08:59 172.16.3.249:8443
[root@k8s-81 certs.d]# systemctl daemon-reload
[root@k8s-81 certs.d]# docker login https://172.16.3.249:8443
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[root@k8s-81 certs.d]# ll
total 0
drwxr-xr-x. 2 root root 93 Feb 23 08:59 172.16.3.249:8443
[root@k8s-81 certs.d]# cd ..
[root@k8s-81 docker]# ll
total 4
drwxr-xr-x. 3 root root  31 Feb 23 08:59 certs.d
-rw-r--r--. 1 root root 276 Feb 23 08:59 daemon.json
[root@k8s-81 docker]#

完整安装记录

     ┌────────────────────────────────────────────────────────────────────┐
     │                        • MobaXterm 10.4 •                          │
     │            (SSH client, X-server and networking tools)             │
     │                                                                    │
     │ ➤ SSH session to root@172.16.3.249                                 │
     │   • SSH compression : ✔                                            │
     │   • SSH-browser     : ✔                                            │
     │   • X11-forwarding  : ✔  (remote display is forwarded through SSH) │
     │   • DISPLAY         : ✔  (automatically set on remote server)      │
     │                                                                    │
     │ ➤ For more info, ctrl+click on help or visit our website           │
     └────────────────────────────────────────────────────────────────────┘

Last login: Mon Jun 26 02:53:46 2023 from 172.16.35.241
[root@vm249 ~]# docker
bash: docker: command not found...
[root@vm249 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
[root@vm249 ~]# vi /etc/hosts
[root@vm249 ~]# yum install docker-ce docker-ce-cli containerd.io docker-compose-plugin
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.nju.edu.cn
 * extras: mirrors.bfsu.edu.cn
 * updates: mirrors.bfsu.edu.cn
base                                                                                                                                          | 3.6 kB  00:00:00
extras                                                                                                                                        | 2.9 kB  00:00:00
updates                                                                                                                                       | 2.9 kB  00:00:00
updates/7/x86_64/primary_db                                                                                                                   |  22 MB  00:00:09
No package docker-ce available.
No package docker-ce-cli available.
No package containerd.io available.
No package docker-compose-plugin available.
Error: Nothing to do
[root@vm249 ~]# curl -o /etc/yum.repos.d/Centos-7.repo http://mirrors.aliyun.com/repo/Centos-7.repo
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
        http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2523  100  2523    0     0  10864      0 --:--:-- --:--:-- --:--:-- 10922
[root@vm249 ~]# curl -o /etc/yum.repos.d/docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2081  100  2081    0     0  11384      0 --:--:-- --:--:-- --:--:-- 11434
[root@vm249 ~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
> [kubernetes]
> name=Kubernetes
> baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
> enabled=1
> gpgcheck=0
> repo_gpgcheck=0
> gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
>         http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
> EOF
[root@vm249 ~]# yum clean all && yum makecache
Loaded plugins: fastestmirror, langpacks
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Cleaning repos: base docker-ce-stable extras kubernetes updates
Cleaning up list of fastest mirrors
Loaded plugins: fastestmirror, langpacks
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Determining fastest mirrors
 * base: mirrors.bfsu.edu.cn
 * extras: mirrors.bfsu.edu.cn
 * updates: mirrors.bfsu.edu.cn
base                                                                                                                                          | 3.6 kB  00:00:00
docker-ce-stable                                                                                                                              | 3.5 kB  00:00:00
extras                                                                                                                                        | 2.9 kB  00:00:00
kubernetes                                                                                                                                    | 1.4 kB  00:00:00
updates                                                                                                                                       | 2.9 kB  00:00:00
(1/17): docker-ce-stable/7/x86_64/filelists_db                                                                                                |  45 kB  00:00:00
(2/17): docker-ce-stable/7/x86_64/updateinfo                                                                                                  |   55 B  00:00:00
(3/17): base/7/x86_64/group_gz                                                                                                                | 153 kB  00:00:00
(4/17): docker-ce-stable/7/x86_64/primary_db                                                                                                  | 111 kB  00:00:00
(5/17): extras/7/x86_64/filelists_db                                                                                                          | 276 kB  00:00:00
(6/17): docker-ce-stable/7/x86_64/other_db                                                                                                    | 133 kB  00:00:00
(7/17): extras/7/x86_64/primary_db                                                                                                            | 249 kB  00:00:00
(8/17): extras/7/x86_64/other_db                                                                                                              | 149 kB  00:00:00
(9/17): kubernetes/filelists                                                                                                                  |  43 kB  00:00:00
(10/17): base/7/x86_64/filelists_db                                                                                                           | 7.2 MB  00:00:04
(11/17): kubernetes/primary                                                                                                                   | 132 kB  00:00:02
(12/17): kubernetes/other                                                                                                                     |  86 kB  00:00:03
(13/17): base/7/x86_64/other_db                                                                                                               | 2.6 MB  00:00:08
(14/17): updates/7/x86_64/primary_db                                                                                                          |  22 MB  00:00:12
(15/17): updates/7/x86_64/filelists_db                                                                                                        |  12 MB  00:00:14
(16/17): updates/7/x86_64/other_db                                                                                                            | 1.4 MB  00:00:18
(17/17): base/7/x86_64/primary_db                                                                                                             | 6.1 MB  00:01:01
kubernetes                                                                                                                                                   980/980
kubernetes                                                                                                                                                   980/980
kubernetes                                                                                                                                                   980/980
Metadata Cache Created
[root@vm249 ~]# yum install docker-ce docker-ce-cli containerd.io docker-compose-plugin
Loaded plugins: fastestmirror, langpacks
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Loading mirror speeds from cached hostfile
 * base: mirrors.bfsu.edu.cn
 * extras: mirrors.bfsu.edu.cn
 * updates: mirrors.bfsu.edu.cn
Resolving Dependencies
--> Running transaction check
---> Package containerd.io.x86_64 0:1.6.21-3.1.el7 will be installed
--> Processing Dependency: container-selinux >= 2:2.74 for package: containerd.io-1.6.21-3.1.el7.x86_64
---> Package docker-ce.x86_64 3:24.0.2-1.el7 will be installed
--> Processing Dependency: docker-ce-rootless-extras for package: 3:docker-ce-24.0.2-1.el7.x86_64
---> Package docker-ce-cli.x86_64 1:24.0.2-1.el7 will be installed
--> Processing Dependency: docker-buildx-plugin for package: 1:docker-ce-cli-24.0.2-1.el7.x86_64
---> Package docker-compose-plugin.x86_64 0:2.18.1-1.el7 will be installed
--> Running transaction check
---> Package container-selinux.noarch 2:2.119.2-1.911c772.el7_8 will be installed
---> Package docker-buildx-plugin.x86_64 0:0.10.5-1.el7 will be installed
---> Package docker-ce-rootless-extras.x86_64 0:24.0.2-1.el7 will be installed
--> Processing Dependency: fuse-overlayfs >= 0.7 for package: docker-ce-rootless-extras-24.0.2-1.el7.x86_64
--> Processing Dependency: slirp4netns >= 0.4 for package: docker-ce-rootless-extras-24.0.2-1.el7.x86_64
--> Running transaction check
---> Package fuse-overlayfs.x86_64 0:0.7.2-6.el7_8 will be installed
--> Processing Dependency: libfuse3.so.3(FUSE_3.2)(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
--> Processing Dependency: libfuse3.so.3(FUSE_3.0)(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
--> Processing Dependency: libfuse3.so.3()(64bit) for package: fuse-overlayfs-0.7.2-6.el7_8.x86_64
---> Package slirp4netns.x86_64 0:0.4.3-4.el7_8 will be installed
--> Running transaction check
---> Package fuse3-libs.x86_64 0:3.6.1-4.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=====================================================================================================================================================================
 Package                                       Arch                       Version                                         Repository                            Size
=====================================================================================================================================================================
Installing:
 containerd.io                                 x86_64                     1.6.21-3.1.el7                                  docker-ce-stable                      34 M
 docker-ce                                     x86_64                     3:24.0.2-1.el7                                  docker-ce-stable                      24 M
 docker-ce-cli                                 x86_64                     1:24.0.2-1.el7                                  docker-ce-stable                      13 M
 docker-compose-plugin                         x86_64                     2.18.1-1.el7                                    docker-ce-stable                      12 M
Installing for dependencies:
 container-selinux                             noarch                     2:2.119.2-1.911c772.el7_8                       extras                                40 k
 docker-buildx-plugin                          x86_64                     0.10.5-1.el7                                    docker-ce-stable                      12 M
 docker-ce-rootless-extras                     x86_64                     24.0.2-1.el7                                    docker-ce-stable                     9.1 M
 fuse-overlayfs                                x86_64                     0.7.2-6.el7_8                                   extras                                54 k
 fuse3-libs                                    x86_64                     3.6.1-4.el7                                     extras                                82 k
 slirp4netns                                   x86_64                     0.4.3-4.el7_8                                   extras                                81 k

Transaction Summary
=====================================================================================================================================================================
Install  4 Packages (+6 Dependent packages)

Total download size: 105 M
Installed size: 372 M
Is this ok [y/d/N]: y
Downloading packages:
(1/10): container-selinux-2.119.2-1.911c772.el7_8.noarch.rpm                                                                                  |  40 kB  00:00:00
warning: /var/cache/yum/x86_64/7/docker-ce-stable/packages/docker-buildx-plugin-0.10.5-1.el7.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 621e9f35: NOKEY0 ETA
Public key for docker-buildx-plugin-0.10.5-1.el7.x86_64.rpm is not installed
(2/10): docker-buildx-plugin-0.10.5-1.el7.x86_64.rpm                                                                                          |  12 MB  00:00:43
(3/10): containerd.io-1.6.21-3.1.el7.x86_64.rpm                                                                                               |  34 MB  00:02:11
(4/10): docker-ce-24.0.2-1.el7.x86_64.rpm                                                                                                     |  24 MB  00:01:42
(5/10): docker-ce-rootless-extras-24.0.2-1.el7.x86_64.rpm                                                                                     | 9.1 MB  00:00:40
(6/10): fuse-overlayfs-0.7.2-6.el7_8.x86_64.rpm                                                                                               |  54 kB  00:00:00
(7/10): fuse3-libs-3.6.1-4.el7.x86_64.rpm                                                                                                     |  82 kB  00:00:00
(8/10): docker-ce-cli-24.0.2-1.el7.x86_64.rpm                                                                                                 |  13 MB  00:00:57
slirp4netns-0.4.3-4.el7_8.x86_ FAILED                                          ==================================================  ] 276 kB/s | 102 MB  00:00:11 ETA
http://mirror.lzu.edu.cn/centos/7.9.2009/extras/x86_64/Packages/slirp4netns-0.4.3-4.el7_8.x86_64.rpm: [Errno 12] Timeout on http://mirror.lzu.edu.cn/centos/7.9.2009/extras/x86_64/Packages/slirp4netns-0.4.3-4.el7_8.x86_64.rpm: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
Trying other mirror.
(9/10): slirp4netns-0.4.3-4.el7_8.x86_64.rpm                                                                                                  |  81 kB  00:00:00
(10/10): docker-compose-plugin-2.18.1-1.el7.x86_64.rpm                                                                                        |  12 MB  00:00:45
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                463 kB/s | 105 MB  00:03:51
Retrieving key from https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
Importing GPG key 0x621E9F35:
 Userid     : "Docker Release (CE rpm) <docker@docker.com>"
 Fingerprint: 060a 61c5 1b55 8a7f 742b 77aa c52f eb6b 621e 9f35
 From       : https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : 2:container-selinux-2.119.2-1.911c772.el7_8.noarch                                                                                               1/10
  Installing : containerd.io-1.6.21-3.1.el7.x86_64                                                                                                              2/10
  Installing : docker-buildx-plugin-0.10.5-1.el7.x86_64                                                                                                         3/10
  Installing : slirp4netns-0.4.3-4.el7_8.x86_64                                                                                                                 4/10
  Installing : fuse3-libs-3.6.1-4.el7.x86_64                                                                                                                    5/10
  Installing : fuse-overlayfs-0.7.2-6.el7_8.x86_64                                                                                                              6/10
  Installing : docker-compose-plugin-2.18.1-1.el7.x86_64                                                                                                        7/10
  Installing : 1:docker-ce-cli-24.0.2-1.el7.x86_64                                                                                                              8/10
  Installing : docker-ce-rootless-extras-24.0.2-1.el7.x86_64                                                                                                    9/10
  Installing : 3:docker-ce-24.0.2-1.el7.x86_64                                                                                                                 10/10
  Verifying  : 3:docker-ce-24.0.2-1.el7.x86_64                                                                                                                  1/10
  Verifying  : docker-compose-plugin-2.18.1-1.el7.x86_64                                                                                                        2/10
  Verifying  : fuse3-libs-3.6.1-4.el7.x86_64                                                                                                                    3/10
  Verifying  : fuse-overlayfs-0.7.2-6.el7_8.x86_64                                                                                                              4/10
  Verifying  : containerd.io-1.6.21-3.1.el7.x86_64                                                                                                              5/10
  Verifying  : slirp4netns-0.4.3-4.el7_8.x86_64                                                                                                                 6/10
  Verifying  : 2:container-selinux-2.119.2-1.911c772.el7_8.noarch                                                                                               7/10
  Verifying  : 1:docker-ce-cli-24.0.2-1.el7.x86_64                                                                                                              8/10
  Verifying  : docker-ce-rootless-extras-24.0.2-1.el7.x86_64                                                                                                    9/10
  Verifying  : docker-buildx-plugin-0.10.5-1.el7.x86_64                                                                                                        10/10

Installed:
  containerd.io.x86_64 0:1.6.21-3.1.el7    docker-ce.x86_64 3:24.0.2-1.el7    docker-ce-cli.x86_64 1:24.0.2-1.el7    docker-compose-plugin.x86_64 0:2.18.1-1.el7

Dependency Installed:
  container-selinux.noarch 2:2.119.2-1.911c772.el7_8        docker-buildx-plugin.x86_64 0:0.10.5-1.el7        docker-ce-rootless-extras.x86_64 0:24.0.2-1.el7
  fuse-overlayfs.x86_64 0:0.7.2-6.el7_8                     fuse3-libs.x86_64 0:3.6.1-4.el7                   slirp4netns.x86_64 0:0.4.3-4.el7_8

Complete!
[root@vm249 ~]# yum install systemd-resolved
Loaded plugins: fastestmirror, langpacks
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Loading mirror speeds from cached hostfile
 * base: mirrors.bfsu.edu.cn
 * extras: mirrors.bfsu.edu.cn
 * updates: mirrors.bfsu.edu.cn
Resolving Dependencies
--> Running transaction check
---> Package systemd-resolved.x86_64 0:219-78.el7_9.7 will be installed
--> Processing Dependency: systemd = 219-78.el7_9.7 for package: systemd-resolved-219-78.el7_9.7.x86_64
--> Running transaction check
---> Package systemd.x86_64 0:219-78.el7 will be updated
--> Processing Dependency: systemd = 219-78.el7 for package: systemd-python-219-78.el7.x86_64
--> Processing Dependency: systemd = 219-78.el7 for package: systemd-sysv-219-78.el7.x86_64
---> Package systemd.x86_64 0:219-78.el7_9.7 will be an update
--> Processing Dependency: systemd-libs = 219-78.el7_9.7 for package: systemd-219-78.el7_9.7.x86_64
--> Running transaction check
---> Package systemd-libs.x86_64 0:219-78.el7 will be updated
--> Processing Dependency: systemd-libs = 219-78.el7 for package: libgudev1-219-78.el7.x86_64
---> Package systemd-libs.x86_64 0:219-78.el7_9.7 will be an update
---> Package systemd-python.x86_64 0:219-78.el7 will be updated
---> Package systemd-python.x86_64 0:219-78.el7_9.7 will be an update
---> Package systemd-sysv.x86_64 0:219-78.el7 will be updated
---> Package systemd-sysv.x86_64 0:219-78.el7_9.7 will be an update
--> Running transaction check
---> Package libgudev1.x86_64 0:219-78.el7 will be updated
---> Package libgudev1.x86_64 0:219-78.el7_9.7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=====================================================================================================================================================================
 Package                                     Arch                              Version                                      Repository                          Size
=====================================================================================================================================================================
Installing:
 systemd-resolved                            x86_64                            219-78.el7_9.7                               updates                            422 k
Updating for dependencies:
 libgudev1                                   x86_64                            219-78.el7_9.7                               updates                            110 k
 systemd                                     x86_64                            219-78.el7_9.7                               updates                            5.1 M
 systemd-libs                                x86_64                            219-78.el7_9.7                               updates                            419 k
 systemd-python                              x86_64                            219-78.el7_9.7                               updates                            146 k
 systemd-sysv                                x86_64                            219-78.el7_9.7                               updates                             97 k

Transaction Summary
=====================================================================================================================================================================
Install  1 Package
Upgrade             ( 5 Dependent packages)

Total download size: 6.2 M
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for updates
(1/6): libgudev1-219-78.el7_9.7.x86_64.rpm                                                                                                    | 110 kB  00:00:00
(2/6): systemd-python-219-78.el7_9.7.x86_64.rpm                                                                                               | 146 kB  00:00:00
(3/6): systemd-sysv-219-78.el7_9.7.x86_64.rpm                                                                                                 |  97 kB  00:00:00
(4/6): systemd-resolved-219-78.el7_9.7.x86_64.rpm                                                                                             | 422 kB  00:00:00
(5/6): systemd-219-78.el7_9.7.x86_64.rpm                                                                                                      | 5.1 MB  00:00:02
systemd-libs-219-78.el7_9.7.x8 FAILED
http://ftp.ksu.edu.tw/pub/CentOS/7.9.2009/updates/x86_64/Packages/systemd-libs-219-78.el7_9.7.x86_64.rpm: [Errno 12] Timeout on http://ftp.ksu.edu.tw/pub/CentOS/7.9.2009/updates/x86_64/Packages/systemd-libs-219-78.el7_9.7.x86_64.rpm: (28, 'Connection timed out after 30001 milliseconds')
Trying other mirror.
(6/6): systemd-libs-219-78.el7_9.7.x86_64.rpm                                                                                                 | 419 kB  00:00:00
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                211 kB/s | 6.2 MB  00:00:30
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : systemd-libs-219-78.el7_9.7.x86_64                                                                                                               1/11
  Updating   : systemd-219-78.el7_9.7.x86_64                                                                                                                    2/11
  Updating   : systemd-sysv-219-78.el7_9.7.x86_64                                                                                                               3/11
  Installing : systemd-resolved-219-78.el7_9.7.x86_64                                                                                                           4/11
  Updating   : systemd-python-219-78.el7_9.7.x86_64                                                                                                             5/11
  Updating   : libgudev1-219-78.el7_9.7.x86_64                                                                                                                  6/11
  Cleanup    : systemd-sysv-219-78.el7.x86_64                                                                                                                   7/11
  Cleanup    : systemd-python-219-78.el7.x86_64                                                                                                                 8/11
  Cleanup    : systemd-219-78.el7.x86_64                                                                                                                        9/11
  Cleanup    : libgudev1-219-78.el7.x86_64                                                                                                                     10/11
  Cleanup    : systemd-libs-219-78.el7.x86_64                                                                                                                  11/11
  Verifying  : systemd-libs-219-78.el7_9.7.x86_64                                                                                                               1/11
  Verifying  : systemd-219-78.el7_9.7.x86_64                                                                                                                    2/11
  Verifying  : systemd-sysv-219-78.el7_9.7.x86_64                                                                                                               3/11
  Verifying  : systemd-resolved-219-78.el7_9.7.x86_64                                                                                                           4/11
  Verifying  : systemd-python-219-78.el7_9.7.x86_64                                                                                                             5/11
  Verifying  : libgudev1-219-78.el7_9.7.x86_64                                                                                                                  6/11
  Verifying  : systemd-python-219-78.el7.x86_64                                                                                                                 7/11
  Verifying  : systemd-sysv-219-78.el7.x86_64                                                                                                                   8/11
  Verifying  : systemd-libs-219-78.el7.x86_64                                                                                                                   9/11
  Verifying  : libgudev1-219-78.el7.x86_64                                                                                                                     10/11
  Verifying  : systemd-219-78.el7.x86_64                                                                                                                       11/11

Installed:
  systemd-resolved.x86_64 0:219-78.el7_9.7

Dependency Updated:
  libgudev1.x86_64 0:219-78.el7_9.7        systemd.x86_64 0:219-78.el7_9.7     systemd-libs.x86_64 0:219-78.el7_9.7     systemd-python.x86_64 0:219-78.el7_9.7
  systemd-sysv.x86_64 0:219-78.el7_9.7

Complete!
[root@vm249 ~]# swapoff -a
[root@vm249 ~]# vi /etc/fstab
[root@vm249 ~]# setenforce 0
[root@vm249 ~]# sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
[root@vm249 ~]# systemctl stop firewalld
[root@vm249 ~]# systemctl disable firewalld
[root@vm249 ~]# cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
> overlay
> br_netfilter
> EOF
overlay
br_netfilter
[root@vm249 ~]#
[root@vm249 ~]# sudo modprobe overlay
[root@vm249 ~]# sudo modprobe br_netfilter
[root@vm249 ~]# cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
> net.bridge.bridge-nf-call-iptables  = 1
> net.bridge.bridge-nf-call-ip6tables = 1
> net.ipv4.ip_forward                 = 1
> EOF
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
[root@vm249 ~]#
[root@vm249 ~]# # 应用 sysctl 参数而不重新启动
[root@vm249 ~]# sudo sysctl --system
* Applying /usr/lib/sysctl.d/00-system.conf ...
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
kernel.yama.ptrace_scope = 0
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.sysrq = 16
kernel.core_uses_pid = 1
kernel.kptr_restrict = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.all.promote_secondaries = 1
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /usr/lib/sysctl.d/60-libvirtd.conf ...
fs.aio-max-nr = 1048576
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.d/k8s.conf ...
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
* Applying /etc/sysctl.conf ...
[root@vm249 ~]# openssl genrsa -out ca.key 4096
Generating RSA private key, 4096 bit long modulus
..............................................................................................................++
......................................................................................++
e is 65537 (0x10001)
[root@vm249 ~]# ll
total 16
-rw-------. 1 root root 4682 Feb  8 12:16 anaconda-ks.cfg
-rw-r--r--. 1 root root 3243 Jun 26 16:59 ca.key
-rw-------. 1 root root 3711 Feb  8 12:16 original-ks.cfg
drwxr-xr-x. 3 root root   49 Jun 25 20:04 software
[root@vm249 ~]# openssl req -x509 -new -nodes -sha512 -days 3650 \
>  -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=172.16.3.249" \
>  -key ca.key \
>  -out ca.crt
[root@vm249 ~]# ll
total 20
-rw-------. 1 root root 4682 Feb  8 12:16 anaconda-ks.cfg
-rw-r--r--. 1 root root 2029 Jun 26 17:00 ca.crt
-rw-r--r--. 1 root root 3243 Jun 26 16:59 ca.key
-rw-------. 1 root root 3711 Feb  8 12:16 original-ks.cfg
drwxr-xr-x. 3 root root   49 Jun 25 20:04 software
[root@vm249 ~]# openssl genrsa -out 172.16.3.249.key 4096
Generating RSA private key, 4096 bit long modulus
................................++
............................++
e is 65537 (0x10001)
[root@vm249 ~]# openssl req -sha512 -new \
> >     -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=172.16.3.249" \
> >     -key 172.16.3.249.key \
> >     -out 172.16.3.249.csr
unknown option /C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=172.16.3.249
req [options] <infile >outfile
where options  are
 -inform arg    input format - DER or PEM
 -outform arg   output format - DER or PEM
 -in arg        input file
 -out arg       output file
 -text          text form of request
 -pubkey        output public key
 -noout         do not output REQ
 -verify        verify signature on REQ
 -modulus       RSA modulus
 -nodes         don't encrypt the output key
 -engine e      use engine e, possibly a hardware device
 -subject       output the request's subject
 -passin        private key password source
 -key file      use the private key contained in file
 -keyform arg   key file format
 -keyout arg    file to send the key to
 -rand file:file:...
                load the file (or the files in the directory) into
                the random number generator
 -newkey rsa:bits generate a new RSA key of 'bits' in size
 -newkey dsa:file generate a new DSA key, parameters taken from CA in 'file'
 -newkey ec:file generate a new EC key, parameters taken from CA in 'file'
 -[digest]      Digest to sign with (see openssl dgst -h for list)
 -config file   request template file.
 -subj arg      set or modify request subject
 -multivalue-rdn enable support for multivalued RDNs
 -new           new request.
 -batch         do not ask anything during request generation
 -x509          output a x509 structure instead of a cert. req.
 -days          number of days a certificate generated by -x509 is valid for.
 -set_serial    serial number to use for a certificate generated by -x509.
 -newhdr        output "NEW" in the header lines
 -asn1-kludge   Output the 'request' in a format that is wrong but some CA's
                have been reported as requiring
 -extensions .. specify certificate extension section (override value in config file)
 -reqexts ..    specify request extension section (override value in config file)
 -utf8          input characters are UTF8 (default ASCII)
 -nameopt arg    - various certificate name options
 -reqopt arg    - various request text options

[root@vm249 ~]# openssl req -sha512 -new >     -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=172.16.3.249" >     -key 172.16.3.249.key >     -out 172.16.3.249.csr^C
[root@vm249 ~]# openssl req -sha512 -new \
>      -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=172.16.3.249" \
>      -key 172.16.3.249.key \
>      -out 172.16.3.249.csr
[root@vm249 ~]# ll
total 28
-rw-r--r--. 1 root root 1704 Jun 26 17:01 172.16.3.249.csr
-rw-r--r--. 1 root root 3247 Jun 26 17:00 172.16.3.249.key
-rw-------. 1 root root 4682 Feb  8 12:16 anaconda-ks.cfg
-rw-r--r--. 1 root root 2029 Jun 26 17:00 ca.crt
-rw-r--r--. 1 root root 3243 Jun 26 16:59 ca.key
-rw-r--r--. 1 root root    0 Jun 26 17:01 -key
-rw-------. 1 root root 3711 Feb  8 12:16 original-ks.cfg
-rw-r--r--. 1 root root    0 Jun 26 17:01 -out
drwxr-xr-x. 3 root root   49 Jun 25 20:04 software
-rw-r--r--. 1 root root    0 Jun 26 17:01 -subj
[root@vm249 ~]# cat > v3.ext <<-EOF
> authorityKeyIdentifier=keyid,issuer
> basicConstraints=CA:FALSE
> keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
> extendedKeyUsage = serverAuth
> subjectAltName = IP:172.16.3.249
> EOF
[root@vm249 ~]# openssl x509 -req -sha512 -days 3650 \
>     -extfile v3.ext \
>     -CA ca.crt -CAkey ca.key -CAcreateserial \
>     -in 172.16.3.249.csr \
>     -out 172.16.3.249.crt
Signature ok
subject=/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=172.16.3.249
Getting CA Private Key
[root@vm249 ~]# mkdir -p /data/cert
[root@vm249 ~]# cp 172.16.3.249.crt  /data/cert/
[root@vm249 ~]# cp 172.16.3.249.key  /data/cert/
[root@vm249 ~]# ll
total 40
-rw-r--r--. 1 root root 2053 Jun 26 17:02 172.16.3.249.crt
-rw-r--r--. 1 root root 1704 Jun 26 17:01 172.16.3.249.csr
-rw-r--r--. 1 root root 3247 Jun 26 17:00 172.16.3.249.key
-rw-------. 1 root root 4682 Feb  8 12:16 anaconda-ks.cfg
-rw-r--r--. 1 root root 2029 Jun 26 17:00 ca.crt
-rw-r--r--. 1 root root 3243 Jun 26 16:59 ca.key
-rw-r--r--. 1 root root   17 Jun 26 17:02 ca.srl
-rw-r--r--. 1 root root    0 Jun 26 17:01 -key
-rw-------. 1 root root 3711 Feb  8 12:16 original-ks.cfg
-rw-r--r--. 1 root root    0 Jun 26 17:01 -out
drwxr-xr-x. 3 root root   49 Jun 25 20:04 software
-rw-r--r--. 1 root root    0 Jun 26 17:01 -subj
-rw-r--r--. 1 root root  204 Jun 26 17:02 v3.ext
[root@vm249 ~]# openssl x509 -inform PEM -in 172.16.3.249.crt -out 172.16.3.249.cert
[root@vm249 ~]# ll
total 44
-rw-r--r--. 1 root root 2053 Jun 26 17:03 172.16.3.249.cert
-rw-r--r--. 1 root root 2053 Jun 26 17:02 172.16.3.249.crt
-rw-r--r--. 1 root root 1704 Jun 26 17:01 172.16.3.249.csr
-rw-r--r--. 1 root root 3247 Jun 26 17:00 172.16.3.249.key
-rw-------. 1 root root 4682 Feb  8 12:16 anaconda-ks.cfg
-rw-r--r--. 1 root root 2029 Jun 26 17:00 ca.crt
-rw-r--r--. 1 root root 3243 Jun 26 16:59 ca.key
-rw-r--r--. 1 root root   17 Jun 26 17:02 ca.srl
-rw-r--r--. 1 root root    0 Jun 26 17:01 -key
-rw-------. 1 root root 3711 Feb  8 12:16 original-ks.cfg
-rw-r--r--. 1 root root    0 Jun 26 17:01 -out
drwxr-xr-x. 3 root root   49 Jun 25 20:04 software
-rw-r--r--. 1 root root    0 Jun 26 17:01 -subj
-rw-r--r--. 1 root root  204 Jun 26 17:02 v3.ext
[root@vm249 ~]# mkdir -p /etc/docker/certs.d/172.16.3.249/
[root@vm249 ~]# cp 172.16.3.249.cert /etc/docker/certs.d/172.16.3.249/
[root@vm249 ~]# cp 172.16.3.249.key /etc/docker/certs.d/172.16.3.249/
[root@vm249 ~]# cp ca.crt /etc/docker/certs.d/172.16.3.249/
[root@vm249 ~]# ll /etc/docker/certs.d/172.16.3.249/
total 12
-rw-r--r--. 1 root root 2053 Jun 26 17:04 172.16.3.249.cert
-rw-r--r--. 1 root root 3247 Jun 26 17:04 172.16.3.249.key
-rw-r--r--. 1 root root 2029 Jun 26 17:04 ca.crt
[root@vm249 ~]# mkdir /etc/docker/certs.d/172.16.3.249:8443^C
[root@vm249 ~]# cd /etc/docker/certs.d/
[root@vm249 certs.d]# ll
total 0
drwxr-xr-x. 2 root root 69 Jun 26 17:04 172.16.3.249
[root@vm249 certs.d]# mv 172.16.3.249 172.16.3.249:8443
[root@vm249 certs.d]# ll
total 0
drwxr-xr-x. 2 root root 69 Jun 26 17:04 172.16.3.249:8443
[root@vm249 certs.d]# systemctl restart docker
[root@vm249 certs.d]# journalctl  -u docker.service
-- Logs begin at Thu 2023-02-09 06:20:43 PST, end at Mon 2023-06-26 17:05:42 PDT. --
Jun 26 17:05:42 vm249 systemd[1]: Starting Docker Application Container Engine...
Jun 26 17:05:42 vm249 dockerd[129391]: time="2023-06-26T17:05:42.156371597-07:00" level=info msg="Starting up"
Jun 26 17:05:42 vm249 dockerd[129391]: time="2023-06-26T17:05:42.191182327-07:00" level=info msg="Loading containers: start."
Jun 26 17:05:42 vm249 dockerd[129391]: time="2023-06-26T17:05:42.415895163-07:00" level=info msg="Loading containers: done."
Jun 26 17:05:42 vm249 dockerd[129391]: time="2023-06-26T17:05:42.437600858-07:00" level=info msg="Docker daemon" commit=659604f graphdriver=overlay2 version=24.0.2
Jun 26 17:05:42 vm249 dockerd[129391]: time="2023-06-26T17:05:42.438899998-07:00" level=info msg="Daemon has completed initialization"
Jun 26 17:05:42 vm249 dockerd[129391]: time="2023-06-26T17:05:42.478909857-07:00" level=info msg="API listen on /run/docker.sock"
Jun 26 17:05:42 vm249 systemd[1]: Started Docker Application Container Engine.
[root@vm249 certs.d]# journalctl  -u docker.service  -n 100 -f
-- Logs begin at Thu 2023-02-09 06:20:43 PST. --
Jun 26 17:05:42 vm249 systemd[1]: Starting Docker Application Container Engine...
Jun 26 17:05:42 vm249 dockerd[129391]: time="2023-06-26T17:05:42.156371597-07:00" level=info msg="Starting up"
Jun 26 17:05:42 vm249 dockerd[129391]: time="2023-06-26T17:05:42.191182327-07:00" level=info msg="Loading containers: start."
Jun 26 17:05:42 vm249 dockerd[129391]: time="2023-06-26T17:05:42.415895163-07:00" level=info msg="Loading containers: done."
Jun 26 17:05:42 vm249 dockerd[129391]: time="2023-06-26T17:05:42.437600858-07:00" level=info msg="Docker daemon" commit=659604f graphdriver=overlay2 version=24.0.2
Jun 26 17:05:42 vm249 dockerd[129391]: time="2023-06-26T17:05:42.438899998-07:00" level=info msg="Daemon has completed initialization"
Jun 26 17:05:42 vm249 dockerd[129391]: time="2023-06-26T17:05:42.478909857-07:00" level=info msg="API listen on /run/docker.sock"
Jun 26 17:05:42 vm249 systemd[1]: Started Docker Application Container Engine.
^C
[root@vm249 certs.d]# date
Mon Jun 26 17:06:13 PDT 2023
[root@vm249 certs.d]# cd
[root@vm249 ~]# wget https://github.com/goharbor/harbor/releases/download/v2.8.2/harbor-online-installer-v2.8.2.tgz
--2023-06-26 17:09:40--  https://github.com/goharbor/harbor/releases/download/v2.8.2/harbor-online-installer-v2.8.2.tgz
Resolving github.com (github.com)... 20.205.243.166
Connecting to github.com (github.com)|20.205.243.166|:443... connected.
^C
[root@vm249 ~]# wget https://github.com/goharbor/harbor/releases/download/v2.8.2/harbor-online-installer-v2.8.2.tgz
--2023-06-26 17:10:04--  https://github.com/goharbor/harbor/releases/download/v2.8.2/harbor-online-installer-v2.8.2.tgz
Resolving github.com (github.com)... 20.205.243.166
Connecting to github.com (github.com)|20.205.243.166|:443... connected.
^C
[root@vm249 ~]# ll
total 56
-rw-r--r--. 1 root root  2053 Jun 26 17:03 172.16.3.249.cert
-rw-r--r--. 1 root root  2053 Jun 26 17:02 172.16.3.249.crt
-rw-r--r--. 1 root root  1704 Jun 26 17:01 172.16.3.249.csr
-rw-r--r--. 1 root root  3247 Jun 26 17:00 172.16.3.249.key
-rw-------. 1 root root  4682 Feb  8 12:16 anaconda-ks.cfg
-rw-r--r--. 1 root root  2029 Jun 26 17:00 ca.crt
-rw-r--r--. 1 root root  3243 Jun 26 16:59 ca.key
-rw-r--r--. 1 root root    17 Jun 26 17:02 ca.srl
-rw-r--r--. 1 root root 11032 Jun 26 17:10 harbor-online-installer-v2.8.2.tgz
-rw-r--r--. 1 root root     0 Jun 26 17:01 -key
-rw-------. 1 root root  3711 Feb  8 12:16 original-ks.cfg
-rw-r--r--. 1 root root     0 Jun 26 17:01 -out
drwxr-xr-x. 3 root root    49 Jun 25 20:04 software
-rw-r--r--. 1 root root     0 Jun 26 17:01 -subj
-rw-r--r--. 1 root root   204 Jun 26 17:02 v3.ext
[root@vm249 ~]# tar xvf harbor-online-installer-v2.8.2.tgz
harbor/prepare
harbor/LICENSE
harbor/install.sh
harbor/common.sh
harbor/harbor.yml.tmpl
[root@vm249 ~]# cd harbor/
[root@vm249 harbor]# ll
total 36
-rw-r--r--. 1 root root  3639 Jun  2 04:46 common.sh
-rw-r--r--. 1 root root 11736 Jun  2 04:46 harbor.yml.tmpl
-rwxr-xr-x. 1 root root  2725 Jun  2 04:46 install.sh
-rw-r--r--. 1 root root 11347 Jun  2 04:46 LICENSE
-rwxr-xr-x. 1 root root  1881 Jun  2 04:46 prepare
[root@vm249 harbor]# vi harbor.yml.tmpl ^C
[root@vm249 harbor]# cp harbor.yml.tmpl  harbor.yml
[root@vm249 harbor]# vi harbor.yml
[root@vm249 harbor]# ./install.sh

[Step 0]: checking if docker is installed ...

Note: docker version: 24.0.2

[Step 1]: checking docker-compose is installed ...

Note: Docker Compose version v2.18.1


[Step 2]: preparing environment ...

[Step 3]: preparing harbor configs ...
prepare base dir is set to /root/harbor
Unable to find image 'goharbor/prepare:v2.8.2' locally
v2.8.2: Pulling from goharbor/prepare
06212d50621c: Pull complete
c43a11a41dd8: Pull complete
9e83342251c5: Pull complete
628dbbe4d69c: Pull complete
e02a6cdc2a2e: Pull complete
74305682af4e: Pull complete
d714712cc6df: Pull complete
2e443303cc8f: Pull complete
7b608b2485a2: Pull complete
dcd76662a9cc: Pull complete
Digest: sha256:72252d93c32f774567400834b95e5446706106ff51efac2a20523238617d760d
Status: Downloaded newer image for goharbor/prepare:v2.8.2
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /data/secret/keys/secretkey
Successfully called func: create_root_cert
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir


Note: stopping existing Harbor instance ...


[Step 4]: starting Harbor ...
[+] Running 60/11
 ✔ proxy 2 layers [⣿⣿]      0B/0B      Pulled                                                                                                                  27.1s
 ✔ portal 3 layers [⣿⣿⣿]      0B/0B      Pulled                                                                                                               115.4s
 ✔ log 7 layers [⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                                              133.2s
 ✔ postgresql 10 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                                    69.9s
 ✔ core 9 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                                            43.6s
 ✔ registry 5 layers [⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                                            16.0s
 ✔ registryctl 6 layers [⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                                        99.7s
 ✔ redis 4 layers [⣿⣿⣿⣿]      0B/0B      Pulled                                                                                                               128.6s
 ✔ jobservice 5 layers [⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                                                          88.9s


[+] Building 0.0s (0/0)
[+] Running 10/10
 ✔ Network harbor_harbor        Created                                                                                                                         0.1s
 ✔ Container harbor-log         Started                                                                                                                         1.2s
 ✔ Container registryctl        Started                                                                                                                         1.9s
 ✔ Container harbor-portal      Started                                                                                                                         1.5s
 ✔ Container redis              Started                                                                                                                         2.1s
 ✔ Container harbor-db          Started                                                                                                                         1.9s
 ✔ Container registry           Started                                                                                                                         1.8s
 ✔ Container harbor-core        Started                                                                                                                         2.4s
 ✔ Container nginx              Started                                                                                                                         3.6s
 ✔ Container harbor-jobservice  Started                                                                                                                         3.5s
✔ ----Harbor has been installed and started successfully.----
[root@vm249 harbor]# cat harbor.yml
# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: 172.16.3.249

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 8080

# https related config
https:
  # https port for harbor, default is 443
  port: 8443
  # The path of cert and key files for nginx
  certificate: /data/cert/172.16.3.249.crt
  private_key: /data/cert/172.16.3.249.key

# # Uncomment following will enable tls communication between all harbor components
# internal_tls:
#   # set enabled to true means internal tls is enabled
#   enabled: true
#   # put your cert and key files on dir
#   dir: /etc/harbor/tls/internal

# Uncomment external_url if you want to enable external proxy
# And when it enabled the hostname will no longer used
# external_url: https://reg.mydomain.com:8433

# The initial password of Harbor admin
# It only works in first time to install harbor
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: Harbor12345

# Harbor DB configuration
database:
  # The password for the root user of Harbor DB. Change this before any production use.
  password: root123
  # The maximum number of connections in the idle connection pool. If it <=0, no idle connections are retained.
  max_idle_conns: 100
  # The maximum number of open connections to the database. If it <= 0, then there is no limit on the number of open connections.
  # Note: the default number of connections is 1024 for postgres of harbor.
  max_open_conns: 900
  # The maximum amount of time a connection may be reused. Expired connections may be closed lazily before reuse. If it <= 0, connections are not closed due to a connection's age.
  # The value is a duration string. A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as "300ms", "-1.5h" or "2h45m". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
  conn_max_lifetime: 5m
  # The maximum amount of time a connection may be idle. Expired connections may be closed lazily before reuse. If it <= 0, connections are not closed due to a connection's idle time.
  # The value is a duration string. A duration string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as "300ms", "-1.5h" or "2h45m". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
  conn_max_idle_time: 0

# The default data volume
data_volume: /data

# Harbor Storage settings by default is using /data dir on local filesystem
# Uncomment storage_service setting If you want to using external storage
# storage_service:
#   # ca_bundle is the path to the custom root ca certificate, which will be injected into the truststore
#   # of registry's containers.  This is usually needed when the user hosts a internal storage with self signed certificate.
#   ca_bundle:

#   # storage backend, default is filesystem, options include filesystem, azure, gcs, s3, swift and oss
#   # for more info about this configuration please refer https://docs.docker.com/registry/configuration/
#   filesystem:
#     maxthreads: 100
#   # set disable to true when you want to disable registry redirect
#   redirect:
#     disable: false

# Trivy configuration
#
# Trivy DB contains vulnerability information from NVD, Red Hat, and many other upstream vulnerability databases.
# It is downloaded by Trivy from the GitHub release page https://github.com/aquasecurity/trivy-db/releases and cached
# in the local file system. In addition, the database contains the update timestamp so Trivy can detect whether it
# should download a newer version from the Internet or use the cached one. Currently, the database is updated every
# 12 hours and published as a new release to GitHub.
trivy:
  # ignoreUnfixed The flag to display only fixed vulnerabilities
  ignore_unfixed: false
  # skipUpdate The flag to enable or disable Trivy DB downloads from GitHub
  #
  # You might want to enable this flag in test or CI/CD environments to avoid GitHub rate limiting issues.
  # If the flag is enabled you have to download the `trivy-offline.tar.gz` archive manually, extract `trivy.db` and
  # `metadata.json` files and mount them in the `/home/scanner/.cache/trivy/db` path.
  skip_update: false
  #
  # The offline_scan option prevents Trivy from sending API requests to identify dependencies.
  # Scanning JAR files and pom.xml may require Internet access for better detection, but this option tries to avoid it.
  # For example, the offline mode will not try to resolve transitive dependencies in pom.xml when the dependency doesn't
  # exist in the local repositories. It means a number of detected vulnerabilities might be fewer in offline mode.
  # It would work if all the dependencies are in local.
  # This option doesn't affect DB download. You need to specify "skip-update" as well as "offline-scan" in an air-gapped environment.
  offline_scan: false
  #
  # Comma-separated list of what security issues to detect. Possible values are `vuln`, `config` and `secret`. Defaults to `vuln`.
  security_check: vuln
  #
  # insecure The flag to skip verifying registry certificate
  insecure: false
  # github_token The GitHub access token to download Trivy DB
  #
  # Anonymous downloads from GitHub are subject to the limit of 60 requests per hour. Normally such rate limit is enough
  # for production operations. If, for any reason, it's not enough, you could increase the rate limit to 5000
  # requests per hour by specifying the GitHub access token. For more details on GitHub rate limiting please consult
  # https://developer.github.com/v3/#rate-limiting
  #
  # You can create a GitHub token by following the instructions in
  # https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line
  #
  # github_token: xxx

jobservice:
  # Maximum number of job workers in job service
  max_job_workers: 10
  # The jobLogger sweeper duration (ignored if `jobLogger` is `stdout`)
  logger_sweeper_duration: 1 #days

notification:
  # Maximum retry count for webhook job
  webhook_job_max_retry: 3
  # HTTP client timeout for webhook job
  webhook_job_http_client_timeout: 3 #seconds

# Log configurations
log:
  # options are debug, info, warning, error, fatal
  level: info
  # configs for logs in local storage
  local:
    # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
    rotate_count: 50
    # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
    # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
    # are all valid.
    rotate_size: 200M
    # The directory on your host that store log
    location: /var/log/harbor

  # Uncomment following lines to enable external syslog endpoint.
  # external_endpoint:
  #   # protocol used to transmit log to external endpoint, options is tcp or udp
  #   protocol: tcp
  #   # The host of external endpoint
  #   host: localhost
  #   # Port of external endpoint
  #   port: 5140

#This attribute is for migrator to detect the version of the .cfg file, DO NOT MODIFY!
_version: 2.8.0

# Uncomment external_database if using external database.
# external_database:
#   harbor:
#     host: harbor_db_host
#     port: harbor_db_port
#     db_name: harbor_db_name
#     username: harbor_db_username
#     password: harbor_db_password
#     ssl_mode: disable
#     max_idle_conns: 2
#     max_open_conns: 0
#   notary_signer:
#     host: notary_signer_db_host
#     port: notary_signer_db_port
#     db_name: notary_signer_db_name
#     username: notary_signer_db_username
#     password: notary_signer_db_password
#     ssl_mode: disable
#   notary_server:
#     host: notary_server_db_host
#     port: notary_server_db_port
#     db_name: notary_server_db_name
#     username: notary_server_db_username
#     password: notary_server_db_password
#     ssl_mode: disable

# Uncomment external_redis if using external Redis server
# external_redis:
#   # support redis, redis+sentinel
#   # host for redis: <host_redis>:<port_redis>
#   # host for redis+sentinel:
#   #  <host_sentinel1>:<port_sentinel1>,<host_sentinel2>:<port_sentinel2>,<host_sentinel3>:<port_sentinel3>
#   host: redis:6379
#   password:
#   # Redis AUTH command was extended in Redis 6, it is possible to use it in the two-arguments AUTH <username> <password> form.
#   # username:
#   # sentinel_master_set must be set to support redis+sentinel
#   #sentinel_master_set:
#   # db_index 0 is for core, it's unchangeable
#   registry_db_index: 1
#   jobservice_db_index: 2
#   trivy_db_index: 5
#   idle_timeout_seconds: 30

# Uncomment uaa for trusting the certificate of uaa instance that is hosted via self-signed cert.
# uaa:
#   ca_file: /path/to/ca

# Global proxy
# Config http proxy for components, e.g. http://my.proxy.com:3128
# Components doesn't need to connect to each others via http proxy.
# Remove component from `components` array if want disable proxy
# for it. If you want use proxy for replication, MUST enable proxy
# for core and jobservice, and set `http_proxy` and `https_proxy`.
# Add domain to the `no_proxy` field, when you want disable proxy
# for some special registry.
proxy:
  http_proxy:
  https_proxy:
  no_proxy:
  components:
    - core
    - jobservice
    - trivy

# metric:
#   enabled: false
#   port: 9090
#   path: /metrics

# Trace related config
# only can enable one trace provider(jaeger or otel) at the same time,
# and when using jaeger as provider, can only enable it with agent mode or collector mode.
# if using jaeger collector mode, uncomment endpoint and uncomment username, password if needed
# if using jaeger agetn mode uncomment agent_host and agent_port
# trace:
#   enabled: true
#   # set sample_rate to 1 if you wanna sampling 100% of trace data; set 0.5 if you wanna sampling 50% of trace data, and so forth
#   sample_rate: 1
#   # # namespace used to differenciate different harbor services
#   # namespace:
#   # # attributes is a key value dict contains user defined attributes used to initialize trace provider
#   # attributes:
#   #   application: harbor
#   # # jaeger should be 1.26 or newer.
#   # jaeger:
#   #   endpoint: http://hostname:14268/api/traces
#   #   username:
#   #   password:
#   #   agent_host: hostname
#   #   # export trace data by jaeger.thrift in compact mode
#   #   agent_port: 6831
#   # otel:
#   #   endpoint: hostname:4318
#   #   url_path: /v1/traces
#   #   compression: false
#   #   insecure: true
#   #   timeout: 10s

# Enable purge _upload directories
upload_purging:
  enabled: true
  # remove files in _upload directories which exist for a period of time, default is one week.
  age: 168h
  # the interval of the purge operations
  interval: 24h
  dryrun: false

# Cache layer configurations
# If this feature enabled, harbor will cache the resource
# `project/project_metadata/repository/artifact/manifest` in the redis
# which can especially help to improve the performance of high concurrent
# manifest pulling.
# NOTICE
# If you are deploying Harbor in HA mode, make sure that all the harbor
# instances have the same behaviour, all with caching enabled or disabled,
# otherwise it can lead to potential data inconsistency.
cache:
  # not enabled by default
  enabled: false
  # keep cache for one day by default
  expire_hours: 24
[root@vm249 harbor]# cat harbor.yml | grep pass
# The initial password of Harbor admin
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: Harbor12345
  # The password for the root user of Harbor DB. Change this before any production use.
  password: root123
#     password: harbor_db_password
#     password: notary_signer_db_password
#     password: notary_server_db_password
#   password:
#   # Redis AUTH command was extended in Redis 6, it is possible to use it in the two-arguments AUTH <username> <password> form.
# if using jaeger collector mode, uncomment endpoint and uncomment username, password if needed
#   #   password:
[root@vm249 harbor]# docker ps
CONTAINER ID   IMAGE                                COMMAND                  CREATED              STATUS                        PORTS                                                                                  NAMES
facae319c827   goharbor/nginx-photon:v2.8.2         "nginx -g 'daemon of…"   About a minute ago   Up About a minute (healthy)   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:8443->8443/tcp, :::8443->8443/tcp   nginx
e1ac0a3eeba4   goharbor/harbor-jobservice:v2.8.2    "/harbor/entrypoint.…"   About a minute ago   Up About a minute (healthy)                                                                                          harbor-jobservice
3768487b45f7   goharbor/harbor-core:v2.8.2          "/harbor/entrypoint.…"   About a minute ago   Up About a minute (healthy)                                                                                          harbor-core
ea1996570673   goharbor/harbor-db:v2.8.2            "/docker-entrypoint.…"   About a minute ago   Up About a minute (healthy)                                                                                          harbor-db
7a69890bf969   goharbor/registry-photon:v2.8.2      "/home/harbor/entryp…"   About a minute ago   Up About a minute (healthy)                                                                                          registry
e2d7461138fc   goharbor/harbor-registryctl:v2.8.2   "/home/harbor/start.…"   About a minute ago   Up About a minute (healthy)                                                                                          registryctl
c3948d042541   goharbor/harbor-portal:v2.8.2        "nginx -g 'daemon of…"   About a minute ago   Up About a minute (healthy)                                                                                          harbor-portal
a88cf970d5fa   goharbor/redis-photon:v2.8.2         "redis-server /etc/r…"   About a minute ago   Up About a minute (healthy)                                                                                          redis
3cca3b17f6da   goharbor/harbor-log:v2.8.2           "/bin/sh -c /usr/loc…"   About a minute ago   Up About a minute (healthy)   127.0.0.1:1514->10514/tcp                                                              harbor-log
[root@vm249 harbor]# ll /etc/docker/
total 0
drwxr-xr-x. 3 root root 31 Jun 26 17:05 certs.d
[root@vm249 harbor]# vi /etc/docker/^C
[root@vm249 harbor]# ll
total 56
drwxr-xr-x. 3 root root    20 Jun 26 17:17 common
-rw-r--r--. 1 root root  3639 Jun  2 04:46 common.sh
-rw-r--r--. 1 root root  5892 Jun 26 17:17 docker-compose.yml
-rw-r--r--. 1 root root 11745 Jun 26 17:16 harbor.yml
-rw-r--r--. 1 root root 11736 Jun  2 04:46 harbor.yml.tmpl
-rwxr-xr-x. 1 root root  2725 Jun  2 04:46 install.sh
-rw-r--r--. 1 root root 11347 Jun  2 04:46 LICENSE
-rwxr-xr-x. 1 root root  1881 Jun  2 04:46 prepare
[root@vm249 harbor]# docker images
REPOSITORY                    TAG       IMAGE ID       CREATED       SIZE
goharbor/redis-photon         v2.8.2    6f4498a430ca   3 weeks ago   121MB
goharbor/harbor-registryctl   v2.8.2    fa61a236a6d6   3 weeks ago   142MB
goharbor/registry-photon      v2.8.2    f80e71363231   3 weeks ago   79.3MB
goharbor/nginx-photon         v2.8.2    3d009028f260   3 weeks ago   120MB
goharbor/harbor-log           v2.8.2    2914d282d9bf   3 weeks ago   127MB
goharbor/harbor-jobservice    v2.8.2    40118f1568a8   3 weeks ago   141MB
goharbor/harbor-core          v2.8.2    0bbbd1f379fc   3 weeks ago   165MB
goharbor/harbor-portal        v2.8.2    3e74e0758aa4   3 weeks ago   127MB
goharbor/harbor-db            v2.8.2    5126635ae9f0   3 weeks ago   174MB
goharbor/prepare              v2.8.2    eb3cf3cdd17a   3 weeks ago   163MB
[root@vm249 harbor]#

参考

https://goharbor.io/docs/2.0.0/install-config/configure-https/

https://blog.csdn.net/networken/article/details/107502461