MinIO S3分布式集群搭建

发表于 作者 laozizhu

本文介绍MinIO的集群搭建，比预想中的简单太多，完全按照官方文档来做就可以了。MinIO提供高性能、S3兼容的对象存储。MinIO是Kubernetes原生的，是每个公有云，每个分布式Kuberneters，私有云和边际云上唯一可用的对象存储套件。MinIO是软件定义的，GNU AGPL v3下是100%开源的。

参考

https://docs.min.io/docs/distributed-minio-quickstart-guide.html
GBase 8a GCDW存算分离主机版安装使用预览
GCDW元数据服务FoundationDB的集群模式配置和高可用测试

环境

部署4节点【虚拟机】集群。因为我启动时，发现要求最少4个节点。

  • 10.0.2.181
  • 10.0.2.182
  • 10.0.2.183
  • 10.0.2.184

操作系统均为redhat 8.3

安装服务

可以参考 https://www.gbase8.cn/9779#S3%E6%9C%8D%E5%8A%A1

在每个节点都安装，不用做后面的额外配置，可以等分布式集群搭建好后再做。我们同样用/data作为数据目录。

[root@redhat8-3 ~]# wget https://dl.min.io/server/minio/release/linux-amd64/minio
[root@redhat8-3 ~]# mv minio /usr/local/bin/
[root@redhat8-3 ~]# chmod +x /usr/local/bin/minio
[root@redhat8-3 ~]# mkdir -p /data

搭建分布式集群

启动服务

通过如下命令，在【每个节点】启动服务，minio将自动处理完成分布式集群的搭建。

 minio server http://10.0.2.181/data http://10.0.2.182/data http://10.0.2.183/data http://10.0.2.184/data

启动日志

我们查看下第一个启动节点日志，可以看到如下的几个步骤

181的minio服务启动后

等待其它3个节点启动

Waiting for atleast 1 remote servers to be online for bootstrap check
Following servers are currently offline or unreachable [http://10.0.2.182:9000/data http://10.0.2.183:9000/data http://10.0.2.184:9000/data]

182的minio服务启动后

日志变成尝试向183，184转储数据，但因无法连接而报错。

注意其中的Waiting for a minimum of 2 disks to come online， 还需要最少2个节点上线。

API: SYSTEM()
Time: 07:21:44 UTC 07/11/2022
Error: Marking http://10.0.2.184:9000/minio/storage/data/v45 temporary offline; caused by Post "http://10.0.2.184:9000/minio/storage/data/v45/readall?disk-id=&file-path=format.json&volume=.minio.sys": dial tcp 10.0.2.184:9000: connect: connection refused (*fmt.wrapError)
       7: internal/logger/logger.go:278:logger.LogIf()
       6: internal/rest/client.go:151:rest.(*Client).Call()
       5: cmd/storage-rest-client.go:152:cmd.(*storageRESTClient).call()
       4: cmd/storage-rest-client.go:523:cmd.(*storageRESTClient).ReadAll()
       3: cmd/format-erasure.go:393:cmd.loadFormatErasure()
       2: cmd/format-erasure.go:332:cmd.loadFormatErasureAll.func1()
       1: internal/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()

API: SYSTEM()
Time: 07:21:44 UTC 07/11/2022
Error: Marking http://10.0.2.183:9000/minio/storage/data/v45 temporary offline; caused by Post "http://10.0.2.183:9000/minio/storage/data/v45/readall?disk-id=&file-path=format.json&volume=.minio.sys": dial tcp 10.0.2.183:9000: connect: connection refused (*fmt.wrapError)
       7: internal/logger/logger.go:278:logger.LogIf()
       6: internal/rest/client.go:151:rest.(*Client).Call()
       5: cmd/storage-rest-client.go:152:cmd.(*storageRESTClient).call()
       4: cmd/storage-rest-client.go:523:cmd.(*storageRESTClient).ReadAll()
       3: cmd/format-erasure.go:393:cmd.loadFormatErasure()
       2: cmd/format-erasure.go:332:cmd.loadFormatErasureAll.func1()
       1: internal/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()

API: SYSTEM()
Time: 07:21:44 UTC 07/11/2022
Error: Read failed. Insufficient number of disks online (*errors.errorString)
       6: internal/logger/logger.go:278:logger.LogIf()
       5: cmd/prepare-storage.go:242:cmd.connectLoadInitFormats()
       4: cmd/prepare-storage.go:302:cmd.waitForFormatErasure()
       3: cmd/erasure-server-pool.go:109:cmd.newErasureServerPools()
       2: cmd/server-main.go:679:cmd.newObjectLayer()
       1: cmd/server-main.go:529:cmd.serverMain()
Waiting for a minimum of 2 disks to come online (elapsed 1s)

183的minio服务启动后

只有184报错了，而且提示信息也变了

需要其它节点上线，以便【格式化】磁盘 Waiting for all other servers to be online to format the disks

API: SYSTEM()
Time: 07:22:12 UTC 07/11/2022
Error: Marking http://10.0.2.184:9000/minio/storage/data/v45 temporary offline; caused by Post "http://10.0.2.184:9000/minio/storage/data/v45/readall?disk-id=&file-path=format.json&volume=.minio.sys": dial tcp 10.0.2.184:9000: connect: connection refused (*fmt.wrapError)
       7: internal/logger/logger.go:278:logger.LogIf()
       6: internal/rest/client.go:151:rest.(*Client).Call()
       5: cmd/storage-rest-client.go:152:cmd.(*storageRESTClient).call()
       4: cmd/storage-rest-client.go:523:cmd.(*storageRESTClient).ReadAll()
       3: cmd/format-erasure.go:393:cmd.loadFormatErasure()
       2: cmd/format-erasure.go:332:cmd.loadFormatErasureAll.func1()
       1: internal/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()
Waiting for all other servers to be online to format the disks (elapses 28s)

184的minio服务启动后

初始化正常。All MinIO sub-systems initialized successfully in 141.467047ms

集群可以对外提供服务了。可以访问WEB端，默认是随机的，可以用参数--console-address ":8080" 固定

API: SYSTEM()
Time: 07:22:29 UTC 07/11/2022
Error: Marking http://10.0.2.184:9000/minio/storage/data/v45 temporary offline; caused by Post "http://10.0.2.184:9000/minio/storage/data/v45/readall?disk-id=&file-path=format.json&volume=.minio.sys": dial tcp 10.0.2.184:9000: connect: connection refused (*fmt.wrapError)
       7: internal/logger/logger.go:278:logger.LogIf()
       6: internal/rest/client.go:151:rest.(*Client).Call()
       5: cmd/storage-rest-client.go:152:cmd.(*storageRESTClient).call()
       4: cmd/storage-rest-client.go:523:cmd.(*storageRESTClient).ReadAll()
       3: cmd/format-erasure.go:393:cmd.loadFormatErasure()
       2: cmd/format-erasure.go:332:cmd.loadFormatErasureAll.func1()
       1: internal/sync/errgroup/errgroup.go:123:errgroup.(*Group).Go.func1()
Waiting for all other servers to be online to format the disks (elapses 45s)

Formatting 1st pool, 1 set(s), 4 drives per set.
WARNING: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables
Waiting for all MinIO sub-systems to be initialized.. lock acquired
Automatically configured API requests per node based on available memory on the system: 24
All MinIO sub-systems initialized successfully in 141.467047ms
Waiting for all MinIO IAM sub-system to be initialized.. lock acquired
Status:         4 Online, 0 Offline.
API: http://10.0.2.181:9000  http://192.168.122.1:9000  http://127.0.0.1:9000
RootUser: minioadmin
RootPass: minioadmin

Console: http://10.0.2.181:33437 http://192.168.122.1:33437 http://127.0.0.1:33437
RootUser: minioadmin
RootPass: minioadmin

Command-line: https://docs.min.io/docs/minio-client-quickstart-guide
   $ mc alias set myminio http://10.0.2.181:9000 minioadmin minioadmin

Documentation: https://docs.min.io

WARNING: Console endpoint is listening on a dynamic port (33437), please use --console-address ":PORT" to choose a static port.
Finished loading IAM sub-system (took 0.0s of 0.2s to load data).

MinIO的集群WEB客户端

从监控里可以看到服务器数量是4个，磁盘是4个。

MinIO 分布式集群搭建完成的WEB客户端

单机多磁盘的分布式集群

如果单机有不低于4块磁盘，则可以通过如下方式启动

minio server http://10.0.2.181/data[1...4]  http://10.0.2.182/data[1...4] http://10.0.2.183/data[1...4] http://10.0.2.184/data[1...4]

注意，其中的点是3个，不是2个。每个磁盘必须是独立mount的分区，不能共用一个分区（否额一个分区坏了，会影响多个分片)。

Post Views: 3,474

相关文章:

  1. 巨杉SequoiaDB 5.0.1日常运维使用命令汇总
  2. firebolt 第三代云数仓引擎官网和特性介绍
  3. firebolt与snowflake的POC性能对比快50-6000倍,便宜10倍
  4. 各类数据库整体架构图汇总
  5. Kafka 增加TOPIC分区partition数量的方法
  6. [转]如何打造一款极速分析型数据库-StarRocks康凯森
  7. 南大通用GBase 8c 分布式数据库集群—在线扩容介绍
  8. 南大通用GBase 8c 分布式数据库集群-事务处理核心技术之2PC协议
  9. 南大通用gcdw云数仓之foundationdb 数据库集群的监控status命令输出
  10. Hadoop不同版本对Java支持情况和支持的JDK