Minio在线扩容踩坑

一、背景

1、基于docker-compose部署，每个节点docker-compose.yml配置如下

version: '3.8'

services:
  minio:
    image: minio/minio:RELEASE.2022-05-19T18-20-59Z
    command: server --address ":9010" --console-address ":9011"  http://node10:9010/data http://node11:9010/data http://node12:9010/data http://node13:9010/data
    restart: always
    hostname: node10
    container_name: minio
    network_mode: host
    volumes:
      - "/etc/localtime:/etc/localtime:ro"
      - "./data:/data"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: xxxxxx
    healthcheck:
      test: ["CMD", "curl", "-f", "http://node10:9010/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

2、4个节点组成的集群

二、扩容

在原有节点基础上新增4个节点【按官方要求，必须是节点倍数扩容】

参考：官方扩容

每个节点都修改配置，如下，同时重启所有节点【按官方要求】

version: '3.8'

services:
  minio:
    image: minio/minio:RELEASE.2022-05-19T18-20-59Z
    command: server --address ":9010" --console-address ":9011"  http://node10:9010/data http://node11:9010/data http://node12:9010/data http://node13:9010/data http://node14:9010/data http://node15:9010/data http://node16:9010/data http://node17:9010/data
    restart: always
    hostname: node10
    container_name: minio
    network_mode: host
    volumes:
      - "/etc/localtime:/etc/localtime:ro"
      - "./data:/data"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: xxxxxx
    healthcheck:
      test: ["CMD", "curl", "-f", "http://node10:9010/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

三、踩坑

报错：ERROR Unable to initialize backend: /data1 disk is already being used in another erasure deployment. (Number of disks specified: 8 but the number of disks found in the 2nd disk’s format.json: 4).

google了一圈，也没解决问题，有的说要清数据重启，有的说要使用…的扩展方式重启，但是都不行，日志还是有各种报错。官方issue也解决不了。

四、最终解决方案

1、搭建一个新集群，配置如下，按…格式进行配置

version: '3.8'

services:
  minio:
    image: minio/minio:RELEASE.2022-05-19T18-20-59Z
    command: server --address ":9010" --console-address ":9011"  http://node{14...17}:9010/data
    restart: always
    hostname: node14
    container_name: minio
    network_mode: host
    volumes:
      - "/etc/localtime:/etc/localtime:ro"
      - "./data:/data"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: xxxxxx
    healthcheck:
      test: ["CMD", "curl", "-f", "http://node14:9010/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

2、迁移旧集群数据到新集群，使用mc mirror命令

3、新集群扩容，配置如下，需要更改所有节点配置，同时重启

version: '3.8'

services:
  minio:
    image: minio/minio:RELEASE.2022-05-19T18-20-59Z
    command: server --address ":9010" --console-address ":9011"  http://node{14...17}:9010/data http://node{10...13}:9010/data
    restart: always
    hostname: node14
    container_name: minio
    network_mode: host
    volumes:
      - "/etc/localtime:/etc/localtime:ro"
      - "./data:/data"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: xxxxxx
    healthcheck:
      test: ["CMD", "curl", "-f", "http://node14:9010/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3