Breaking Data Barriers: The Gold Combo of Harbor and MinIO(En)
I’ve previously demonstrated how to deploy the Harbor registry, where the data was primarily stored internally in Harbor. However, in practical scenarios, it’s common to store data on external object storage nodes. According to my preference, I usually integrate MinIO with Harbor for this purpose.
This article will start by exploring the functionalities of MinIO, then proceed to discuss how to integrate it with Harbor, and finally, outline the benefits of this approach. Following the convention, the main points of this article include:
- What is MinIO?
- Harbor x MinIO
- Deployment and Testing
- Conclusion
1. What is MinIO?
Minio is an open-source object storage service designed to provide a simple and efficient distributed storage solution. It allows users to establish their private cloud storage and supports applications of object storage. Key features of Minio include easy deployment, strong scalability, compatibility with the S3 API (Amazon S3 cloud storage service), and high reliability and availability. Users can leverage Minio in various applications such as building scalable data lakes, backup and restore processes, and storage for multimedia applications.
Currently, MinIO operates under a dual licensing model, comprising the GNU Affero General Public License v3.0 and the MinIO Commercial License. Deployments registered through the MinIO SUBNET utilize the commercial license, inclusive of 24/7 MinIO support.
Key advantages include:
1. Simple:
Simplicity is the cornerstone of exascale data infrastructure — both technically and operationally. No other object storage solution enables you to go from download to production in a shorter timeframe.
2. High Performance:
MinIO stands as the world’s fastest object storage solution, showcasing published GETs/PUTs results exceeding 325 GiB/sec and 165 GiB/sec on 32 nodes of NVMe drives and a 100GbE network.
3. Kubernetes-Native:
Integrated seamlessly with native Kubernetes, MinIO supports all major Kubernetes distributions, including those on public, private, and edge clouds.
4. AI Ready:
Specifically designed for artificial intelligence, MinIO seamlessly integrates with every major AI/ML technology. From predictive models to GenAI, MinIO delivers the performance and scalability required to empower AI enterprises.
2. Harbor x MinIO
※ Delpoy Minio
#------------------------------
# S2-1. download minio & mc
#------------------------------
[minio]# mkdir /minio ; cd /minio
[minio]# wget https://dl.min.io/server/minio/release/linux-amd64/archive/minio-20231209181751.0.0-1.x86_64.rpm -O minio.rpm
[minio]# dnf install minio.rpm
[minio]# wget https://dl.min.io/client/mc/release/linux-amd64/mc
[minio]# chmod +x mc
[minio]# sudo mv mc /usr/local/bin/mc
[minio]# groupadd -r minio-user
[minio]# useradd -M -r -g minio-user minio-user
[minio]# chown minio-user:minio-user /minio/data#------------------------------
# S2-2. Create systemd service
#------------------------------
[minio]# vim /usr/lib/systemd/system/minio.service
[Unit]
Description=MinIO
Documentation=https://min.io/docs/minio/linux/index.html
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio
[Service]
WorkingDirectory=/usr/local
User=minio-user
Group=minio-user
ProtectProc=invisible
EnvironmentFile=-/etc/default/minio
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi"
ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES
# MinIO RELEASE.2023-05-04T21-44-30Z adds support for Type=notify (https://www.freedesktop.org/software/systemd/man/systemd.service.html#Type=)
# This may improve systemctl setups where other services use `After=minio.server`
# Uncomment the line to enable the functionality
# Type=notify
# Let systemd restart this service always
Restart=always
# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536
# Specifies the maximum number of threads this process can create
TasksMax=infinity
# Disable timeout logic and wait until process is stopped
TimeoutStopSec=infinity
SendSIGKILL=no
[Install]
WantedBy=multi-user.target
# Built for ${project.name}-${project.version} (${project.name})#------------------------------
# S2-3. Create variables
#------------------------------
[minio]# mkdir -p /minio/data
[minio]# vim /etc/default/minio
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minioadmin123
MINIO_VOLUMES="/minio/data"#------------------------------
# S2-4. Eabling service
#------------------------------
[minio]# sudo systemctl start minio.service
[minio]# sudo systemctl status minio.service
[minio]# sudo systemctl enable --now minio.service
[minio]# journalctl -f -u minio.service#------------------------------
# S2-5. console
#------------------------------
https://minio.test.example.poc:9000
※ Deploy Harbor
Resource requirements：
#-----------------------------------------
# S2-6. Add Docker repo
#-----------------------------------------
[harbor]# yum install yum-utils
[harbor]# yum config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
[harbor]# yum clean all ; yum repolist#-----------------------------------------
# S2-7. docker install
#-----------------------------------------
[harbor]# yum remove podman buildah
[harbor]# yum install containerd.io docker-ce docker-ce-cli docker-ce-rootless-extras
[harbor]# systemctl enable --now docker
#-----------------------------------------
# S2-8. docker-compose
#-----------------------------------------
[harbor]# curl -SL https://github.com/docker/compose/releases/download/v2.20.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
[harbor]# mv docker-compose /usr/local/bin/
[harbor]# chmod +x docker-compose
[harbor]# docker-compose version
Docker Compose version v2.20.0#-----------------------------------------
# S2-9. sign-certificate
#-----------------------------------------
[harbor]# openssl genrsa -out ca.key 4096
[harbor]# openssl req -x509 -new -nodes -sha512 -days 3650 -key ca.key -out ca.crt
Common Name (eg, your name or your server's hostname) []:harbor.test.example.poc
[harbor]# openssl genrsa -out harbor.test.example.poc.key 4096
[harbor]# openssl req -sha512 -new -key harbor.test.example.poc.key -out harbor.test.example.poc.csr
Common Name (eg, your name or your server's hostname) []:harbor.test.example.poc
[harbor]# vim v3.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1=habor.test.example.poc
DNS.2=harbor.test.example.poc
DNS.3=harbor.test.example.poc
[harbor]# openssl x509 -req -sha512 -days 3650 \
-extfile v3.ext \
-CA ca.crt -CAkey ca.key -CAcreateserial \
-in harbor.test.example.poc.csr \
-out harbor.test.example.poc.crt
※ copy certificates to docker and harbor
[harbor]# mkdir -p /data/cert/
[harbor]# cp -rp harbor.test.example.poc.crt harbor.test.example.poc.key /data/cert/
[harbor]# ls -al /data/cert/
harbor.test.example.poc.crt
harbor.test.example.poc.key
※ convert crt to cert for docker login
[harbor]# cd /root/docker/certs/
[harbor]# openssl x509 -inform PEM -in harbor.test.example.poc.crt -out harbor.test.example.poc.cert
※ copy to docker certificate folder
[harbor]# cpcp
[harbor]# cp harbor.test.example.poc.cert /etc/docker/certs.d/harbor.test.example.poc
[harbor]# cpcp harbor.test.example.poc.key /etc/docker/certs.d/harbor.test.example.poc
[harbor]# cp ca.crt /etc/docker/certs.d/harbor.test.example.poc
[harbor]# systemctl restart docker#-----------------------------------------
# S2-10. harbor.yml (online installer) + minio backend
#-----------------------------------------
[harbor]# wget https://github.com/goharbor/harbor/releases/download/v2.10.0/harbor-offline-installer-v2.10.0.tgz
[harbor]# tar zxvf harbor-offline-installer-v2.10.0.tgz
[harbor]# cp harbor.yml.tmpl harbor.yml
[harbor]# vim harbor.yml
hostname: harbor1.test.example.poc
external_url: https://harbor1.test.example.poc
http: # do not use in production
port: 80
https:
port: 443
certificate: /etc/docker/certs.d/harbor1.test.example.poc/harbor1.test.example.poc.cert #SSL certificate
private_key: /etc/docker/certs.d/harbor1.test.example.poc/harbor1.test.example.poc.key #SSL key
harbor_admin_password: Harbor12345
storage_service:
s3:
accesskey: minioadmin
secretkey: minioadmin123
region: us-east-1
regionendpoint: http://minio.test.example.poc:9000
bucket: harbor
secure: false
v4auth: true#-----------------------------------------
# S2-11. install (online installer)
#-----------------------------------------
[harbor]#./install.sh --with-trivy
=> Make sure "healthy" status
[root@harbor]# docker-compose ps
#-----------------------------------------
# S2-12. UI & docker login
#-----------------------------------------
[harbor]# docker login harbor1.test.example.poc
Username: admin
Password: Harbor12345
3. Deployment and Testing
#-------------------------------------------------
# S3-1. Confirm CA location
#-------------------------------------------------
[worker01]# mkdir -p /etc/containers/certs.d/harbor1.test.example.poc/
[harbor]# scp -rp ca.crt root@worker01:/etc/containers/certs.d/harbor1.test.example.poc/
[worker01]# podman pull harbor.test.example.poc/nginx-test/nginx:1.14.2#-------------------------------------------------
# S3-2. deploy nginx
#-------------------------------------------------
[master]# wget https://k8s.io/examples/application/deployment.yaml
[master]# vim deployment.yaml (修改image url)
[master]# kubectl create -f deployment.yaml
[master]# kubectl get all -n nginx-test
4. Conclusion
Choosing MinIO as Harbor’s storage backend has some great perks:
- Fast and Powerful:
MinIO is speedy and scalable, making it a big help for Harbor to quickly store and retrieve images. - Works Well with Others:
MinIO plays nice with platforms that support the Amazon S3 API, giving us more options for advanced use. - Easy to Use:
MinIO is a breeze to deploy and fits right into the mix. - Grow as You Need:
With MinIO’s excellent scalability, we can easily expand storage and performance based on what Harbor needs. It’s flexible for different situations. - Community Support:
An active community means there’s a bunch of folks out there ready to help and improve things.
Keeping backend storage separate from the image registry has its advantages. Harbor can run smoothly inside Kubernetes, and our data is stored externally to avoid any accidental losses.
That’s it for today; catch you in the next one!
