Deploying on Kubernetes

2 min

Garage can also be deployed on a kubernetes cluster via helm chart.

Deploying

Firstly clone the repository:

git clone https://git.deuxfleurs.fr/Deuxfleurs/garage
cd garage/scripts/helm

Deploy with default options:

helm install --create-namespace --namespace garage garage ./garage

Or deploy with custom values:

helm install --create-namespace --namespace garage garage ./garage -f values.override.yaml

After deploying, cluster layout must be configured manually as described in Creating a cluster layout. Use the following command to access garage CLI:

kubectl exec --stdin --tty -n garage garage-0 -- ./garage status

Overriding default values

All possible configuration values can be found with:

helm show values ./garage

This is an example values.overrride.yaml for deploying in a microk8s cluster with a https s3 api ingress route:

garage:
  # Use only 2 replicas per object
  replicationMode: "2"

# Start 4 instances (StatefulSets) of garage
deployment:
  replicaCount: 4

# Override default storage class and size
persistence:
  meta:
    storageClass: "openebs-hostpath"
    size: 100Mi
  data:
    storageClass: "openebs-hostpath"
    size: 1Gi

ingress:
  s3:
    api:
      enabled: true
      className: "public"
      annotations:
        cert-manager.io/cluster-issuer: "letsencrypt-prod"
        nginx.ingress.kubernetes.io/proxy-body-size: 500m
      hosts:
        - host: s3-api.my-domain.com
          paths:
            - path: /
              pathType: Prefix
      tls:
        - secretName: garage-ingress-cert
          hosts:
            - s3-api.my-domain.com

Removing

helm delete --namespace garage garage

Note that this will leave behind custom CRD garagenodes.deuxfleurs.fr, which must be removed manually if desired.

Increase PVC size on running Garage instances

Since the Garage Helm chart creates the data and meta PVC based on StatefulSet templates, increasing the PVC size can be a bit tricky.

Confirm the StorageClass used for Garage supports volume expansion

Confirm the storage class used for garage.

kubectl -n garage get pvc 
NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     VOLUMEATTRIBUTESCLASS   AGE
data-garage-0   Bound    pvc-080360c9-8ce3-4acf-8579-1701e57b7f3f   30Gi       RWO            longhorn-local   <unset>                 77d
data-garage-1   Bound    pvc-ab8ba697-6030-4fc7-ab3c-0d6df9e3dbc0   30Gi       RWO            longhorn-local   <unset>                 5d8h
data-garage-2   Bound    pvc-3ab37551-0231-4604-986d-136d0fd950ec   30Gi       RWO            longhorn-local   <unset>                 5d5h
meta-garage-0   Bound    pvc-3b457302-3023-4169-846e-c928c5f2ea65   3Gi        RWO            longhorn-local   <unset>                 77d
meta-garage-1   Bound    pvc-49ace2b9-5c85-42df-9247-51c4cf64b460   3Gi        RWO            longhorn-local   <unset>                 5d8h
meta-garage-2   Bound    pvc-99e2e50f-42b4-4128-ae2f-b52629259723   3Gi        RWO            longhorn-local   <unset>                 5d5h

In this case, the storage class is longhorn-local. Now, check if ALLOWVOLUMEEXPANSION is true for the used StorageClass.

kubectl get storageclasses.storage.k8s.io longhorn-local
NAME             PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
longhorn-local   driver.longhorn.io   Delete          Immediate           true                   103d

If your StorageClass does not support volume expansion, double check if you can enable it. Otherwise, your only real option is to spin up a new Garage cluster with increased size and migrate all data over.

If your StorageClass supports expansion, you are free to continue.

Increase the size of the PVCs

Increase the size of all PVCs to your desired size.

kubectl -n garage edit pvc data-garage-0
kubectl -n garage edit pvc data-garage-1
kubectl -n garage edit pvc data-garage-2
kubectl -n garage edit pvc meta-garage-0
kubectl -n garage edit pvc meta-garage-1
kubectl -n garage edit pvc meta-garage-2

Increase the size of the StatefulSet PVC template

This is an optional step, but if not done, future instances of Garage will be created with the original size from the template.

kubectl -n garage delete sts --cascade=orphan garage 
statefulset.apps "garage" deleted

This will remove the Garage StatefulSet but leave the pods running. It may seem destructive but needs to be done this way since edits to the size of PVC templates are prohibited.

Redeploy the StatefulSet

Now the size of future PVCs can be increased, and the Garage Helm chart can be upgraded. The new StatefulSet should take ownership of the orphaned pods again.

Navigation