How To Back Up and Restore a Kubernetes Cluster on DigitalOcean Using Heptio Ark

Introduction

Heptio Ark is a convenient backup tool for Kubernetes clusters that compresses and backs up Kubernetes objects to object storage. It also takes snapshots of your cluster's Persistent Volumes using your cloud provider's block storage snapshot features, and can then restore your cluster's objects and Persistent Volumes to a previous state.

StackPointCloud's DigitalOcean Ark Plugin allows you to use DigitalOcean block storage to snapshot your Persistent Volumes, and Spaces to back up your Kubernetes objects. When running a Kubernetes cluster on DigitalOcean, this allows you to quickly back up your cluster's state and restore it should disaster strike.

In this tutorial we'll set up and configure the Ark client on a local machine, and deploy the Ark server into our Kubernetes cluster. We'll then deploy a sample Nginx app that uses a Persistent Volume for logging, and simulate a disaster recovery scenario.

Prerequisites

Before you begin this tutorial, you should have the following available to you:

On your local computer:

  • The kubectl command-line tool, configured to connect to your cluster. You can read more about installing and configuring kubectl in the official Kubernetes documentation.
  • The git command-line utility. You can learn how to install git in Getting Started with Git.

In your DigitalOcean account:

  • A DigitalOcean Kubernetes cluster, or a Kubernetes cluster (version 1.7.5 or later) on DigitalOcean Droplets
  • A DNS server running inside of your cluster. If you are using DigitalOcean Kubernetes, this is running by default. To learn more about configuring a Kubernetes DNS service, consult Customizing DNS Service from the official Kuberentes documentation.
  • A DigitalOcean Space that will store your backed-up Kubernetes objects. To learn how to create a Space, consult the Spaces product documentation.
  • An access key pair for your DigitalOcean Space. To learn how to create a set of access keys, consult How to Manage Administrative Access to Spaces.
  • A personal access token for use with the DigitalOcean API. To learn how to create a personal access token, consult How to Create a Personal Access Token.

Once you have all of this set up, you're ready to begin with this guide.

Step 1 — Installing the Ark Client

The Heptio Ark backup tool consists of a client installed on your local computer and a server that runs in your Kubernetes cluster. To begin, we'll install the local Ark client.

In your web browser, navigate to the Ark GitHub repo releases page, find the latest release corresponding to your OS and system architecture, and copy the link address. For the purposes of this guide, we'll use an Ubuntu 18.04 server on an x86-64 (or AMD64) processor as our local machine.

Then, from the command line on your local computer, navigate to the temporary /tmp directory and cd into it:

  • cd /tmp

Use wget and the link you copied earlier to download the release tarball:

  • wget https://link_copied_from_release_page

Once the download completes, extract the tarball using tar (note the filename may differ depending on the current release version and your OS):

  • tar -xvzf ark-v0.9.6-linux-amd64.tar.gz

The /tmp directory should now contain the extracted ark binary as well as the tarball you just downloaded.

Verify that you can run the ark client by executing the binary:

  • ./ark --help

You should see the following help output:

OutputHeptio Ark is a tool for managing disaster recovery, specifically for Kubernetes cluster resources. It provides a simple, configurable, and operationally robust way to back up your application state and associated data.  If you're familiar with kubectl, Ark supports a similar model, allowing you to execute commands such as 'ark get backup' and 'ark create schedule'. The same operations can also be performed as 'ark backup get' and 'ark schedule create'.  Usage:   ark [command]  Available Commands:   backup      Work with backups   client      Ark client related commands   completion  Output shell completion code for the specified shell (bash or zsh)   create      Create ark resources   delete      Delete ark resources   describe    Describe ark resources   get         Get ark resources   help        Help about any command   plugin      Work with plugins   restic      Work with restic   restore     Work with restores   schedule    Work with schedules   server      Run the ark server   version     Print the ark version and associated image  . . . 

At this point you should move the ark executable out of the temporary /tmp directory and add it to your PATH. To add it to your PATH on an Ubuntu system, simply copy it to /usr/local/bin:

  • sudo mv ark /usr/local/bin/ark

You're now ready to configure the Ark server and deploy it to your Kubernetes cluster.

Step 2 — Installing and Configuring the Ark Server

Before we deploy Ark into our Kubernetes cluster, we'll first create Ark's prerequisite objects. Ark's prerequisites consist of:

  • A heptio-ark Namespace

  • The ark Service Account

  • Role-based access control (RBAC) rules to grant permissions to the ark Service Account

  • Custom Resources (CRDs) for the Ark-specific resources: Backup, Schedule, Restore, Config

A YAML file containing the specs for the above Kubernetes objects can be found in the official Ark Git repository. While still in the /tmp directory, download the Ark repo using git:

  • git clone https://github.com/heptio/ark.git

Once downloaded, navigate into the ark directory:

  • cd ark

The prerequisite resources listed above can be found in the examples/common/00-prereqs.yaml YAML file. We'll create these resources in our Kubernetes cluster by using kubectl apply and passing in the file:

  • kubectl apply -f examples/common/00-prereqs.yaml

You should see the following output:

Outputcustomresourcedefinition.apiextensions.k8s.io/backups.ark.heptio.com created customresourcedefinition.apiextensions.k8s.io/schedules.ark.heptio.com created customresourcedefinition.apiextensions.k8s.io/restores.ark.heptio.com created customresourcedefinition.apiextensions.k8s.io/configs.ark.heptio.com created customresourcedefinition.apiextensions.k8s.io/downloadrequests.ark.heptio.com created customresourcedefinition.apiextensions.k8s.io/deletebackuprequests.ark.heptio.com created customresourcedefinition.apiextensions.k8s.io/podvolumebackups.ark.heptio.com created customresourcedefinition.apiextensions.k8s.io/podvolumerestores.ark.heptio.com created customresourcedefinition.apiextensions.k8s.io/resticrepositories.ark.heptio.com created customresourcedefinition.apiextensions.k8s.io/backupstoragelocations.ark.heptio.com created namespace/heptio-ark created serviceaccount/ark created clusterrolebinding.rbac.authorization.k8s.io/ark created 

Now that we've created the necessary Ark Kubernetes objects in our cluster, we can download and install the Ark DigitalOcean Plugin, which will allow us to use DigitalOcean Spaces as a backupStorageProvider (for Kubernetes objects), and DigitalOcean Block Storage as a persistentVolumeProvider (for Persistent Volume backups).

Move back out of the ark directory and fetch the plugin from StackPointCloud's repo using git:

  • cd ..
  • git clone https://github.com/StackPointCloud/ark-plugin-digitalocean.git

Move into the plugin directory:

  • cd ark-plugin-digitalocean

We'll now save the access keys for our DigitalOcean Space as a Kubernetes Secret. First, open up the examples/credentials-ark file using your favorite editor:

  • nano examples/credentials-ark

Replace <AWS_ACCESS_KEY_ID> and <AWS_SECRET_ACCESS_KEY> with your Spaces access key and secret key:

examples/credentials-ark

[default] aws_access_key_id=your_spaces_access_key_here aws_secret_access_key=your_spaces_secret_key_here 

Save and close the file.

Now, create the cloud-credentials Secret using kubectl, inserting your Personal Access Token using the digitalocean_token data item:

  • kubectl create secret generic cloud-credentials
  • --namespace heptio-ark
  • --from-file cloud=examples/credentials-ark
  • --from-literal digitalocean_token=your_personal_access_token

You should see the following output:

Outputsecret/cloud-credentials created 

To confirm that the cloud-credentials Secret was created successfully, you can describe it using kubectl:

  • kubectl describe secrets/cloud-credentials --namespace heptio-ark

You should see the following output describing the cloud-credentials secret:

OutputName:         cloud-credentials Namespace:    heptio-ark Labels:       <none> Annotations:  <none>  Type:  Opaque  Data ==== cloud:               115 bytes digitalocean_token:  64 bytes 

We can now move on to creating an Ark Config object named default. To do this, we'll edit a YAML configuration file and then create the object in our Kubernetes cluster.

Open examples/10-ark-config.yaml in your favorite editor:

  • nano examples/10-ark-config.yaml

Insert your Space's name and region in the highlighted fields:

examples/10-ark-config.yaml

--- apiVersion: ark.heptio.com/v1 kind: Config metadata:   namespace: heptio-ark   name: default persistentVolumeProvider:   name: digitalocean backupStorageProvider:   name: aws   bucket: space_name_here   config:     region: space_region_here     s3ForcePathStyle: "true"     s3Url: https://space_region_here.digitaloceanspaces.com backupSyncPeriod: 30m gcSyncPeriod: 30m scheduleSyncPeriod: 1m restoreOnlyMode: false 

persistentVolumeProvider sets DigitalOcean Block Storage as the the provider for Persistent Volume backups. These will be Block Storage Volume Snapshots.

backupStorageProvider sets DigitalOcean Spaces as the provider for Kubernetes object backups. Ark will create a tarball of all your Kubernetes objects (or some, depending on how you execute it), and upload this tarball to Spaces.

When you're done, save and close the file.

Create the object in your cluster using kubectl apply:

  • kubectl apply -f examples/10-ark-config.yaml

You should see the following output:

Outputconfig.ark.heptio.com/default created 

At this point, we've finished configuring the Ark server and can create its Kubernetes deployment, found in the examples/20-deployment.yaml configuration file. Let's take a quick look at this file:

  • cat examples/20-deployment.yaml

You should see the following text:

examples/20-deployment.yaml

--- apiVersion: apps/v1beta1 kind: Deployment metadata:   namespace: heptio-ark   name: ark spec:   replicas: 1   template:     metadata:       labels:         component: ark       annotations:         prometheus.io/scrape: "true"         prometheus.io/port: "8085"         prometheus.io/path: "/metrics"     spec:       restartPolicy: Always       serviceAccountName: ark       containers:         - name: ark           image: gcr.io/heptio-images/ark:latest           command:             - /ark           args:             - server           volumeMounts:             - name: cloud-credentials               mountPath: /credentials             - name: plugins               mountPath: /plugins             - name: scratch               mountPath: /scratch           env:             - name: AWS_SHARED_CREDENTIALS_FILE               value: /credentials/cloud             - name: ARK_SCRATCH_DIR               value: /scratch             - name: DIGITALOCEAN_TOKEN               valueFrom:                 secretKeyRef:                   key: digitalocean_token                   name: cloud-credentials       volumes:         - name: cloud-credentials           secret:             secretName: cloud-credentials         - name: plugins           emptyDir: {}         - name: scratch           emptyDir: {} 

We observe here that we're creating a Deployment called ark that consists of a single replica of the gcr.io/heptio-images/ark:latest container. The Pod is configured using the cloud-credentials secret we created earlier.

Create the Deployment using kubectl apply:

  • kubectl apply -f examples/20-deployment.yaml

You should see the following output:

Outputdeployment.apps/ark created 

We can double check that the Deployment has been successfully created using kubectl get on the heptio-ark Namespace :

  • kubectl get deployments --namespace=heptio-ark

You should see the following output:

OutputNAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE ark       1         1         1            0           8m 

The Ark server Pod may not start correctly until you install the Ark DigitalOcean plugin. To install the ark-blockstore-digitalocean plugin, use the ark client we installed earlier:

  • ark plugin add quay.io/stackpoint/ark-blockstore-digitalocean:latest

You can specify the kubeconfig to use with the --kubeconfig flag. If you don't use this flag, ark will check the KUBECONFIG environment variable and then fall back to the kubectl default (~/.kube/config).

At this point Ark is running and fully configured, and ready to back up and restore your Kubernetes cluster objects and Persistent Volumes to DigitalOcean Spaces and Block Storage.

In the next section, we'll run a quick test to make sure that the backup and restore functionality works as expected.

Step 3 — Testing Backup and Restore Procedure

Now that we've successfully installed and configured Ark, we can create a test Nginx Deployment and Persistent Volume, and run through a backup and restore drill to ensure that everything is working properly.

The ark-plugin-digitalocean repository contains a sample Nginx deployment called nginx-pv.yaml.

Let's take a quick look:

  • cat examples/nginx-pv.yaml

You should see the following text:

Output--- kind: PersistentVolumeClaim apiVersion: v1 metadata:   name: nginx-logs   namespace: nginx-example   labels:     app: nginx spec:   storageClassName: do-block-storage   accessModes:     - ReadWriteOnce   resources:     requests:       storage: 5Gi  --- apiVersion: apps/v1beta1 kind: Deployment metadata:   name: nginx-deployment   namespace: nginx-example spec:   replicas: 1   template:     metadata:       labels:         app: nginx     spec:       volumes:         - name: nginx-logs           persistentVolumeClaim:            claimName: nginx-logs       containers:       - image: nginx:1.7.9         name: nginx         ports:         - containerPort: 80         volumeMounts:           - mountPath: "/var/log/nginx"             name: nginx-logs             readOnly: false  --- apiVersion: v1 kind: Service metadata:   labels:     app: nginx   name: my-nginx   namespace: nginx-example spec:   ports:   - port: 80     targetPort: 80   selector:     app: nginx   type: LoadBalancer 

In this file, we observe specs for:

  • An Nginx Deployment consisting of a single replica of the nginx:1.7.9 container image
  • A 5Gi Persistent Volume Claim (called nginx-logs), using the do-block-storage StorageClass
  • A LoadBalancer Service that exposes port 80

Create the deployment using kubectl apply:

  • kubectl apply -f examples/nginx-pv.yml

You should see the following output:

Outputnamespace/nginx-example created persistentvolumeclaim/nginx-logs created deployment.apps/nginx-deployment created service/my-nginx created 

Check that the Deployment succeeded:

  • kubectl get deployments --namespace=nginx-example

You should see the following output:

OutputNAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE nginx-deployment   1         1         1            1           1h 

Once Available reaches 1, fetch the Nginx load balancer’s external IP using kubectl get:

  • kubectl get services --namespace=nginx-example

You should see both the internal CLUSTER-IP and EXTERNAL-IP for the my-nginx Service:

OutputNAME       TYPE           CLUSTER-IP     EXTERNAL-IP       PORT(S)        AGE my-nginx   LoadBalancer   10.32.27.0      203.0.113.0   80:30754/TCP   3m 

Note the EXTERNAL-IP and navigate to it using your web browser.

You should see the following NGINX welcome page:

Nginx Welcome Page

This indicates that your Nginx Deployment and Service are up and running.

Before we simulate our disaster scenario, let’s first check the Nginx access logs (stored on a Persistent Volume attached to the Nginx Pod):

Fetch the Pod’s name using kubectl get:

  • kubectl get pods --namespace nginx-example
OutputNAME                                READY     STATUS    RESTARTS   AGE nginx-deployment-77d8f78fcb-zt4wr   1/1       Running   0          29m 

Now, exec into the running Nginx container to get a shell inside of it:

  • kubectl exec -it nginx-deployment-77d8f78fcb-zt4wr --namespace nginx-example -- /bin/bash

Once inside the Nginx container, cat the Nginx access logs:

  • cat /var/log/nginx/access.log

You should see some Nginx access entries:

Output10.244.17.1 - - [01/Oct/2018:21:47:01 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/203.0.113.11 Safari/537.36" "-" 10.244.17.1 - - [01/Oct/2018:21:47:01 +0000] "GET /favicon.ico HTTP/1.1" 404 570 "http://203.0.113.0/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/203.0.113.11 Safari/537.36" "-" 

Note these down (especially the timestamps), as we will use them to confirm the success of the restore procedure.

We can now perform the backup procedure to copy all nginx Kubernetes objects to Spaces and take a Snapshot of the Persistent Volume we created when deploying Nginx.

We'll create a backup called nginx-backup using the ark client:

  • ark backup create nginx-backup --selector app=nginx

The --selector app=nginx instructs the Ark server to only back up Kubernetes objects with the app=nginx Label Selector.

You should see the following output:

OutputBackup request "nginx-backup" submitted successfully. Run `ark backup describe nginx-backup` for more details. 

Running ark backup describe nginx-backup should provide the following output after a short delay:

OutputName:         nginx-backup Namespace:    heptio-ark Labels:       <none> Annotations:  <none>  Phase:  Completed  Namespaces:   Included:  *   Excluded:  <none>  Resources:   Included:        *   Excluded:        <none>   Cluster-scoped:  auto  Label selector:  app=nginx  Snapshot PVs:  auto  TTL:  720h0m0s  Hooks:  <none>  Backup Format Version:  1  Started:    2018-09-26 00:14:30 -0400 EDT Completed:  2018-09-26 00:14:34 -0400 EDT  Expiration:  2018-10-26 00:14:30 -0400 EDT  Validation errors:  <none>  Persistent Volumes:   pvc-e4862eac-c2d2-11e8-920b-92c754237aeb:     Snapshot ID:        2eb66366-c2d3-11e8-963b-0a58ac14428b     Type:               ext4     Availability Zone:     IOPS:               <N/A> 

This output indicates that nginx-backup completed successfully.

From the DigitalOcean Cloud Control Panel, navigate to the Space containing your Kubernetes backup files.

You should see a new directory called nginx-backup containing the Ark backup files.

Using the left-hand navigation bar, go to Images and then Snapshots. Within Snapshots, navigate to Volumes. You should see a Snapshot corresponding to the PVC listed in the above output.

We can now test the restore procedure.

Let's first delete the nginx-example Namespace. This will delete everything in the Namespace, including the Load Balancer and Persistent Volume:

  • kubectl delete namespace nginx-example

Verify that you can no longer access Nginx at the Load Balancer endpoint, and that the nginx-example Deployment is no longer running:

  • kubectl get deployments --namespace=nginx-example
OutputNo resources found. 

We can now perform the restore procedure, once again using the ark client:

  • ark restore create --from-backup nginx-backup

Here we use create to create an Ark Restore object from the nginx-backup object.

You should see the following output:

Output
  • Restore request "nginx-backup-20180926143828" submitted successfully.
  • Run `ark restore describe nginx-backup-20180926143828` for more details.

Check the status of the restored Deployment:

  • kubectl get deployments --namespace=nginx-example
OutputNAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE nginx-deployment   1         1         1            1           1m 

Check for the creation of a Persistent Volume:

  • kubectl get pvc --namespace=nginx-example
OutputNAME         STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE nginx-logs   Bound     pvc-e4862eac-c2d2-11e8-920b-92c754237aeb   5Gi        RWO            do-block-storage   3m 

Navigate to the Nginx Service’s external IP once again to confirm that Nginx is up and running.

Finally, check the logs on the restored Persistent Volume to confirm that the log history has been preserved post-restore.

To do this, once again fetch the Pod’s name using kubectl get:

  • kubectl get pods --namespace nginx-example
OutputNAME                                READY     STATUS    RESTARTS   AGE nginx-deployment-77d8f78fcb-zt4wr   1/1       Running   0          29m 

Then exec into it:

  • kubectl exec -it nginx-deployment-77d8f78fcb-zt4wr --namespace nginx-example -- /bin/bash

Once inside the Nginx container, cat the Nginx access logs:

  • cat /var/log/nginx/access.log
Output10.244.17.1 - - [01/Oct/2018:21:47:01 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/203.0.113.11 Safari/537.36" "-" 10.244.17.1 - - [01/Oct/2018:21:47:01 +0000] "GET /favicon.ico HTTP/1.1" 404 570 "http://203.0.113.0/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/203.0.113.11 Safari/537.36" "-" 

You should see the same pre-backup access attempts (note the timestamps), confirming that the Persistent Volume restore was successful. Note that there may be additional attempts in the logs if you visited the Nginx landing page after you performed the restore.

At this point, we've successfully backed up our Kubernetes objects to DigitalOcean Spaces, and our Persistent Volumes using Block Storage Volume Snapshots. We simulated a disaster scenario, and restored service to the test Nginx application.

Conclusion

In this guide we installed and configured the Ark Kubernetes backup tool on a DigitalOcean-based Kubernetes cluster. We configured the tool to back up Kubernetes objects to DigitalOcean Spaces, and back up Persistent Volumes using Block Storage Volume Snapshots.

Ark can also be used to schedule regular backups of your Kubernetes cluster. To do this, you can use the ark schedule command. It can also be used to migrate resources from one cluster to another. To learn more about these two use cases, consult the official Ark documentation.

To learn more about DigitalOcean Spaces, consult the official Spaces documentation. To learn more about Block Storage Volumes, consult the Block Storage Volume documentation.

This tutorial builds on the README found in StackPointCloud's ark-plugin-digitalocean GitHub repo.