How To Enable Data Encryption For Volumes

Overview

Encrypting a volume is done by simply creating a volume with the storageos.com/encryption=true label. This label can be applied against a PersistentVolumeClaim (PVC) resource definition or through a custom Ondat StorageClass resource definition.

Only the label is required to enable encryption. Once the label is present - during the volume creation, the MutatingAdmissionWebhook that runs as part of the Ondat API Manager, will create the volume encryption key, link it to the PVC and store it in a Kubernetes secret.

⚠️ Encryption can only be enabled before an Ondat volume is provisioned. Once the encrypted volume has been created, encryption cannot be removed during the volume’s lifetime.

💡 For more information on the Ondat’s Data Encryption feature, review the Data Encryption feature page.

Example - Enable Volume Data Encryption Through a `PersistentVolumeClaim` Definition

The following guidance will demonstrate how to enable Ondat’s Data Encryption through a PersistentVolumeClaim (PVC) definition.

The instructions will enable data encryption on a PVC that will be used by a StatefulSet resource in the encrypted namespace.

Create a namespace called encrypted where the encrypted volume and StatefulSet will reside.

# Create namespace called "encrypted".
cat <<EOF | kubectl create --filename -
apiVersion: v1
kind: Namespace
metadata:
  name: encrypted
  labels:
    name: encrypted
EOF

Create a custom PersistentVolumeClaim named encrypted and ensure that you add the following label » storageos.com/encryption=true to the manifest.

# Create a "encrypted" PVC.
cat <<EOF | kubectl create --filename -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: encrypted
  namespace: encrypted
  labels:
    storageos.com/encryption: "true"         # Enable Data Encryption.
spec:
  storageClassName: storageos                # Use the default Ondat StorageClass to provision persistent volumes.
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
EOF

Once the PVC resource has been successfully created, review and confirm that the storageos.com/encryption=true label has been applied.
```
# Get the label applied to the "encrypted" PVC.
kubectl get pvc encrypted --output=wide --show-labels --namespace=encrypted
```

Create a StatefulSet workload in the encrypted namespace that uses the encrypted PVC that was created in Step 2.

# Create a StatefulSet workload that uses the "encrypted" PVC.
cat <<EOF | kubectl create --filename -
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: encrypted
  namespace: encrypted                             # Create the StatefulSet workload in the "encrypted" namespace.
spec:
  selector:
    matchLabels:
      app: encrypted
  serviceName: encrypted
  replicas: 1
  template:
    metadata:
      labels:
        app: encrypted
    spec:
      volumes:
      - name: encrypted
        persistentVolumeClaim:
          claimName: encrypted                     # Use the "encrypted" PVC for the StatefulSet workload.
      containers:
        - name: encrypted
          image: spurin/gatsby:latest
          imagePullPolicy: Always
          volumeMounts:
          - mountPath: "/data"
            name: encrypted
EOF

# Review and confirm that StatefulSet workload was successfully created.
kubectl get pod --namespace=encrypted
kubectl get statefulsets.apps --namespace=encrypted

To review and confirm that volume encryption is enabled and working, run the following commands below to inspect the resources created.

# Review the secrets created in "encrypted" namespace. 
# There will be a "namespace" and "volume" encryption key generated by Ondat.
kubectl get secrets --namespace=encrypted

NAME                                                        TYPE                                  DATA   AGE
default-token-zm5dk                                         kubernetes.io/service-account-token   3      81s
storageos-namespace-key                                     Opaque                                1      63s
storageos-volume-key-e186b6c8-ba47-4bef-9293-56ad58a2b572   Opaque                                4      63s

# Describe all of the secrets in the "encrypted" namespace related to Ondat.
# Notice that there is a "storageos-namespace-key" - 1 unique key per namespace)
# Notice that there is a "storageos-volume-key-$RANDOM" - 1 unique key per volume in the namespace.
# Take note of the annotation "storageos.com/pvc=encrypted" under the "storageos-volume-key-$RANDOM" secret
kubectl describe secrets "storageos" --namespace=encrypted

Name:         storageos-namespace-key
Namespace:    encrypted
Labels:       app.kubernetes.io/component=storageos-api-manager
              app.kubernetes.io/managed-by=storageos-operator
              app.kubernetes.io/name=storageos
              app.kubernetes.io/part-of=storageos
Annotations:  <none>

Type:  Opaque

Data
====
key:  32 bytes

Name:         storageos-volume-key-e186b6c8-ba47-4bef-9293-56ad58a2b572
Namespace:    encrypted
Labels:       app.kubernetes.io/component=storageos-api-manager
              app.kubernetes.io/managed-by=storageos-operator
              app.kubernetes.io/name=storageos
              app.kubernetes.io/part-of=storageos
              storageos.com/pvc=encrypted
Annotations:  <none>

Type:  Opaque

Data
====
hmac:  32 bytes
iv:    32 bytes
key:   64 bytes
vuk:   80 bytes

# Review "storageos-volume-key-$RANDOM".
# Notice that there are 4 data objects stored in the secret > "hmac", "iv", "vuk", "iv" respectively.
kubectl get secrets storageos-volume-key-e186b6c8-ba47-4bef-9293-56ad58a2b572 --namespace=encrypted --output=yaml

apiVersion: v1
data:
  hmac: mDIGqgm1mcjdJycz0Mg807+j+B9cCL59tuQwdvW0ow4=
  iv: gTto6RstRUamHwbn/QDZODTOuE2T2D9WzjX1ck8nZSg=
  key: FNiJp5hyoKxMTgJismLHuD0fuFgW6ozsg3vKHs2Zpzqj0enLLfT+yIiFtWW88TIi+ZpNP4hwnqpKMvKOKzU3jA==
  vuk: kWqmRear2ew6lEv48b/d3U6JJh58LCzeuY9r06Xi6dCKTBaHrLQD2A+gtxuZ0HHhoTwO1nEYAIE20QALRAi61GrosRmQzm05Jov0D/aZJ9Q=
kind: Secret
metadata:
  creationTimestamp: "2022-07-25T13:08:12Z"
  labels:
    app.kubernetes.io/component: storageos-api-manager
    app.kubernetes.io/managed-by: storageos-operator
    app.kubernetes.io/name: storageos
    app.kubernetes.io/part-of: storageos
    storageos.com/pvc: encrypted
  name: storageos-volume-key-e186b6c8-ba47-4bef-9293-56ad58a2b572
  namespace: encrypted
  resourceVersion: "64424"
  uid: 481ace83-8fb8-4656-8820-a8b8dec88017

The last step will be to try and read the data on the node where the encrypted volume resides.

# Get the node locations where "encrypted-0" pod is running.
kubectl get pods --namespace=encrypted --output=wide

NAME          READY   STATUS    RESTARTS   AGE    IP           NODE                              NOMINATED NODE   READINESS GATES
encrypted-0   1/1     Running   0          8m4s   10.244.3.6   aks-storage-26370890-vmss000000   <none>           <none>

# Use "kubectl debug" to temporarily run a privileged container on the node where "encrypted-0" is located.
kubectl debug node/aks-storage-26370890-vmss000000 -it --image=ubuntu:latest

# Through the privileged container, update the repository index and install "binutils" to access the "strings" utility.
apt update && apt install --yes binutils

# Navigate to where data is being stored as blob files on the node and list the files.
cd /host/var/lib/storageos/data/dev1/
ls -lah

total 340M
drwxr-xr-x 2 root root 4.0K Jul 25 13:08 .
drwxr-xr-x 4 root root 4.0K Jul 25 13:08 ..
-rw------- 1 root root  42M Jul 25 13:09 deployment.094b604f-a30a-4879-8976-0614ae15d5af.0.blob
-rw------- 1 root root  42M Jul 25 13:09 deployment.094b604f-a30a-4879-8976-0614ae15d5af.1.blob

# Use the "strings" utilty to try and read the content of the blob files in the directory.
# Note - the output will return multiple strings of random, unreadable characters as the content is encrypted.
strings *.blob | head -10

SbP4E
=oZ5
ey{*t|N
`0|_
xdR>
q6Yul
(OF(a
kbY#(
5- n
02p8

# exit from the pod and return to your local shell
exit

Example - Enable Volume Data Encryption Through a `StorageClass` Definition

The following guidance will demonstrates how to enable Ondat’s Data Encryption through a StorageClass definition.

The instructions will enable data encryption for PVCs through a custom StorageClass that will be used by a StatefulSet resource in the encrypted namespace.

# Create the "ondat-encryption" StorageClass.
cat <<EOF | kubectl create --filename -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ondat-encryption
provisioner: csi.storageos.com
allowVolumeExpansion: true
parameters:
  csi.storage.k8s.io/fstype: ext4
  storageos.com/encryption: "true"
  csi.storage.k8s.io/secret-name: storageos-api
  csi.storage.k8s.io/secret-namespace: storageos
EOF

# Review and confirm that "ondat-encryption" was created.
kubectl get sc | grep "ondat-encryption"

Create a namespace called encrypted where the encrypted volume and StatefulSet will reside.

# Create namespace called "encrypted".
cat <<EOF | kubectl create --filename -
apiVersion: v1
kind: Namespace
metadata:
  name: encrypted
  labels:
    name: encrypted
EOF

Create a PersistentVolumeClaim that will use ondat-encryption as its StorageClass and confirm that it was successfully created.

# Create a "encrypted-2" PVC.
cat <<EOF | kubectl create --filename -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: encrypted-2
  namespace: encrypted
spec:
  storageClassName: ondat-encryption          # Use the custom Ondat StorageClass called "ondat-encryption" to provision encrypted persistent volumes.
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
EOF

# Ensure that the PVC was successfully provisioned with "ondat-encryption".
kubectl get pvc encrypted-2 --output=wide --show-labels --namespace=encrypted

NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE   VOLUMEMODE   LABELS
encrypted-2   Bound    pvc-fe3958cb-c3d9-43aa-9ae7-530d1362e560   5Gi        RWO            ondat-encryption   46s   Filesystem   <none>

Create a StatefulSet workload in the encrypted namespace that uses the encrypted-2 PVC that was created in Step 2.

# Create a StatefulSet workload that uses the "encrypted" PVC.
cat <<EOF | kubectl create --filename -
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: encrypted-2
  namespace: encrypted                             # Create the StatefulSet workload in the "encrypted" namespace.
spec:
  selector:
    matchLabels:
      app: encrypted
  serviceName: encrypted
  replicas: 1
  template:
    metadata:
      labels:
        app: encrypted
    spec:
      volumes:
      - name: encrypted
        persistentVolumeClaim:
          claimName: encrypted-2                   # Use the "encrypted-2" PVC for the StatefulSet workload.
      containers:
        - name: encrypted
          image: spurin/gatsby:latest
          imagePullPolicy: Always
          volumeMounts:
          - mountPath: "/data"
            name: encrypted
EOF

# Review and confirm that StatefulSet workload was successfully created.
kubectl get pod --namespace=encrypted
kubectl get statefulsets.apps --namespace=encrypted

To review and confirm that volume encryption is enabled and working, run the following commands below to inspect the resources created.

# Review the secrets created in "encrypted" namespace. 
# There will be a "namespace" and "volume" encryption key generated by Ondat.
kubectl get secrets --namespace=encrypted

NAME                                                        TYPE                                  DATA   AGE
default-token-wb76h                                         kubernetes.io/service-account-token   3      10m
storageos-namespace-key                                     Opaque                                1      9m48s
storageos-volume-key-bc175830-3ed4-4d0c-90eb-df070b9930eb   Opaque                                4      9m48s

# Describe all of the secrets in the "encrypted" namespace related to Ondat.
# Notice that there is a "storageos-namespace-key" - 1 unique key per namespace)
# Notice that there is a "storageos-volume-key-$RANDOM" - 1 unique key per volume in the namespace.
# Take note of the annotation "storageos.com/pvc=encrypted" under the "storageos-volume-key-$RANDOM" secret
kubectl describe secrets "storageos" --namespace=encrypted

Name:         storageos-namespace-key
Namespace:    encrypted
Labels:       app.kubernetes.io/component=storageos-api-manager
              app.kubernetes.io/managed-by=storageos-operator
              app.kubernetes.io/name=storageos
              app.kubernetes.io/part-of=storageos
Annotations:  <none>

Type:  Opaque

Data
====
key:  32 bytes

Name:         storageos-volume-key-bc175830-3ed4-4d0c-90eb-df070b9930eb
Namespace:    encrypted
Labels:       app.kubernetes.io/component=storageos-api-manager
              app.kubernetes.io/managed-by=storageos-operator
              app.kubernetes.io/name=storageos
              app.kubernetes.io/part-of=storageos
              storageos.com/pvc=encrypted-2
Annotations:  <none>

Type:  Opaque

Data
====
hmac:  32 bytes
iv:    32 bytes
key:   64 bytes
vuk:   80 bytes

# Review "storageos-volume-key-$RANDOM".
# Notice that there are 4 data objects stored in the secret > "hmac", "iv", "vuk", "iv" respectively.
kubectl get secrets storageos-volume-key-bc175830-3ed4-4d0c-90eb-df070b9930eb --namespace=encrypted --output=yaml

apiVersion: v1
data:
  hmac: K26qP2LJF0GnLBLOSPmGlLatPGhLnW4l6LUFapnlKnI=
  iv: zDrINiKUNS8CAZfOPPzXxDRjdQETR6weTgN8S6irmzQ=
  key: 9rXsIPru1P9X6C+PGBRuGIH5pdRh1Y5LZ4psLjyzy0oyM/yylqUIG9Ez8XBkOW8pCrwY303TC6aBAOdlzEjJtw==
  vuk: CokhOv1bVPySm/afmpZA9RHSTm2uWd2cPD2WGbw+xB7xAmPx3NqKws2yFcLAf2370K3Sh/iHMeAfNMQoMPWVUZdqIsZcSnUcvtfd1YOHZv8=
kind: Secret
metadata:
  creationTimestamp: "2022-07-25T13:41:05Z"
  labels:
    app.kubernetes.io/component: storageos-api-manager
    app.kubernetes.io/managed-by: storageos-operator
    app.kubernetes.io/name: storageos
    app.kubernetes.io/part-of: storageos
    storageos.com/pvc: encrypted-2
  name: storageos-volume-key-bc175830-3ed4-4d0c-90eb-df070b9930eb
  namespace: encrypted
  resourceVersion: "19756"
  uid: bb793310-0ad9-46d1-a590-34556dffe3a6
type: Opaque

The last step will be to try to read the data on the node where the encrypted volume resides.

# Get the node locations where "encrypted-2-0" pod is running.
kubectl get pods --namespace=encrypted --output=wide

NAME            READY   STATUS    RESTARTS   AGE     IP          NODE                 NOMINATED NODE   READINESS GATES
encrypted-2-0   1/1     Running   0          4m46s   10.42.4.7   demo-worker-node-5   <none>           <none>

# Use "kubectl debug" to temporarily run a privileged container on the node where "encrypted-2-0" is located.
kubectl debug node/demo-worker-node-5 -it --image=ubuntu:latest

# Through the privileged container, update the repository index and install "binutils" to access the "strings" utility.
apt update && apt install --yes binutils

# Navigate to where data is being stored as blob files on the node and list the files.
cd /host/var/lib/storageos/data/dev1/
ls -lah

total 340M
drwxr-xr-x 2 root root 4.0K Jul 25 13:41 .
drwxr-xr-x 4 root root 4.0K Jul 25 13:41 ..
-rw------- 1 root root  42M Jul 25 13:48 deployment.b9ea560c-5fc2-4f6f-9f43-c2d33483ed89.0.blob
-rw------- 1 root root  42M Jul 25 13:49 deployment.b9ea560c-5fc2-4f6f-9f43-c2d33483ed89.1.blob

# Use the "strings" utilty to try and read the content of the blob files in the directory.
# Note - the output will return multiple strings of random, unreadable characters as the content is encrypted.
strings *.blob | head -10

n.@LG
c%rs/
r`R;
|'U%
E"k9^
*59G
#q={
uOb
JC#rtb
]=?b

# exit from the pod and return to your local shell
exit

How To Enable Data Encryption For Volumes

Overview

Example - Enable Volume Data Encryption Through a PersistentVolumeClaim Definition

Example - Enable Volume Data Encryption Through a StorageClass Definition

Example - Enable Volume Data Encryption Through a `PersistentVolumeClaim` Definition

Example - Enable Volume Data Encryption Through a `StorageClass` Definition