Compliant Backups With Minio and Restic

This articles outlines a backup strategy that leverages the S3-compatible object storage Minio and the backup tool restic. The main focus is to be able to recover from Ransomware and similar attacks on the host that is backed up. As positive side-effect the data storage provides, according to Minio, key data retention compliances and meets SEC17a-4(f), FINRA 4511(C), and CFTC 1.31(c)-(d).

The main motivation for this article is to outline step by step the setup process how to configure Minio in such a way that it can be used by restic. When the host that runs the periodic backup via restic is compromised older backups can be recovered with the help of Minio’s versioning, object locking and retention.

The setup is self-contained and outlines all the necessary configurations and commands that need to be run. For a production deployment setup Minio on a 24/7 running host, virtual or physical, or run Minio in Kubernetes. Change all the password to something secure and random. Do not reuse passwords.

Requirements for the Minio Bucket

  • Bucket name: restic
  • Bucket policy (see JSON file below)
    • Versioning enable
    • Object locking and retention enabled. Mode: COMPLIANCE, not even root user can remove files. Retention of 30 days No legal hold

Store the following JSON content under configs/restic-policy.json

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "s3:*"
        ],
        "Resource": [
          "arn:aws:s3:::restic/locks",
          "arn:aws:s3:::restic/locks/*"
        ]
      },
      {
        "Effect": "Allow",
        "Action": [
          "s3:ListBucket",
          "s3:GetObject",
          "s3:PutObject"
        ],
        "Resource": [
          "arn:aws:s3:::restic/*"
        ]
      }
    ]
}

Store the following YAML content under docker-compose.json. Please make sure to change the root username and password in production.

services:
  minio:
    image: minio/minio:latest
    ports:
      - "9000:9000"
      - "9099:9099"
    environment:
      MINIO_ROOT_USER: minioadmin
      MINIO_ROOT_PASSWORD: minioadmin
    volumes:
      - storage-minio:/data
    command: server --address ":9099" --console-address ":9000" /data
    restart: always # necessary since it's failing to start sometimes

volumes:
  storage-minio:
    external: true

Setup Minio for Restic

Start Minio Server via docker-compose and configure mc (minio client).

$ docker volume create storage-minio
$ docker-compose up -d
$ mc alias set myminio http://127.0.0.1:9099 minioadmin minioadmin

Navigate to the web interface http://127.0.0.1:9000 to validate if the bucket was created and setup correctly, see screenshots below for reference.

Create and configure restic bucket.

$ mc mb --with-lock myminio/restic
$ mc version enable myminio/restic
$ mc retention set --recursive --default COMPLIANCE "30d" myminio/restic

Bucket Configuration

Setup a new user restic-user that is used by restic to interact with Minio.

$ mc admin user add myminio restic-user
$ mc admin policy add myminio restic-policy configs/restic-policy.json
$ mc admin policy set myminio restic-policy user=restic-user

$ mc admin user policy myminio restic-user

# Substitute 'password' with the password you typed when creating the restic-user
$ mc alias set myminio http://127.0.0.1:9099 restic-user password

Restic User in Minio

Restic Policy in Minio

Setup Restic Repository

$ export AWS_ACCESS_KEY_ID=restic-user
$ export AWS_SECRET_ACCESS_KEY=password
$ export RESTIC_REPOSITORY=s3:http://127.0.0.1:9099/restic
$ export RESTIC_PASSWORD=v3r3s3cr3t
$ restic init
created restic repository bf67082884 at s3:http://127.0.0.1:9099/restic

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

$ restic backup ~/tmp/minio-backup/screenshots
repository bf670828 opened (repository version 2) successfully, password is correct
no parent snapshot found, will read all files

Files:           3 new,     0 changed,     0 unmodified
Dirs:            5 new,     0 changed,     0 unmodified
Added to the repository: 155.661 KiB (143.404 KiB stored)

processed 3 files, 152.570 KiB in 0:00
snapshot 0c0baa66 saved

Checkout all the versions. Note that all the lock files that are normally deleted are only marked as deleted.

$ mc ls --versions --recursive myminio/restic
[2022-09-10 18:00:34 EDT]     0B c5bd96f7-900f-433a-9423-9fb989b57ff0 v1 DEL locks/05813fcbf256674bbe72093206be9358ff3c0b1fc3edd63bc77d1a84706d8315
[2022-09-10 18:02:14 EDT]     0B 081491a1-13c7-46be-acb0-15c04d592498 v1 DEL locks/4a0b609f52a09d56b612f0a40c26ae500b9abc15e3a8e7d6fac8c33805cdba5b
[2022-09-10 18:02:42 EDT]     0B 81d67137-5c2c-4f9c-9e5d-03bbc36d3966 v1 DEL locks/827e8736ebf1e0f4f7fd9fa6965ee6642244bfec569dda6e1a8ec01e81049ce2
[2022-09-10 18:01:10 EDT]     0B 9098058d-271a-4669-a540-1b83273e3ba7 v1 DEL locks/8849d86b3debbc1ff866511354b41e4da1b713606d8b4b7cd7d1c1d6befb5c76
[2022-09-10 18:02:45 EDT]     0B 8130d07a-5e6b-4d67-abfb-319600356802 v1 DEL locks/a24c52e857073d2d6a735a21fa03e275ef1d1bb462e83c5e5bf4b24176d64b5f
[2022-09-10 18:00:29 EDT]     0B da9ddf85-63c9-4ed1-9c7e-381a7192d301 v1 DEL locks/ad90870cdb2f32764e3aa872fd66ffd5fce1524ef09ae3a2dc656aadfdbff05e
[2022-09-10 18:00:41 EDT]     0B 19d2fc8f-2173-4bc1-a49d-779062da716d v1 DEL locks/c130734e4a35639da582bf97d7450032ea9cc6c5f2552a7ff4e5705b045c8cd1
[2022-09-10 18:00:38 EDT]     0B 2b6d24d8-dd1b-4171-97b9-64b12e4a2000 v1 DEL locks/f1f03e5434863197e9fb4d956691a32fb2a2ab0264d21fa49042c72c244d4148
[2022-09-10 17:59:52 EDT]   155B STANDARD ecd50cce-cd5b-43a1-b2e5-20bfd29b1cbc v1 PUT config
[2022-09-10 18:01:10 EDT] 2.0KiB STANDARD 9eaee64e-e4ae-4892-b07e-531bdb836fc7 v1 PUT data/44/445b9bd9e00f0d23af901dad902f547495fe274c01f781cb3604f8f38f531116
[2022-09-10 18:01:10 EDT] 141KiB STANDARD 4178cb76-5aca-4b05-82fc-d2372a164741 v1 PUT data/e5/e55d306aef149e33d8de348f555e62e376f502965afec19baba6c2cb03999f3b
[2022-09-10 18:01:10 EDT]   657B STANDARD bfe3c346-b24d-42b3-a3ec-cd833c65255b v1 PUT index/d6961c4108f0c058c1f93f6275a4e370656a583a7f82fe3d19dfade3d4c65036
[2022-09-10 17:59:52 EDT]   449B STANDARD 20fb6cb4-15d6-4c6c-a3f9-06c9d520e41b v1 PUT keys/bce79b667d3ae56f5add9d5f42eeb9dea92ee39e967c73d4250ba8a9cfc2e9cd
[2022-09-10 18:00:34 EDT]   156B STANDARD 1bb80247-11c3-4ad5-8061-ecbdd42b0023 v1 PUT locks/05813fcbf256674bbe72093206be9358ff3c0b1fc3edd63bc77d1a84706d8315
[2022-09-10 18:02:14 EDT]   156B STANDARD c1bbed80-12c3-45c4-9e44-d60b839923e6 v1 PUT locks/4a0b609f52a09d56b612f0a40c26ae500b9abc15e3a8e7d6fac8c33805cdba5b
[2022-09-10 18:02:38 EDT]   156B STANDARD 9aea42ee-e027-4e2b-b5c9-3dfb894a9203 v1 PUT locks/827e8736ebf1e0f4f7fd9fa6965ee6642244bfec569dda6e1a8ec01e81049ce2
[2022-09-10 18:01:10 EDT]   157B STANDARD dd9c26d9-d4eb-4f8f-8095-f0f9d6b79529 v1 PUT locks/8849d86b3debbc1ff866511354b41e4da1b713606d8b4b7cd7d1c1d6befb5c76
[2022-09-10 18:02:45 EDT]   156B STANDARD e806a9e7-31e8-4829-aada-641a288a3545 v1 PUT locks/a24c52e857073d2d6a735a21fa03e275ef1d1bb462e83c5e5bf4b24176d64b5f
[2022-09-10 18:00:29 EDT]   157B STANDARD 60b8f1a3-1cf4-4ac1-af05-5797f4a062de v1 PUT locks/ad90870cdb2f32764e3aa872fd66ffd5fce1524ef09ae3a2dc656aadfdbff05e
[2022-09-10 18:00:41 EDT]   153B STANDARD a8ba7512-b66c-49ef-bf91-c8c34157478a v1 PUT locks/c130734e4a35639da582bf97d7450032ea9cc6c5f2552a7ff4e5705b045c8cd1
[2022-09-10 18:00:38 EDT]   155B STANDARD 21906787-c180-46e8-9618-91ad9538581e v1 PUT locks/f1f03e5434863197e9fb4d956691a32fb2a2ab0264d21fa49042c72c244d4148
[2022-09-10 18:01:10 EDT]   223B STANDARD 458a5607-08e1-49fa-a0fe-7251af5a7b2a v1 PUT snapshots/0c0baa666925e8eb2fd669b74da444138d7ca9952da5091ec354bf735427ff7a

$ mc ls --recursive myminio/restic
[2022-09-10 17:59:52 EDT]   155B STANDARD config
[2022-09-10 18:01:10 EDT] 2.0KiB STANDARD data/44/445b9bd9e00f0d23af901dad902f547495fe274c01f781cb3604f8f38f531116
[2022-09-10 18:01:10 EDT] 141KiB STANDARD data/e5/e55d306aef149e33d8de348f555e62e376f502965afec19baba6c2cb03999f3b
[2022-09-10 18:01:10 EDT]   657B STANDARD index/d6961c4108f0c058c1f93f6275a4e370656a583a7f82fe3d19dfade3d4c65036
[2022-09-10 17:59:52 EDT]   449B STANDARD keys/bce79b667d3ae56f5add9d5f42eeb9dea92ee39e967c73d4250ba8a9cfc2e9cd
[2022-09-10 18:01:10 EDT]   223B STANDARD snapshots/0c0baa666925e8eb2fd669b74da444138d7ca9952da5091ec354bf735427ff7a

Does it work?

In order to test if it works we can use restic forget to try to remove the backup we just created. If everything works this command should fail.

$ restic snapshots
repository bf670828 opened (repository version 2) successfully, password is correct
ID        Time                 Host        Tags        Paths
-----------------------------------------------------------------------------------------------
0c0baa66  2022-09-10 18:01:10  myhost                  /home/user/tmp/minio-backup/screenshots
-----------------------------------------------------------------------------------------------
1 snapshots

$ restic forget 0c0baa66
repository bf670828 opened (repository version 2) successfully, password is correct
Remove(<snapshot/0c0baa6669>) returned error, retrying after 552.330144ms: client.RemoveObject: Access Denied.
Remove(<snapshot/0c0baa6669>) returned error, retrying after 1.080381816s: client.RemoveObject: Access Denied.
Remove(<snapshot/0c0baa6669>) returned error, retrying after 1.31013006s: client.RemoveObject: Access Denied.
  signal interrupt received, cleaning up
unable to remove <snapshot/0c0baa6669> from the repository
[0:02] 0.00%  0 / 1 files deleted

It works!

Author

Alex Oberhauser

Alex Oberhauser is a tech-entrepreneur, innovator and former C-level executive. He is currently working on user controlled identities and the empowerment of the end-users, with privacy and security as part of the value proposition, not as an afterthought.