Module 6: Backup & Maintenance

Learning Objectives

By the end of this module, you will have:

• ✅ Understood the automated backup system
• ✅ Checked backup status and logs
• ✅ Downloaded backups to your local computer
• ✅ Learned how to restore from backups
• ✅ Created manual backups when needed

Estimated Time: 20-30 minutes

Prerequisites

Before starting this module, make sure you have:

• ✅ EvoNEST running in production mode (using docker-compose.yml)
• ✅ Basic understanding of Docker commands
• ✅ Terminal/command prompt access

Development vs Production

The development environment (docker-compose.dev.yml) does NOT include automated backups. This module applies to production deployments only.

Overview

EvoNEST includes a comprehensive automated backup system that:

• Creates daily database backups automatically
• Organizes backups into daily, weekly, and monthly archives
• Applies retention policies to manage disk space
• Compresses backups for efficient storage

In this module, you'll learn how to monitor, download, and restore your data.

Understanding the backup system

Automated backup schedule

When running in production mode, EvoNEST automatically backs up your database:

Frequency	Schedule	Retention
Daily	Every night at 00:00 (midnight)	7 days
Weekly	Every Sunday	4 weeks
Monthly	1st of each month	12 months

Backup storage structure

Backups are stored in Docker volumes with this organization:

/backups/
├── daily/              # Last 7 days of backups
│   ├── mongodb_backup_20241024_000000.tar.gz
│   ├── mongodb_backup_20241025_000000.tar.gz
│   └── ...
├── weekly/             # Last 4 weeks (Sundays only)
│   ├── week_42_mongodb_backup_20241020_000000.tar.gz
│   └── ...
├── monthly/            # Last 12 months (1st of month)
│   ├── month_10_mongodb_backup_20241001_000000.tar.gz
│   └── ...
└── logs/               # Backup operation logs
    ├── backup_20241024_000000.log
    ├── latest_backup.txt
    └── backup_status.txt

What gets backed up?

✅ Included in backups:

• All database collections (samples, traits, experiments, users, etc.)
• Database indexes
• User accounts and permissions
• Configuration settings

❌ NOT included in backups:

• Uploaded files (images, documents) stored in /file_storage
• Docker container configurations
• Environment variables

File Storage Backups

Uploaded files are stored separately in the file_storage Docker volume. To back up files, you'll need to copy this volume separately (covered later in this module).

Step 1: check backup status

1.1 Verify backup container is running

Open your terminal and check that the backup container is active:

docker ps

Expected output:

CONTAINER ID   IMAGE              COMMAND                  STATUS
abc123def456   evonest_backup     "crond -f -l 8"         Up 2 days
...

Look for a container with evonest_backup or mongo_backup in the name.

Backup container not running?

If you don't see the backup container:

1. Check if you're using the production docker-compose file:

   docker compose -f docker-compose.yml ps

2. If the container is stopped, start it:

   docker compose -f docker-compose.yml up -d backup

3. Check logs for errors:

   docker logs mongo_backup

1.2 Check last backup status

View the status file to see when the last backup completed:

docker exec mongo_backup cat /backups/backup_status.txt

Expected output:

Last successful backup: Thu Oct 24 00:00:15 UTC 2024
Backup file: mongodb_backup_20241024_000015.tar.gz
Backup size: 15M

1.3 View detailed backup information

Check the latest backup details:

docker exec mongo_backup cat /backups/logs/latest_backup.txt

Expected output:

Backup completed at Thu Oct 24 00:00:15 UTC 2024

Backup details:
- File: mongodb_backup_20241024_000015.tar.gz
- Size: 15M
- Available space: 450G

To download via SSH:
scp user@your_server:/var/lib/docker/volumes/mongo_backups/_data/daily/mongodb_backup_20241024_000015.tar.gz /local/path/

Backup rotation policy:
- Daily backups kept for 7 days
- Weekly backups kept for 4 weeks
- Monthly backups kept for 12 months

1.4 List all available backups

Daily backups:

docker exec mongo_backup ls -lh /backups/daily/

Weekly backups:

docker exec mongo_backup ls -lh /backups/weekly/

Monthly backups:

docker exec mongo_backup ls -lh /backups/monthly/

Expected output example:

total 75M
-rw-r--r--  1 root  15M Oct 24 00:00 mongodb_backup_20241024_000015.tar.gz
-rw-r--r--  1 root  14M Oct 23 00:00 mongodb_backup_20241023_000012.tar.gz
-rw-r--r--  1 root  14M Oct 22 00:00 mongodb_backup_20241022_000008.tar.gz
...

Step 2: Download backups locally

It's strongly recommended to keep local copies of your backups on a separate computer or external drive.

Method 1: copy from Docker volume (local server)

If EvoNEST is running on your local computer:

1. Find the latest backup filename:

docker exec mongo_backup ls /backups/daily/ | tail -1

2. Copy the backup to your current directory:

docker cp mongo_backup:/backups/daily/mongodb_backup_20241024_000015.tar.gz ./

Replace mongodb_backup_20241024_000015.tar.gz with the actual filename from step 1.

3. Verify the download:

# Windows (PowerShell)
dir mongodb_backup_*.tar.gz

# macOS/Linux
ls -lh mongodb_backup_*.tar.gz

Method 2: Download from remote server (SSH)

If EvoNEST is running on a remote server:

1. Get the backup filename from the server:

ssh user@your-server "docker exec mongo_backup ls /backups/daily/ | tail -1"

2. Find the Docker volume path on the server:

ssh user@your-server "docker volume inspect mongo_backups --format '{{.Mountpoint}}'"

Expected output:

/var/lib/docker/volumes/mongo_backups/_data

3. Download using scp:

scp user@your-server:/var/lib/docker/volumes/mongo_backups/_data/daily/mongodb_backup_20241024_000015.tar.gz ~/EvoNEST/backups/

Permissions

You may need sudo privileges on the server to access Docker volumes directly. If you get permission errors, ask your server administrator for help.

Method 3: Schedule automated downloads

For production servers, consider setting up automated backup downloads using:

• rsync with cron (Linux/macOS)
• Task Scheduler with PowerShell scripts (Windows)
• Cloud backup services (AWS S3, Google Drive, Dropbox)

Best Practice

Download backups to at least two locations:

1. Your local computer
2. An external drive or cloud storage

This protects against both server failures and local computer failures.

Step 3: Restore from backup

When to restore

You might need to restore from backup if:

• Data was accidentally deleted
• Database corruption occurred
• You need to revert to an earlier state
• Migrating to a new server

Warning

Restoring from backup will overwrite all current data. Make sure you have a current backup before restoring!

3.1 Prepare for restoration

1. Stop the EvoNEST application (but keep database running):

docker stop evonest_backbone_prod

2. Verify you have the backup file:

# If restoring from a backup inside Docker:
docker exec mongo_backup ls /backups/daily/mongodb_backup_20241024_000015.tar.gz

# If restoring from a local file, copy it into the container:
docker cp mongodb_backup_20241024_000015.tar.gz mongo_backup:/backups/

3.2 Extract the backup

1. Enter the backup container:

docker exec -it mongo_backup sh

2. Extract the backup archive:

cd /backups
tar -xzf daily/mongodb_backup_20241024_000015.tar.gz

This creates a directory named temp_mongodb_backup_20241024_000015 with the database dump.

3.3 Restore to MongoDB

Still inside the backup container, run the restore command:

mongorestore --uri="mongodb://root:pass@mongo:27017" --drop /backups/temp_mongodb_backup_20241024_000015

Command explanation:

• --uri: Connection string to MongoDB (uses container network)
• --drop: Drops existing collections before restoring (ensures clean restore)
• Last argument: Path to the extracted dump directory

Expected output:

preparing collections to restore from
reading metadata for evonest.samples from /backups/temp_mongodb_backup_20241024_000015/evonest/samples.metadata.json
reading metadata for evonest.traits from /backups/temp_mongodb_backup_20241024_000015/evonest/traits.metadata.json
...
finished restoring evonest.samples (120 documents, 0 failures)
finished restoring evonest.traits (450 documents, 0 failures)
...
120 document(s) restored successfully. 0 document(s) failed to restore.

3.4 Clean up and restart

1. Exit the container:

exit

2. Remove the temporary extracted files:

docker exec mongo_backup rm -rf /backups/temp_mongodb_backup_20241024_000015

3. Restart the EvoNEST application:

docker start evonest_backbone_prod

4. Verify the restoration:

• Open EvoNEST in your browser: http://localhost:3000
• Log in and check that your data has been restored
• Verify sample counts, recent entries, etc.

Verify Before and After

Before restoring, note down:

• Number of samples, traits, experiments
• Recent entries

After restoring, verify these match the backup date.

Step 4: Create manual backups

Sometimes you may want to create a backup immediately (not waiting for the scheduled backup):

Manual backup via backup container

1. Trigger a manual backup:

docker exec mongo_backup /backup_scripts/backup.sh

2. Check the new backup was created:

docker exec mongo_backup ls -lt /backups/daily/ | head -5

3. View the backup log:

docker exec mongo_backup cat /backups/logs/latest_backup.txt

Quick manual backup (alternative method)

If you need a quick backup without the container:

1. Create a backup using mongodump directly:

docker exec evonest_mongodb mongodump --uri="mongodb://root:pass@localhost:27017" --out=/tmp/manual_backup

2. Copy the backup out of the container:

docker cp evonest_mongodb:/tmp/manual_backup ./manual_backup_$(date +%Y%m%d_%H%M%S)

3. Compress the backup:

# macOS/Linux
tar -czf manual_backup_$(date +%Y%m%d_%H%M%S).tar.gz manual_backup_*

# Windows (PowerShell)
Compress-Archive -Path manual_backup_* -DestinationPath "manual_backup_$(Get-Date -Format 'yyyyMMdd_HHmmss').zip"

Step 5: Backup file storage

Remember that database backups do NOT include uploaded files (images, documents). Back these up separately:

Check file storage size

1. Check the file storage volume:

docker exec evonest_backbone_prod du -sh /usr/evonest/file_storage

Expected output:

2.3G    /usr/evonest/file_storage

Backup file storage

1. Copy the entire file storage directory:

docker cp evonest_backbone_prod:/usr/evonest/file_storage ./file_storage_backup_$(date +%Y%m%d)

2. Compress the backup:

# macOS/Linux
tar -czf file_storage_backup_$(date +%Y%m%d).tar.gz file_storage_backup_*

# Windows (PowerShell)
Compress-Archive -Path file_storage_backup_* -DestinationPath "file_storage_backup_$(Get-Date -Format 'yyyyMMdd').zip"

3. Store the backup safely:

• Move to external drive
• Upload to cloud storage
• Keep alongside database backups

Restore file storage

If you need to restore files:

1. Extract the backup:

tar -xzf file_storage_backup_20241024.tar.gz

2. Copy files back to the container:

docker cp file_storage_backup_20241024/. evonest_backbone_prod:/usr/evonest/file_storage/

3. Fix permissions (if needed):

docker exec evonest_backbone_prod chown -R node:node /usr/evonest/file_storage

Checkpoint: verify backup system

Before finishing this module, make sure you can:

• ✅ View backup status and logs
• ✅ List available daily, weekly, and monthly backups
• ✅ Download a backup to your local computer
• ✅ Understand how to restore from backup
• ✅ Create a manual backup when needed
• ✅ Back up file storage separately

All good?

If you can check all boxes above, your backup system is working correctly!

Best practices

1. Regular backup verification

Monthly checklist:

• Verify backups are being created automatically
• Check backup logs for errors
• Download latest backup to local storage
• Test restoration process (on test environment)

2. Backup storage strategy

Implement the 3-2-1 backup rule:

• 3 copies of your data
• 2 different storage media
• 1 copy off-site

Example setup:

1. Automated backups on server (Docker volume)
2. Weekly downloads to office computer
3. Monthly uploads to cloud storage

3. Monitor disk space

If your database grows large, backups can fill up disk space:

1. Check available space:

docker exec mongo_backup df -h /backups

2. Adjust retention if needed:

Edit docker-compose.yml:

backup:
  environment:
    DAILY_RETENTION: "5" # Reduce from 7 to 5 days
    WEEKLY_RETENTION: "3" # Reduce from 4 to 3 weeks
    MONTHLY_RETENTION: "6" # Reduce from 12 to 6 months

3. Restart the backup container:

docker compose up -d backup

4. Test your backups

Regularly test restoration on a separate test environment:

# Create a test docker-compose for restoration testing
# Use different ports to avoid conflicts
# Restore backup and verify data integrity

Don't Wait for a Disaster

Many people never test their backups until they need them. Don't be one of them! Test restoration at least once per quarter.

Troubleshooting

Backup container not starting

Problem: docker ps doesn't show mongo_backup container

Solutions:

1. Check if you're using production docker-compose:

   docker compose -f docker-compose.yml up -d

2. Check Docker logs:

   docker logs mongo_backup

3. Verify backup service is defined in docker-compose.yml

No backups being created

Problem: /backups/daily/ is empty

Solutions:

1. Check cron is running:

   docker exec mongo_backup ps aux | grep crond

2. Manually trigger backup to see error:

   docker exec mongo_backup /backup_scripts/backup.sh

3. Check MongoDB connection:

   docker exec mongo_backup mongodump --uri="mongodb://root:pass@mongo:27017" --out=/tmp/test

"Permission denied" when downloading backups

Problem: Can't access Docker volumes on remote server

Solutions:

1. Use docker cp instead of direct file access:

   docker cp mongo_backup:/backups/daily/latest_backup.tar.gz ./

2. Ask administrator for sudo access

3. Set up a backup export script on the server

Restore failed or incomplete

Problem: mongorestore shows errors

Solutions:

1. Verify backup file is not corrupted:

   tar -tzf backup_file.tar.gz | head

2. Check MongoDB is running:

   docker ps | grep mongo

3. Use --drop flag to clear existing data:

   mongorestore --uri="..." --drop /path/to/dump

4. Check MongoDB logs:

   docker logs evonest_mongodb

Summary

In this module, you learned:

• Automated backup system: Daily, weekly, and monthly backups with retention policies
• Monitoring backups: Check status, view logs, and list available backups
• Downloading backups: Copy backups locally using Docker commands or SSH
• Restoring data: Extract and restore from backup archives
• Manual backups: Create on-demand backups when needed
• File storage: Separately back up uploaded files and images
• Best practices: 3-2-1 rule, regular testing, monitoring disk space

Remember: Your backup system is only as good as your last successful test restore!

Congratulations!

You have completed the EvoNEST tutorial! You have learned how to install EvoNEST, how to configure it, how to insert new samples, experiments and traits, and how to make sure your data is safe.

Next steps

If you want to keep exploring, you can:

• Learn how to fix common problems: Troubleshooting
• Explore advanced features: User documentation
• Import existing data: Data Import Guide
• Analyze your data: Data Analysis Guide