Configure automatic backups for Discourse

This guide explains how to configure automatic backups for Discourse, including storage options on local servers and S3-compatible storage.

Learn how to set up automatic backups for your Discourse platform.

This guide covers configuring automatic backups, storing them on local servers or S3-compatible storage, and managing storage retention options like Amazon Glacier.

Configuring automatic backups

1. Navigate to /admin settings.
2. Select the Backup section.
3. Set backup_frequency to the desired interval in days. The default is 7 (weekly). Set to 1 for daily backups, or 0 to disable automatic backups. The maximum is 30.

backup_frequency100%75%50%

Additional backup settings

• backup_time_of_day — the time of day (UTC) when backups run. Default: 3:30.
• backup_with_uploads — include uploads in scheduled backups. Default: enabled. Disabling this will only back up the database.
• maximum_backups — the maximum number of backups to keep. Older backups are automatically deleted. Default: 5.
• remove_older_backups — remove backups older than the specified number of days. Leave blank to disable.

Store backups on the local server

By default, backups are stored on your local server. For self-hosted instances, access them at /var/discourse/shared/standalone/backups/default.

Store backups on S3-compatible storage

Using the admin panel

1. Create an S3 bucket.
2. Set the s3_backup_bucket in the admin panel.

• Follow steps in setting up S3 uploads.

3. Configure s3_access_key_id, s3_secret_access_key, and s3_region.
4. Set backup_location to “S3”.

S3 configuration854×202 15.8 KB

WARNING

Storing backups and regular uploads in the same bucket and folder is no longer supported and will not work.

The s3_backup_bucket path should only be used for backups. If you need to use a bucket that contains other files please make sure that you provide a prefix when you configure the s3_backup_bucket setting (example: my-awesome-bucket/backups) and make sure that files with that prefix are private.

From now on all backups will be uploaded to S3 and not be stored locally anymore. Local storage will only be used for temporary files during backups and restores.

Go to the Backups tab in the admin dashboard to browse the backups – you can download them any time to do a manual offsite backup.

image807×287 11.7 KB

Using environment variables in app.yml

You can also configure S3 backups using environment variables in app.yml. For more information, see Configure an S3 compatible object storage provider for uploads

Note that the above article covers app.yml covers S3 setup for backups and for file/image uploads. If you only want to use S3 for backups (and not for uploads of files/images), then you can omit the following parameters from your app.yml config:

• DISCOURSE_USE_S3
• DISCOURSE_S3_CDN_URL
• DISCOURSE_S3_BUCKET

You also don’t need to configure the after_assets_precompile step in this case, nor configure a CDN.

Be sure to include all other parameters that are required for your storage provider, as mentioned in the article. Here is one example configuration that only activates S3 for backups (for Scaleway S3):

DISCOURSE_S3_REGION: nl-ams
DISCOURSE_S3_ENDPOINT: https://s3.nl-ams.scw.cloud
DISCOURSE_S3_ACCESS_KEY_ID: my_access_key
DISCOURSE_S3_SECRET_ACCESS_KEY: my_secret_access_key
DISCOURSE_S3_BACKUP_BUCKET: my_bucket/my_folder
DISCOURSE_BACKUP_LOCATION: s3

Archiving to storage with a lower cost

Note that on AWS S3, you can also enable an automatic move to Glacier bucket lifecycle rule to keep your S3 backup costs low. Other storage providers often have a similar offering.

Last edited by @SaraDev 2024-11-07T20:36:45Z

Check document

Perform check on document:

You are able to Archive Backups from your S3 Bucket to Glacier.
It is cheaper, but an Restore attemps more Time.

This Site will Help you to reduce Backup costs.:

Setting this up can be rather confusing. Here’s a simple guide to help you out.

• Log into your Discourse admin panel
• Configure daily backups
• Set maximum backups to 7
• Log into your Amazon Web Services account
• Go in the S3 Dashboard
• Open the bucket containing the backups
• Click on the properties tab
• Activate versioning
• Open the Lifecycle menu
• Add a rule for the whole bucket
• Set current version to expire after 15 days
• Set previous version to
• Archive to Glacier after 1 days
• expire after 91 days
• Save and logout

How it works

Versioning will keep backups automaticly deleted by Discourse. One day after beeing deleted it will be moved to the Glacier storage. After 91 days it will be delete from the Glacier storage.

Warning

Amazon charge you for item stored in Glacier for 90 days even if you delete them before. Make sure your Glacier Lyfecicle keep your file at least 90 days.

I may have missed this, but how to I make sure a bucket for backups is private the proper way? Setting up file and image uploads to S3 doesn’t seem to be described here.

Looks like there is no such option anymore.

UPD: ah, now it’s split into 2 steps of this wizard.

So I guess it should be like this:

image646×595 78.2 KB

image636×550 49.2 KB

image682×504 31 KB

Does S3 backup functionality play nice without IAM access key and secret if using AWS roles assigned to the instance and s3 use iam profile is enabled in the settings menu?

Edit: The answer is YES! Just ensure you have region set appropriately.

I had this problem too, which was resolved by adding a couple of lines to the policy document:

"Resource": [
        "arn:aws:s3:::your-uploads-bucket",
        "arn:aws:s3:::your-uploads-bucket/*",
        "arn:aws:s3:::your-backups-bucket",
        "arn:aws:s3:::your-backups-bucket/*"
      ]

Should this be added to the OP (which isn’t a wiki)? Or perhaps in the OP of Set up file and image uploads to S3?

Is there a good reason not to make daily backups aren’t the default?

Fixes needed for OP

Add something about moving S3 backups to Glacier?
Delete the S3 stuff and link to Configure an S3 compatible object storage provider for uploads

It’d be overkill and expensive. Same reason we don’t recommend changing the oil in your car every 500 miles?

You back up your servers once a week?

I think your 500 mile oil change analogy would be a good answer for “why not keep 30 backups?”, but it doesn’t make sense to me here.

Do you not think that if you’re going to have 5 backups most people would rather have backups every day so that if they do have to revert to a backup they’ve lost no more than a day? The only time I remember an older backup being useful is when images fell out of tombstone.

If they have so little disk space that they can’t hold 5 backups, it would seem better to learn that in 5 days when they still have some memory of what they did than wait 4-5 weeks.

Yes, that is correct, for my self hosted servers.

If you prefer different settings, feel free to edit them from the defaults. What’s stopping you from doing so?

Well, I’ll be!

I’m cleaning up topics like this one. If the defaults were changed, it wouldn’t be needed!

I’m convinced that the defaults won’t be changed, so I’ll move ahead. :wink

Once a week is a good starting point and a solid default.

I had the very same problem!

i suggest to add a note regarding this on the initial post too!

Would be nice if this could be used to upload to a different S3 compatible provider, such as running MinIO.

If you want to use minio, see Using Object Storage for Uploads (S3 & Clones). If you want different services for backups and assets then you’re out of luck (though there are ways to set up triggers to copy from one bucket 8 another)

No, just want for backups, so they get automatically pushed to a different machine on my network.

Then the guide I linked is what you’re looking for.

Is it possible to be more frequent than once per day? I’d rather not make assumptions and set backup freqeuncy to a float

If you want more frequent backups you’ll need to do them externally.

Thanks for the hint. That pushed me towards the command line option which we can schedule to do whenever:

I got this to work, but it seems that the uploads checkbox was not really needed, nor do I understand the purpose. What is the purpose? The only thing I want is backups to s3 instead of local for my server. The server only has weekly automatic backups…

The Json also had problems… I was able to get it to work using another website reference. However, nobody could upload any images because I had the uploads checkbox checked (as described here)…Unchecking that box fixed the image upload problem for users and their profile pics.

What is the purpose of the images upload? I’m seriously hoping images are inside the s3 backups.
I had to do the instructions twice because I didn’t understand “uploads” and only made one bucket. Then I had to do it again with 2 buckets, and then I had to remove the checkbox for uploads. It might be good if there was a separate more simple topic for s3 backups … and only backups.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:List*",
                "s3:Get*",
                "s3:AbortMultipartUpload",
                "s3:DeleteObject",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:PutObjectVersionAcl",
                "s3:PutLifecycleConfiguration",
                "s3:CreateBucket",
                "s3:PutBucketCORS"
            ],
            "Resource": [
                "arn:aws:s3:::classicaltheravadabucket",
                "arn:aws:s3:::classicaltheravadabucket/*",
                "arn:aws:s3:::classicaltheravadabackupbucket",
                "arn:aws:s3:::classicaltheravadabackupbucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:*"
            ],
            "Resource": "*"
        }
    ]
}

Though I think that topic should be updated to recommend that the S3 configuration be moved to app.yml rather than the database, that way you can do a command line restore of the database with only the yml file and not have to configure it with a user and s3 confige before doing a restore.

I’m not sure what you are talking about . My backups are working see picture
.
I use s3 because digital ocean backups are only weekly, and if the server crashes and gets deleted, it is not useful.
On the other hand. I’m hoping that restoring from s3 or downloaded s3 bucket will be fine.
I’m not uploading the images, and I am hoping that the s3 backups are being backed up including images (although very few).

image1391×394 76.6 KB

Generally: no.
It does not make much sense to backup images in an S3 bucket into another S3 bucket?

can you be less ambiguous?
The instructions had 2 s3 buckets. I could not get that to work.
I have only one s3 bucket. Hopefully pictures are included in that backup… is that correct?

I would imagine that local backups work the same, Right?

Please answer in full sentences regarding my questions. The tutorial was very confusing as well.

What is ambiguous about “no” ? (And what’s not ambiguous about “backups being backed up” )

Let me try again.

If you have configured uploads to be on S3 then uploads are not included in your backup.

Let’s use the term “pictures” instead of uploads even though it can be other media.
This way we don’t confuse the text content as an upload which I’m uploading to s3.

So the 62 MB, backup files on s3 as pictured and uploaded in this thread does not include pictures ?

So how do I make sure the backups have these?
Do the local backups have the pictures too?

When I configured the s3 for "uploads (of media) " which was ambiguous as well. Nobody could post pictures because they were rejected from s3…

Is there a way to have both local and s3 daily backups?
I could care less if 5 days of pictures were lost, we are mostly a text based group.
But I would care if 5 days of text were lost. Digital Ocean only does 7-day backups if you pay them.
So even though I can backup daily, if the droplet gets hacked or damaged, then we lose those backups… I’m starting to think there is not much added value in s3.

I wish there were simple backups similar to WordPress which lets me backup to my google or dropbox account.

No, that is a bad idea, if you upload a text file as an attachment, it’s an upload as well, it will cause confusion. And text in a post is stored into the database. So I’m sticking to the term uploads.

If your uploads are on S3, they are not included in backups. In that case the backups only contain a copy of the database. It does not matter whether your backups are local or on S3.

If your uploads are not on S3, they are included in the backups. In that case the backups contain a copy of the database, and a copy of the uploads. It does not matter whether your backups are local or on S3.

If you are storing something on S3, be it uploads or backups of the database, they will not get lost if your DO droplet gets hacked or damaged. So I don’t see your point.

Since your posts are about backups and not about file and image uploads, I’m moving these to another topic.

I’d like to automatically move my S3 Backups to Glacier but I’m confused by the steps linked in the first post, which doesn’t explain much, maybe because there’s outdated stuff.

Which options should be checked here?

May I ask again in case someone did these steps and knows about it?

Also, do you know what makes these fluctuations in S3 fees?

image1348×381 7.75 KB

Plus, since the launch of the forum (September 2020), the size of the backups has increased by roughly 15%, but the S3 bills have doubled, from 2.50$ to 5$. Any idea why that much?

That’s why I’d like to use Glacier.

Edit: I’ve followed the steps described here and I’ll see how it goes.

Well, it doesn’t go.

My lifecycle configuration:

image965×996 49.9 KB

My S3 bucket:

image1163×584 67 KB

No backup is on Glacier.

So… Two questions for those who have been able to achieve this automated S3 to Glacier transition:

1. What could be wrong in my configuration?

2. The minimum storage duration charge in Glacier is 90 days. Does that mean that if I do 1 backup a day, I’ll be eventually charged for 90 backups in Glacier each month?
   If this is the case, then this Glacier solution won’t be a good idea, unless I reduce a lot my backups frequency.

where in the vps are the backups stored?

I added this to the OP:

Can we chose the folder of the backups or there is a workaround without coding?

I’m using an storage data from my hosting provider so I can mount and use like local but it not supposed to be saved into default path.

If you want it to be saved in a different place then you’d need to change that in your app.yml

Automatic Backups on Backblaze B2

here’s how i have it set up for a hypothetical site hosted on example.com

1. make an account on backblaze (atm, no need to enter payment for <10GB which is free)
2. create a bucket (backblaze > B2 Cloud Storage)
   • name: $sitename-discourse-$random padded to 30char
     • in this example: example-discourse-g87he56ht8vg
     • discourse needs bucket name to be lowercase letters, numbers, and dashes only
     • i suggest keeping it 30 char or less since that shows up nicely in backblaze’s webui without wrapping
   • private bucket
   • enable encryption (SSE-B2)
   • enable object lock
3. create an application key (backblaze > account > app keys)
   • keyName: example-discourse
   • bucketName (Allow access to Bucket(s)): example-discourse-g87he56ht8vg
   • capabilities: read and write
   • leave namePrefix and validDurationSeconds blank
4. configure discourse B2 settings (discourse > admin > settings)
   • backup_location: s3
   • s3_backup_bucket: example-discourse-g87he56ht8vg
   • s3_endpoint: this is shown on the bucket page – make sure to prepend with https://
   • s3_access_key_id: (from previous step)
   • s3_secret_access_key: (from previous step)
     • backblaze only shows you the key once (at creation)!
   • btw, you can also set these as env vars in your container yml instead. this would let you restore with only that file and nothing else:

env:
  ## Backblaze B2 Backups
  # DISCOURSE_BACKUP_LOCATION: 's3' # uncomment to recover from cli
  DISCOURSE_S3_ENDPOINT: 'https://....backblazeb2.com'
  DISCOURSE_S3_BACKUP_BUCKET: 'example-discourse-g87he56ht8vg'
  DISCOURSE_S3_ACCESS_KEY_ID: '...'
  DISCOURSE_S3_SECRET_ACCESS_KEY: '...'
  # DISCOURSE_DISABLE_EMAILS: 'non-staff' # uncomment to disable email during a test restore
  ## you can restore with no data beyond this container yml.
  ## uncomment DISCOURSE_BACKUP_LOCATION above, build container (./launcher rebuild ...),
  ## and then run this inside container (it will restore from B2 bucket):
  ##   discourse enable_restore
  ##   discourse restore <example-com-...tar.gz> # choose restore filename by browsing B2 webui
  ## remember to disable restore afterwards

5. configure backup retention
   • discourse:
     • backup_frequency: 1 (daily backups in this example, but you could do weekly)
     • maximum_backups: disregard this setting – let backblaze handle it
     • s3_disable_cleanup: true (Prevent removal of old backups from S3 when there are more backups than the maximum allowed)
   • backblaze (go to your bucket’s settings):
     • Object Lock (Default Retention Policy): 7 days
     • Lifecycle Settings (custom):
       • fileNamePrefix: default/example-com (optional)
       • daysFromUploadingToHiding: 8 days
         • this should be object lock + 1
       • daysFromHidingToDeleting: 1 day

to summarize retention in this example:

• discourse creates backups every 1 day
• each backup file is immutable for 7 days after upload to B2 (object lock). this protects you against accidents, ransomware, etc.
• 8 days after upload, the object lock on the backup expires. since it’s mutable again, a lifecycle rule can hide the backup file
• the next part of the lifecycle rule deletes any file 1 day after it’s hidden

so you get daily backups. retention time is one week during which backups can’t be deleted no matter what. then backups are deleted 2 days later. so really a backup lives for 9 days or so.

hope that helps someone

on second thought, maybe it’s better to let discourse handle retention (maximum_backups). that way, your backups won’t automatically start expiring if discourse is down. you wouldn’t want a clock ticking on them while trying to recover. if you went that way, you could set maximum_backups=8 and s3_disable_cleanup=false in this example and not use a lifecycle policy in B2. you would still use the object lock policy (7 days), though.

edit: actually, i think you do still need a B2 lifecycle policy because i think files only get ‘hidden’ and not deleted when an S2 client deletes them. i’m using the “Keep only the last version of the file” policy, which is equivalent to daysFromHidingToDeleting=1, daysFromUploadingToHiding=null.

i guess think it over and decide which approach is right for you.

btw, i realize there’s some back in forth in this post. i think it’s informative as-is, but if someone wants, i could make another slightly simpler post with my actual recommendations.

If you put those in environment variables as described in Configure an S3 compatible object storage provider for uploads then you can restore your site to a new server from the command line with only your yml file.

The rest seems like a good plan.

discourse restore <backup.tar.gz>

this will look in your bucket if you have the env vars set? pretty cool if so.

and in that case, you could probably also set them manually with export in bash in the unlikely event that you have to recover. that is, if you don’t want to keep secrets in your container yml for some reason.

Just for confirmation, once I have moved to S3 backups and tested that they works, can I safely delete the content of that folder to reclaim used space?

Yes. If backups are on S3 and you no longer want the backups on the local drive, you can delete them.

Can I only enable S3 for backups but not files or images upload?

I had the exact same question, and could not find an explicit answer here or in Configure an S3 compatible object storage provider for uploads.

But, I’ve set up Scaleway S3 Object Storage for backups only with these environment variables, and it seems to work. I just made a manual backup to the bucket this way using the Admin dashboard.

# Leaving the below parameters commented out 
# because they configure S3 for uploads/CDN distribution, which we don't want.
# DISCOURSE_USE_S3: true
# DISCOURSE_S3_CDN_URL:
# DISCOURSE_S3_BUCKET: 
DISCOURSE_S3_REGION: nl-ams
DISCOURSE_S3_ENDPOINT: https://s3.nl-ams.scw.cloud
DISCOURSE_S3_ACCESS_KEY_ID: <redacted>
DISCOURSE_S3_SECRET_ACCESS_KEY: <redacted>
DISCOURSE_S3_BACKUP_BUCKET: <redacted>
DISCOURSE_BACKUP_LOCATION: s3

Static assets are still served from my VPS, as expected & desired, and I’m not seeing any errors. I’m not sure if the scheduled backups will work, but if the manual backup worked, they should too.

I’ll add a note to the wiki above that these parameters can be left out if you only want to use S3 for backups, and not for file or image uploads.
Update: it’s not a wiki.
Update 2: it was made a wiki at my request, thanks!

If I have 2 redundant web containers, how can I ensure only 1 runs a backup?

you don’t need to do anything - Discourse handles this automatically - sidekiq and redis will only run the job once, and sidekiq will only grab one of the web containers, but i don’t think you can be sure which one it is on any given backup if you are using local backups.

thus, you should use an external S3 bucket if you aren’t already, otherwise your backup files will likely be randomly stored across the local disks of both containers. using S3 bucket means your backup will always land in the same place.