Building a backup script in bash

Table of Contents

Introduction

After going through the basics of bash I started work on a bash script that would be used for backing up MySQL databases on a web server that I currently operate to run a few websites.

The benefit of working on a project for a specific purpose is that it allowed me to learn more about bash scripting because when I wanted to perform some action I didn’t know how to do, I had to research and figure out how to do it.

New things learned in Bash

This section covers new things that I learned while building the backup script as before starting this script I only covered the basics of bash scripting. When I started work on this script I had to learn how to solve the problems that I encountered due to my lack of experience with Bash combined with not remembering any of the Bash scripts which I had written years ago.

Firstly open up the terminal, change to a suitable directory to store the script file, and then use the touch command to create the bash script file.

Creating a function

As a lot of code was being written, it made sense to optimize the code by moving parts into functions to make the code easier to manage.

The below example shows a basic hello world example of a function that outputs the text “Hello World!”.

#!/bin/bash

test_function() {
  echo "Hello World!"
}

test_function

Create a new file such as test.sh and set the execute permission to run the file.

sudo chmod u+x test.sh

Now run the script to see the result.

./test.sh

Passing a parameter to a function

I needed a way to pass data to the function, it’s possible to reference arguments while inside of a function as shown in the below example.

#!/bin/bash

test_function() {
  PARAM="${!1}"
  echo $PARAM
}

TEXT="This is text passed into the function"
test_function TEXT

Run the script file.

./test.sh

You will see the following output.

This is text passed into the function

Reading environment variables

I needed a way to pass environment variables to the script.

Start by creating a new file named .env and adding the following key-value pair to it.

TEST=123

Update the test.sh file to the following code.

#!/bin/bash

source .env

echo $TEST

Run the script to get the output of the variable.

123

Getting a formatted date

The date command can be used to return the date, however, this shows the date in full with time, timezone, etc.

If you want to get the date only use the following.

#!/bin/bash

echo $(date '+%Y-%m-%d')

The current date will be returned, the date below is at the time of running this.

2024-07-08

Adding to an array

Originally I needed a way to generate an array of file paths however I changed the code and didn’t need the use of the array.

However an example will be included as work was originally done to research this.

#!/bin/bash

ARRAY=()
ARRAY+=('apples')
ARRAY+=('pears')
ARRAY+=('oranges')
ARRAY+=('bananas')

echo ${ARRAY[@]}

This is the output after running the above script.

apples pears oranges bananas

Building the script

The backup script is made up of multiple files which will be gone through in each section representing the file.

.env file

The first file is the .env file that's used to store the environment variables to be used. The reason for having a separate file for key/value pairs is so it is easier to change the values and to make sure that the code doesn’t get updated accidentally causing an error to occur. 

DB_LIST="movies songs"
DB_CONFIG_PATH="/home/mesh/config.cnf"

BACKUP_OUTPUT_PATH="/home/mesh/dumps"

S3_BUCKET_URL=s3://devnudge

These variables will be required to run the script,

  • DB_LIST - A list of databases to back up.
  • DB_CONFIG_PATH - The file path to the config.cnf file used for the sqldump command.
  • BACKUP_OUTPUT_PATH - The file directory to output the generated backup files.
  • S3_BUCKET_URL - The Amazon AWS S3 bucket URL to upload files to.

common.sh file

The next file is used to include the .env file to load the values from that file and also has common functions that would be used in more than one bash script.

The check_var function will be used to check that the required environment variables are set, if any of them aren’t set then the script will stop running.

#!/bin/bash

source .env

check_var() {
  if [[ -z "${!1}" ]]
  then
    echo "${1} is not set"
    exit
  fi
}

backup.sh file

This file is the main script file that will be run to back up the databases.

Create the backup.sh file and add the following code below.

This code will include the common.sh file and use the check_var function.

#!/bin/bash

source common.sh

check_var "DB_LIST"
check_var "DB_CONFIG_PATH"
check_var "BACKUP_OUTPUT_PATH"
check_var "S3_BUCKET_URL"

Create a function named generate_db_backup that will do the following.

  1. Take an argument representing the database name and set it to the DATABASE variable.
  2. Use that name, the backup output path, and date variables to create a file path.
  3. Call the mysqldump command which uses the DB_CONFIG_PATH variable to get the user authentication details, the DATABASE and DATE variables.
  4. After the sqldump command generates the db data file created at the file path set in the DB_FILE_PATH variable then call the s3cmd command to upload that file to Amazon AWS S3 using the S3_BUCKET_URL representing the bucket name.

The DATE variable will be set in the next code snippet, the rest of the variables come from the .env file.

generate_db_backup() {
  DATABASE="${!1}"
  echo "Backing up $DATABASE database"

  DB_FILE_PATH="$BACKUP_OUTPUT_PATH/$DATE-$DATABASE.sql"

  mysqldump --defaults-file=$DB_CONFIG_PATH $DATABASE > $DB_FILE_PATH
  s3cmd put $DB_FILE_PATH $S3_BUCKET_URL
}

Create another function named backup_databases that will loop through the DB_LIST containing the names of the databases and pass each database name set to the DATABASE variable into the generate_db_backup function.

The reason the DATE variable is set up here is because it should only be created once rather than inside of the loop being created multiple times.

backup_databases() {
  DATE=$(date '+%Y-%m-%d')

  for DATABASE in $DB_LIST
  do 
    generate_db_backup DATABASE
  done
}

The final code snippet will be used to call the backup_databases function that’s defined and output the text “Done!” after the whole backup process has been completed.

backup_databases

echo "Done!"

Setup script on a web server

The next step is to get this script running on a web server.

I decided the best thing to do is to create a new git repository for all my bash scripts, this makes it easier to add to the server and update with new scripts.

Copy example files

Copy the .env.example file to .env and then fill in the values.

cp .env.example .env

Copy the config.cnf.example file to config.cnf and then fill in the values.

cp config.cnf.example config.cnf

Setup mysqldump

Ubuntu server should already come with the mysqldump command, if it's not available then use the following command to install MySQL client which will include mysqldump.

sudo apt install -y mysql-client

Setup s3cmd

After logging into the server the s3cmd needed to be installed and setup.

Run the following to install, as the server is ubuntu s3cmd is available in the APT package manager and can be installed easily. 

sudo apt install -y s3cmd

Retrieve the Amazon AWS secret and access keys so that s3cmd can connect to your Amazon AWS account when uploading files.

Now run s3cmd with the configure flag to add the server details.

s3cmd --configure

Run the script

Go to the scripts directory and use the following to run the backup script.

./backup.sh

Wait for the script to finish running and then check the S3 dashboard to see if the files were uploaded to it.

Setup cronjob

Now that the script has been tested and is working the final step is to set up a cronjob so the databases get backed up automatically.

I recommend using setting up a cronjob that will run at a specific interval.

Use the following to enter the crontab. 

crontab -e

Add the following line and save the file.

The below line will run the backup,sh file every Saturday at midnight, you can adjust the values at the beginning to change how often the script gets called.

0 0 * * 6 bash /srv/scripts/backup.sh

Conclusion

Once you reach this part you would have created a bash script to create a backup of your databases for your server or local machine.

This project has provided an opportunity to develop skills and experience with bash scripting, so after this project, you should be more confident in creating your bash script.

A new script that you could try and create that’s related to this is to create a script that removes old backup files for example backups that are older than one month. This is because the backup script in this guide will keep creating new backup files and eventually will build up a big list of backups.

Hope this guide has been useful to you regarding bash scripting.