Nightly volume backup script for dns-323 soho NAS (rsync+cron)
Published January 6th, 2009 in dns-323, hacks, unix
Hey kids, so remember my original post on dns-323 hacking? Well I’m back at it. In Part 1 I decided not to use RAID 1 mirroring for a couple of reasons (see section 2 — RAID or no RAID?). Instead, what I’m doing for redundancy is selectively backing up a set of “important” data from the first NAS disk to the second. Note: This selective approach leaves me some free space on the second volume for other things, like Time Machine on my laptop.
Basically, it goes something like this: “back up all my mp3s and digital pictures from HD 1 to HD 2 every night. On the first run, copy everything. On subsequent runs, copy only what’s changed”.
Nifty? Yes. Easy? Yes. Quick? Very! (Thanks to the simple elegance of UNIX rsync and cron). Hit the jump for the tutorial and script.
Step 1: Get the script and set up the backup directory
Here’s the script. Stick it in/mnt/HD_a2/ffp/var/scripts/, using vi or ftp or however you like. Don’t forget to make it executable by chmod +x rbackup.sh after you put it in that directory.
#!/bin/sh
###########################################################################
# rbackup.sh
#
# script to rsync a set of backup directories from HD 1 to HD 2.
# Important: Add the directories you wish to backup to the SOURCEDIRS
# variable, space-delimited.
# Also, don’t forget to cron this script!
#
# (loosely based on http://www.scrounge.org/linux/rsync.html)
# w 6/1/09 horto
###########################################################################
# paths and such
RSYNC=/ffp/bin/rsync
SOURCEDIRS=”/mnt/HD_a2/ffp \
/mnt/HD_a2/media/mp3 \
/mnt/HD_a2/media/pictures \
”
DESTINATION=/mnt/HD_b2/backups/HD_a2# logfile. keep an eye on the space of it over time.
LOG=/mnt/HD_a2/logs/rbackup.log# excludes file - Contains wildcard patterns of files to exclude.
# i.e., *~, *.bak, etc. One “pattern” per line.
# You must create this file.
EXCLUDES=”"# rsync options, for testing
# -v Verbose -vv More verbose. -vvv Even more verbose.
# -a Archive. Mainly propogate file permissions, ownership, timestamp, etc.
# -n Don’t do any copying, but display what rsync *would* copy. For testing.
# NOTE: never use -z option, don’t want to compress locally#OPTS=”-van”
OPTS=”-va”if [ ! -d $DESTINATION ] || [ ! -w $DESTINATION ]; then
echo “ERROR: Destination $DESTINATION does not exist or is not writable.” >> $LOG
fifor SOURCE in $SOURCEDIRS; do
if [ ! -d $SOURCE ]; then
echo “ERROR: Directory $SOURCE does not exist” >> $LOG
else
echo “Starting rsync of source directory $SOURCE …” >> $LOG
$RSYNC $OPTS $SOURCE $DESTINATION >> $LOG
fi
done
Next, create a backup directory; This is the directory where you want the files to be backed up to. In this case I’m using a subdirectory of /backups on Volume 2 (disk #2) of the NAS: /mnt/HD_b2/backups/HD_a2. You can use whatever you want, but the directory you choose needs to match the DESTINATION variable in the script above.
<ssh into the 323>
cd /mnt/HD_b2
mkdir backups
cd backups
mkdir HD_a2
Also, you might want to make sure that /mnt/HD_a2/logs exists since that’s where this script will write logs to.
Step 2: Choose the data you want to back up.
As you might remember from my first article, the data I consider important to me is my mp3 library and digital pictures. I also want to back up the contents of my /ffp directory just in case I need to recover my ffp setup and custom scripts, should anything happen to them.
All you need to do is pick which directories you want to include in your backup, by editing the SOURCEDIRS variable at the top of the script. Specify the full path to each directory you want to include in the backup.
Keep in mind that rsync will recursively pick up subdirectories. So for me, specifying /mnt/HD_a2/media/mp3 will grab any and all subdirectories below it.
Step 3: Cron it to run nightly
Remember the editcron.sh script we used in Part 1 to make our cron updates stick when rebooted? Just edit that script (it’s in /mnt/HD_a2/ffp/start/editcron.sh) to include a line for the backup script. I set mine to run nightly at 1am:
/bin/echo "0 1 * * * /ffp/var/scripts/rbackup.sh >> /mnt/HD_a2/logs/rbackup.log 2>&1" >> $CRONTXT
For my devouted DNS-323 hacker fans, FYI… I run the backup script at 1am, followed by the unpacker.sh script at 2am, followed by torrentwatchdog at 3am, all daily. I’ve found this is the best combination for my needs and allowing the drives to be mostly spun down during the day when I’m not around. And you know, saving electricity during peak hours 
Step 4 (Optional): Run it by hand the first time
Open two terminal (or putty) windows; log in as root in both of them.
In the first terminal window:
cd /mnt/HD_a2/ffp/var/scripts/
./rbackup.sh
The lights should start blinking wildly as it begins to rsync everything for the first time.
You can then view the output of the log in the second terminal window:
tail -f /mnt/HD_a2/logs/rbackup.log
You should see something like…
Starting rsync of source directory /mnt/HD_a2/ffp …
sending incremental file list
Wait for it to finish (10gb took me about 10 minutes), and you’re all set! Just reboot your 323 to make sure that the cron changes kick in, and it will automatically incrementally back up any changes to the source directories!
14 Comments to “Nightly volume backup script for dns-323 soho NAS (rsync+cron)”
- 1 Pingback on Jul 20th, 2009 at 5:25 pm
First!
… I always wanted to do that. Now that it’s out of my system, I can never do it again.
BTW, nice job on this script. I’m going to try it out this week. Have you upgraded the firmware to 1.6 yet? Any issues?
Thanks for the great tutorial. I appreciate you taking the time to type it up.
ditto on appreciating the writeup - I will give it a go this evening. Thanks again.
Hello!
Thanks for the cool Guide!
I have to sync my data not to an second internal drive but to an another older NAS Drive over network. How can i do this? What parameters and pathes are written in the rsync script? The second older NAS is an Longshine 8311 without SSH or any other hacks. I can log in with an admin and password.
Thanks for help!
Thanks for the tuto. I really appreciate it. The only problem I have is that for an unknown reason, my script work well when I run it manually (./editcron.sh) but it doesn’t run at startup. Do I need to do something else than put it in /mnt/HD_a2/ffp/start? Thank you again!
Excellent how-to.
I modified the script slightly. I wanted the date shown before the backup so I added the following before the ‘for’ loop
TODAY=`date`
echo $TODAY >> $LOG
Also I noticed that you check if the destination exists but the script, as posted, doesn’t abort if the directory isn’t there. I added an “exit 1″ after error message is echoed.
And finally you have a variable for an exclude file but no option in the rsync to use it. I changed the opts line to:
OPTS=”-va –exclude-from=$EXCLUDES”
Thanks!!!
The manual rsync worked like a charm but I think I’m running into the same problem as Don on getting the editcron.sh to run. I can’t get it to set up the cron job.
I verified the editcron.sh file is there with the right ownership (root:root) and has the execute permissions. However when I look at the ffp.log it’s giving the error:
“/ffp/etc/rc: line 45: /ffp/start/editcron.sh: not found”
The file is there but I must have some simple permissions problem in the setup. I can’t even seem to get the editcron.sh to run for the test. Suggestions?
Thanks.
Thanks a lot for your help! Your instructions actually helps us! Excellent work!!! 8)))
Thanks for the great script. The only issue I am having is adding SOURCEDIRS where I have spaces in the directory names (E.g. iTunes Music). I have tried using additional quotation marks (”) and also using backslashes before each space (\) where there is a space, but the script does not seem to recognize these. I have added below everything that I have tried for one of the Source directories I want to back up.
Attempt #1: SOURCEDIRS= “/mnt/HD_a2/iTunes Music \”
Attempt #2: SOURCEDIRS= “/mnt/HD_a2/”iTunes Music” \”
Attempt #3: SOURCEDIRS= “/mnt/HD_a2/iTunes\ Music \”
Attempt #4: SOURCEDIRS= “/mnt/HD_a2/iTunes\\\ Music \”
Attempt #5: SOURCEDIRS= “/mnt/HD_a2/’iTunes Music’ \”
Attempt #6: SOURCEDIRS= “/mnt/HD_a2/’iTunes\ Music’ \”
After running ./rbackup.sh each of the above attempts, I get:
: No such file or directory
Any help would be greatly appreciated.
Thanks
GT
gt, maybe you can try:
SOURCEDIRS=”/mnt/HD_a2/iTunes?Music”
if that’s the only directory you’re backing up.
where i have:
SOURCEROOT=”/mnt/HD_a2″
SOURCEDIRS=”\
$SOURCEROOT/ffp \
$SOURCEROOT/Files \
$SOURCEROOT/Pictures \
$SOURCEROOT/My?Music \
$SOURCEROOT/Videos \
”
and this works for me.
- gho
Hey there, I used your script and love all the other info on your page. I used it pretty much exclusively while setting up my DNS-323.
I started this with very limited unix/linux knowledge, but slowly got up to speed and started to edit some of the scripts.
The question I have is in regards to the rbackup script. I have Volume 1 backing up new data to Volume 2. I often delete movies I’ve watched, or find a file that’s corrupt and delete it from Volume 1, but it still remains on Volume 2. Outside of deleting everything on Volume 2 and copying everything on Volume 1, do you have any recommendations for removing files on Volume 2 that don’t appear on Volume 1? I figured there’s a way to check if files on Volume 2 exist on Volume 1, and if they don’t, then delete them.
Any recommendations?
Thanks,
Rob
hi. i am trying to implement the recommendations on this thread but am totally new to UNIX/LINUX.
got as far as setting ssh to the DNS-323 but have some basic questions like:
1) why are there 4 drives in the mnt directory as the DNS-323 only has two? they are HD_a2, HD_a4, HD_b2 and HD_b4.
2) I got stuck trying to copy the rsynch script to the correct folder. no clue how to copy files from my “mac os” over to UNIX in the DNS-323.
3) why can’t a cd to any of the folders on the drives in UNIX?
4) what does the .localized suffix mean?
sorry for what must seem like silly questions but i am determined to set up cron/rsynch on this thing.
ok i made some progess on my own. #2 and #3 resolved. ran the script. it worked. however when i delete a file from a source directory, the rsynch does NOT delete it from the backup directors. this is a problem since as files are added and deleted the backup will be a cummulative backup of every file on the machine and be much bigger then the source. hardly a “synch” of the files. am i doing something wrong?