There's a few new computers in my circle of family - my parents just bought one and my wife's laptop and phone - and I need a better solution for backing up everything.
My previous solution CFTBackup was no longer working for windows computers as they were reinstalled or replaced. And I never got the whole part of the offline sync working; plus after using a simple rsync for almost 4 years, I've determined that the 150Kbit/s limit was more than adequate to back things up in a timely fasion and a lot less hastle.
So it's time for a new solution. I recently bought two new HGST Deskstar NAS 4TB hard drives as space was starting to get full on my 1.5TB (at my house) and 1TB (my parents house) drive's.
For refresher the hardware the drive is going into:
- Intel BOXD510MO - An Intel Atom based board; populated with 2GB RAM
- Mini ITX Case
Now I needed to find a better solution to sync the two backup systems and backup the files off the desired systems.
Requirements
- Files are stored on the backup system in the same hierachy as on the target systems. I don't want to be tied to a particular software platform. This means no binary blobs of random names.
- Client should exist for Windows and Linux (at a minimum) and preferably Android
- Automatic startup on windows and linux
- Ease of use
- I shouldn't need to have a copy of everything everwhere. E.g. I don't need my parents data on my laptop.
- The solution should support some form of versioning so that I can restore an older version of a file (up to some point in time)
- Some level of access control; not necesarily by username/password
- Ability to have multiple "servers" with copies of the data. I should be able to configure it so that files are only sync'd to one location if I desire.
- "Headless" mode as a full feature. Not some hacky solution
- Ability to handle 350,000 files; approx 1TB right now.
- Free
- Preferably open source
Things it doesn't need:
- Enterprise level number of users
- Git integration
- Cloud based backup
Other things that would be nice
- Ability to share a file publicly
- Ability to have a append only backup
- E.g. I pull pictures from my camera onto my laptop, once they are backed up and I'm done with them, I want to delete them from my laptop, but have them still be on the server.
The Products
I've basically found three categories of products so far:
- Cloud Backup Solutions
- Cloud Storage Solutions - (think DropBox)
- P2P Sync Solution
Cloud Backup Solutions
The only product in this category that I found with a linux client was CrashPlan.
Crashplan
Pros:
- Easy setup
- Has Windows, Linux & Android Clients
- Easy setup, ability to share space with not just yourself, but friends
- Cloud backup option
- Not tested, but you can have multiple backup locations
- Has versioning, but limited to Week old, 90 day old, 1 year old, and 2 years old
Cons:
- Requires java
- The headless mode isn't supported, but is possible because it takes advantage of how the management is client/server
- All backups are in a proprietary binary format
- Not open source
- Not real time backup (only backs up on interval)
Cloud Storage Solutions
I found a few in this category, along with some that are no longer maintained. One of the requirements for this category was that I need to be able to run my own server.
It turns out I didn't look at any of these solutions for the backup portion for the following reasons:
- Lack of multi-server support (with distint storage, not just load balancing with shared storage)
- Data is stored in binary "proprietary format"
- Poor performance reviews
P2P Sync Solution
This solution is closest to what I was using with my rsync setup. By definition it copies files in the same directory tree as the source.
Both of these programs are very similar in that all they do is attempt to keep two folder in sync.
- Both use a cryptographic key to identify peers
- Both are a pure sync and do not have the ability to share a file publicly
- Both can add a device to an android phone using a QR code
- Both have upload and download limits
BitTorrent Sync
Pros:
- Windows, Linux and Android Clients
- Can place placeholder files in a directory, showing that they exist, but not downloaded
- Can work through NAT firewalls as it uses a public tracker
Cons:
- Not open source
- No versioning
- When trying it out I somehow managed to create two identities and didn't know how to delete one on my android device.
- I think identities see everything shared from every computer, but you can share between identies.
- Shows up using CPU even though no files are changing. (1-5% on windows)
Syncthing
Pros:
- Open Source
- Can control which shares are shared with what computers
- Can set a folder as a "Folder Master" which means it's essentially read-only to other computers - changes on other computers are not synced to this computer
- Supports two types of file versioning
- Simple versions, with a max count
- Staggered versions up to a set time. A version up to every 30 seconds the first hour, Every hour up to 30 days, and then a week after that - Similar to a Grandfather-father-son type backup. Can keep versions forever
- Has ignore patters to ignore syncing of certain files/directories - e.g. Can ignore .git directories!
- Maxes out my upload if I let it.
Cons:
- Requires opening firewall (UPnP or by hand) for communication - not an issue as I have a VPN configured between the backup systems
- Windows automatic startup requires a separate program
- Any configuration change requires a restart of the services, which breaks all connections
- Android app is VERY alpha.. (not even beta)
- Folders are identified by name only. So if computer one shares a folder named "Photos" and shared with Computer 2, it will be treated the same as a folder named "Photos" shared by Computer 3
Meh's (for me.. might be Cons for a lot of people):
- No concept of users - every computer/installation is unique.
- If a folder is shared with one computer, that computer can share it with others.
Decision
I think I'm going to try Syncthing for starters. I will probably continue to use rsync for my Pictures backup (and probably most of the the things from my laptop). Unrelated, I plan on using Gitolite for keeping a copy of my code and using it's mirror feature to back up my repositories.
If I find I absolutely need file sharing, I've played around with Pydio a little bit and it looks like you can have a standard filesystem be the backing (with a normal tree layout) and configure it to just do sharing. Though that seems like overkill for something so simple that I want to do.