This is a text-only version of the following page on https://raymii.org:
---
Title : Set up your own truly secure, encrypted and shared file synchronization, aka Dropbox clone
Author : Remy van Elst
Date : 15-10-2013
URL : https://raymii.org/s/articles/Set_up_your_own_truly_secure_encrypted_shared_storage_aka_Dropbox_clone.html
Format : Markdown/HTML
---
### TL;DR
This article describes my truly secure, encrypted file synchronization service.
It used EncFS and dvcs-autosync which lets me share only the encrypted data and
mount that locally to get the plaintext. It works on OS X, Linux and ARM linux.
This article has setup instructions for all those platforms.
<p class="ad"> <b>Recently I removed all Google Ads from this site due to their invasive tracking, as well as Google Analytics. Please, if you found this content useful, consider a small donation using any of the options below:</b><br><br> <a href="https://leafnode.nl">I'm developing an open source monitoring app called Leaf Node Monitoring, for windows, linux & android. Go check it out!</a><br><br> <a href="https://github.com/sponsors/RaymiiOrg/">Consider sponsoring me on Github. It means the world to me if you show your appreciation and you'll help pay the server costs.</a><br><br> <a href="https://www.digitalocean.com/?refcode=7435ae6b8212">You can also sponsor me by getting a Digital Ocean VPS. With this referral link you'll get $100 credit for 60 days. </a><br><br> </p>
### Diagram
![Diagram][2] **Overview of the solution we are building.**
My data is in an EncFS encrypted folder. The unencrypted contents are available
after unlocking the folder. The encrypted files are synced to an ssh server an
to a few other machines and devices using dvcs-autosync. The encryption happens
on my machines before the data leaves the to internet.
### Preface
Recently I've had to stop using SpiderOak for my file backup and synchronization
across machines. The main reason being that there is no ARM version of SpiderOak
and the RAM usage was getting out of hand for me. And there still is no open
source client, sadly. However, my time with SpiderOak was good, I've paid for it
and most of the time it just works fine.
But since I recently bought an ARM Laptop on which I also need my files, it
became time to switch to another secure shared file storage. I have a few
demands for such a service:
* It should support synchronization to multiple (more than 2) devices.
* It has to run on both OS X and any reasonable recent Linux version.
* It should encrypt files on my machine(s) before going to the internet.
* It has to be easy to add or remove storage nodes (like vps servers).
* It has to be open source.
* It should run on both x86 and ARM (debian armhf) (Chromebook ARM, Raspberry Pi).
Then all current commercial services drop off, including SpiderOak, Bittorrent
Sync and git-annex. This resulted in a clever combination of [EncFS][3] and
[dvcs-autosync][4]. Because, in this day and age, you cannot trust any "cloud"
provider with your unencrypted data. (And you can only trust those who say they
do it securely when they release there source code, wink wink Wuala/Spideroak).
### Overview
I'll describe the steps and requirements needed to set this up first. Then we
get started with the setup. First we'll set up the server. Then the first Linux
client. If needed, steps are provided for adding other Linux clients. Then
instructions for OS X are provided. It is a little long, but if you want privacy
and security a one time investment is required.
#### Requirements
* Linux machine (With python 2.6+, tested on Debian 7 armhf and Ubuntu 12.04)
* [VPS/External ssh server][5]
Not mandatory:
* OS X machine (iMac, Macbook) with python 2.6+ (Included in Lion and above), git, xcode, command line tools for xcode and homebrew.
#### Steps
* Prepare the SSH/git server
* Prepare the Linux client
* Install EncFS
* Creating the secure EncFS folder
* Install dvcs-autosync
* Create an XMPP account
* Set up dvcs-autosync
* Special steps for an ARM Chromebook
* Set up another Linux client
* Prepare the OS X client
* Install MacFUSE
* Install EncFS
* Get the secure folder
* Install dvcs-autosync
* Set up dvcs-autosync
So, lets get started. In about half an hour you have your own secure encrypted
file synchronization service.
### Set up the SSH server
As said, you'll need an SSH server which will act as your central data
repository. Here your encrypted data will reside, and clients push and pull
changes to and from here. If you have a few laptops which are not on all the
time, this server makes sure all the clients have the most recent data.
If you don't have a VPS, [InceptionHosting has good VPS servers for a nice
price. (Affiliation link)][6]. [Digital Ocean][1] also does a good job. (Also
affiliation link).
I won't cover the installation and setup of the server. SSH, a user account and
a passwordless SSH key is all you need. Google can help you with the setup of
that.
First install git:
apt-get install git
Now, go to your home folder and create the "repository":
cd ~
git init --bare autosync.git
That's it. Now we are going to set up the clients.
### rsync.net
rsync.net is a service which provides online storage available via rsync. You
can use them instead of your own VPS server, I'll show you how in a moment. Why
would you store your data there, and why do I recommend them when this is an
article about a truly private system? Because they use a standard open protocol,
support git and don't require a local client to be installed. Dropbox for
example does. Any service which provides git/ssh access would be fine because
you are just dumping encrypted data there and rsync.net is one of the better
ones to do that.
Add you ssh key to rsync.net:
scp ~/.ssh/backup_nopasswd.pub <user-id>@<server>.rsync.net:.ssh/authorized_keys
To add more than one key, make sure the first key is added like above, for each
subsequent key use the following method:
cat ~/.ssh/other_key.pub | ssh <user-id>@<server>.rsync.net 'dd of=.ssh/authorized_keys oflag=append conv=notrunc'
[See also their manual on ssh keys][7]
Create a git repo on rsync.net:
ssh <user>@<server>.rsync.net "git init --bare autosync.git"
Initialized empty Git repository in /data5/home/<user>/autosync.git/
Remember/copy that path to somewhere, we'll need it later on to add the git
remote.
That's it for now. Continue with the tutorial, when we get to the other git
parts where applicable there will be a part for rsync.net.
[Read more on rsync.net git/svn support][8]
### Set up the Linux machine(s)
#### Install EncFS
This one is easy:
apt-get install encfs
This will automatically install and set up both FUSE and EncFS. It'll also put
you in the right user groups (FUSE).
#### Creating the secure EncFS folder
_(Note that this is required only once on the first machine, not on all the
others added later on)._
Now in your home folder, execute the following commands to create the Encfs
folders and set them up:
cd ~
mkdir share
mkdir secure
encfs ~/secure ~/share
The first option of the encfs command specifies the folder where all the
encrypted data will be. The second options specifies the mount point where the
unencrypted data will be. It will ask you a few questions. The first can be
answered with **p**. The second question is for the folder password. Make sure
this is a **long and strong** password.
Output:
Creating new encrypted volume.
Please choose from one of the following options:
enter "x" for expert configuration mode,
enter "p" for pre-configured paranoia mode,
anything else, or an empty line will select standard mode.
?> p
Paranoia configuration selected.
Configuration finished. The filesystem to be created has
the following properties:
Filesystem cipher: "ssl/aes", version 3:0:2
Filename encoding: "nameio/block", version 3:0:1
Key Size: 256 bits
Block Size: 1024 bytes, including 8 byte MAC header
Each file contains 8 byte header with unique IV data.
Filenames encoded using IV chaining mode.
File data IV is chained to filename IV.
File holes passed through to ciphertext.
-------------------------- WARNING --------------------------
The external initialization-vector chaining option has been
enabled. This option disables the use of hard links on the
filesystem. Without hard links, some programs may not work.
The programs 'mutt' and 'procmail' are known to fail. For
more information, please see the encfs mailing list.
If you would like to choose another configuration setting,
please press CTRL-C now to abort and start over.
Now you will need to enter a password for your filesystem.
You will need to remember this password, as there is absolutely
no recovery mechanism. However, the password can be changed
later using encfsctl.
New Encfs Password:
Verify Encfs Password:
That's it. Try to create a file in the `~/share` folder and you'll see the
encrypted file show up in the `~/secure` folder.
_To access the folder any time later, use the above EncFS command to mount it
again._
We need to prepare the secure folder for usage with dvcs-autosync.
Make sure the folder is mounted:
encfs ~/secure ~/share
EncFS Password:
_IMPORTANT! Make sure to remove the .encfs_ files from the secure folder before
syncing. IF THESE FILES ARE IN THE SYNCED FOLDER, YOUR FILES ARE MUCH MORE
EASIER TO BE CRACKED.*
_You can also add them to the`.gitignore file`:_
-rw-rw-r-- 1 remy remy 1.1K Aug 29 16:09 .encfs6.xml
echo .encfs6.xml >> ~/secure/.gitignore
Now continue.
Then:
cd ~/secure
git init
Initialized empty Git repository in /home/remy/secure/.git/
date > ./date
git add date
git commit -m "Initial Commit"
[master (root-commit) 3cc0fba] Initial Commit
1 file changed, 1 insertion(+)
create mode 100644 date
Now make sure you have your SSH server domain name ready. For me, it is
`sync.raymii.nl`. Also have the autosync folder ready. For me this is
`/home/remy/autosync.git`. We need this in the following command:
git remote add origin ssh://sync.raymii.nl/home/remy/autosync.git
If you are using rsync.net, you have to add an upstream like this, with the path
you remembered from above:
git remote add origin ssh://<server>.rsync.net/data5/home/<user>/autosync.git/
This adds the SSH server as `origin` in git. Now push the first change:
git push -u origin master
The `-u` option also sets up the master branch as default and starts tracking in
from origin.
#### Install dvcs-autosync
On debian/Ubuntu this part is easy because you can install a package. On other
distro's, follow the instructions [here on the official repo][9].
So, to install the package:
apt-get install dvcs-autosync
#### Create an XMPP account
dvcs-autosync uses XMPP as a way to send changes to other online nodes. So, you
need a XMPP account. You can use your Google (talk) account for this, but you
can also create an account via Pidgin at services like
<http://www.swissjabber.ch/> or <http://xabber.de>. I have my own XMPP server
network, so for me that is solved. Make sure you have the username
(you@xabber.de) and the password ready.
#### Set up dvcs-autosync
This is also quite simple. Copy the example file from [here][10] to
`~/.autosync` and edit at least the following sections:
[autosync]
path = ~/secure
Change the path to your freshly created secure folder (`~/secure`).
[xmpp]
username = you@yourXMPPserver.tld
password = Your_Passw0rd
alsonotify = you@yourXMPPserver.tld
And change the XMPP account data. That's it. You can change more things, but
that is all explained in the config file.
Now, with all the above set up, start `dvcs-autosync` from the command line:
dvcs-autosync
You'll get a lot of output, which can be safely ignored when you experience no
errors:
DEBUG:jabberbot:Got presence: you@yourXMPPserver.tld/AutosyncJabberBot on MyHostName.tld (type: None, show: None, status: None, subscription: None)
Could not load one of the supported DNS libraries (dnspython or pydns). SRV records will not be queried and you may need to set custom hostname/port for some servers to be accessible.
INFO:root:pynotify initialized successfully, will use desktop notifications
INFO:root:Growl does not seem to be installed
INFO:root:Watching path /home/remy/secure
DEBUG:root:Checking/writing pidfile /home/remy/.autosync.pid
WARNING:root:PID file /home/remy/.autosync.pid already exists, but no process seems to be running, removing file now
INFO:root:Using only XMPP notification
INFO:root:Ignoring files matching any of the patterns
INFO:root:Adding list to inotify exclude filter: ['/home/remy/secure/.git', '/home/remy/secure/.svn', '/home/remy/secure/.hg', '/home/remy/secure/src/packages', '/home/remy/secure/src/java/openuat', '/home/remy/secure/src/csharp/sparkleshare', '/home/remy/secure/src/cpp/cross/keepassx', '/home/remy/secure/src/android/ipv6config']
DEBUG:jabberbot:Registered command: help
DEBUG:jabberbot:Registered command: login
DEBUG:jabberbot:Registered command: ping
DEBUG:jabberbot:Registered command: pushed
DEBUG:jabberbot:Registered command: unknown
DEBUG:jabberbot:Registered command: whoami
INFO:jabberbot:*** roster ***
INFO:jabberbot: ddg@gg.im
INFO:jabberbot: you@yourXMPPserver.tld
INFO:jabberbot:*** roster ***
INFO:jabberbot:bot connected. serving forever.
Now try to add some files, you'll see that they are automatically added:
DEBUG:root:Starting coalesce timer with 2 seconds until coalescing events for file /home/remy/secure/,fdgh4878rgHHDBa would occur (if no other changes happen in between)
DEBUG:root:Resetting already active coalesce timer to new timeout of 2 seconds until coalescing events for file /home/remy/secure/,fdgh4878rgHHDBa would occur
INFO:root:Coalesce event triggered for file /home/remy/secure/,fdgh4878rgHHDBa
DEBUG:root:Considering file /home/remy/secure/,fdgh4878rgHHDBa, which has the following events recorded:
DEBUG:root: Event type=IN_CREATE, action=git add %s
DEBUG:root: Event type=IN_CLOSE_WRITE, action=git add %s
INFO:root:Final action for file /home/remy/secure/,fdgh4878rgHHDBa: type=IN_CREATE, action=git add %s
INFO:root:NOTIFICATION: Local change: Committing changes in /home/remy/secure/,fdgh4878rgHHDBa: git add %s
DEBUG:root:Substituting cmd part %s with /home/remy/secure/,fdgh4878rgHHDBa
WARNING:root:NOTIFICATION: Command failed: Command 'git add /home/remy/secure/,fdgh4878rgHHDBa' in '/home/remy/secure' failed. Output:
fatal: pathspec ',fdgh4878rgHHDBa' did not match any files
DEBUG:root:Substituting cmd part %s with Autocommit of file /home/remy/secure/,fdgh4878rgHHDBa changed on host localhost
DEBUG:jabberbot:*** props = [u'jabber:client']
DEBUG:jabberbot:*** jid = you@yourXMPPserver.tld/AutosyncJabberBot on localhost
DEBUG:jabberbot:*** username = raymii
DEBUG:jabberbot:*** type = chat
DEBUG:jabberbot:*** text = [Local change]: Committing changes in /home/remy/secure/,fdgh4878rgHHDBa: git add %s
DEBUG:jabberbot:*** cmd = [local
DEBUG:jabberbot:*** props = [u'jabber:client', u'http://jabber.org/protocol/xhtml-im']
DEBUG:jabberbot:*** jid = you@yourXMPPserver.tld/AutosyncJabberBot on MyHostName.tld
DEBUG:jabberbot:*** username = raymii
DEBUG:jabberbot:*** type = chat
Now you can also see with a `git log` that the files are added. It works!
To make sure it keeps running, add a cronjob:
crontab -e
Then add the following:
*/5 * * * * dvcs-autosync
This runs dvcs-autosync every 5 minutes. It sees when it is already running,
then it does not run again.
#### Special steps for an ARM Chromebook
I have an ARM Chromebook with Ubuntu running in a chroot. In the chroot cron
does not run, so I have created a simple script to autostart dvcs-autosync on
login. You [can find it here on my github][11]. Add it to your window manager to
open on login and you are also set.
# Set up another Linux client
If you need to set up another Linux client, first install `encfs` and `dvcs-
autosync` as explained above. Also, configure `dvcs-autosync` with the existing
XMPP account and set up the dvcs-autosync cronjob.
Then, instead of creating an EncFS folder, clone the git repo with the encrypted
EncFS data:
git clone ssh://sync.raymii.nl/home/remy/autosync.git ~/secure
If you are using rsync.net, clone the repository like this:
git clone ssh://<server>.rsync.net/data5/home/<user>/autosync.git/ ~/secure
Also remember to add another ssh key to your rsync.net account (presuming you
already have added an ssh key to your account):
cat ~/.ssh/other_key.pub | ssh <user-id>@<server>.rsync.net 'dd of=.ssh/authorized_keys oflag=append conv=notrunc'
Make sure to change the address and path to your own server.
Also create the `~/share` folder:
mkdir `~/share`
Now you can mount the folder with EncFS:
encfs ~/secure ~/share
You can now also test if you create or remove files/folders on one Linux
Machine, they are also created or removed on the other Linux machine(s).
### Prepare the OS X client
I also use OS X machines, so I want to have secure access to my files there as
well. Luckily, that is possible with the above setup. The tools required are a
bit more spartan in setup, but after setup is just as simple in use. You have to
have XCode and the Command Line Developer Tools installed.
### Install OSXFUSE
Download the .pkg file from <http://osxfuse.github.io/> and install it. This is
needed for EncFS. OSXFuse is the continuation of MacFUSE, that seems to be
discontinued. OSXFuse works on both Lion and Mountain Lion.
#### Install EncFS
Make sure you have installed Homebrew (from <http://brew.sh/>). We use homebrew
to install a version of EncFS configured to use OSXFuse instead of MacFUSE. When
brew and OSXFuse are installed, use the following command to install EncFS:
brew install https://raw.github.com/jollyjinx/encfs.macosx/master/encfsmacosxfuse.rb
It takes a while to compile and build the required dependencies, `boost` for
example took 20 minutes on my 2012 Macbook Pro with an Intel Core i7.
When everything is installed and working continue to the next step.
#### Get the secure folder
This one is simple. Clone the git repository from the SSH server:
git clone ssh://sync.raymii.nl/home/remy/autosync.git ~/secure
If you are using rsync.net, clone the repository like this:
git clone ssh://<server>.rsync.net/data5/home/<user>/autosync.git/ ~/secure
Also remember to add another ssh key to your rsync.net account (presuming you
already have added an ssh key to your account):
cat ~/.ssh/other_key.pub | ssh <user-id>@<server>.rsync.net 'dd of=.ssh/authorized_keys oflag=append conv=notrunc'
Make sure to change the address and path to your own server.
Also create the `~/share` folder:
mkdir ~/share
#### Install dvcs-autosync and dependencies
First clone the repo of dvcs-autosync:
mkdir ~/src
cd ~/src
git clone git://gitorious.org/~olivierg/dvcs-autosync/olivierg-dvcs-autosync.git
cd olivierg-dvcs-autosync
Now build dvcs-autosync:
python setup.py build
sudo python setup.py install
Now download and build `xmpppy`. Save the file from
<http://downloads.sourceforge.net/project/xmpppy/xmpppy/0.5.0-rc1/xmpppy-0.5.0rc1.tar.gz>
to your home folder. Then extract and build it:
cd ~
tar -xf xmpppy-0.5.0rc1.tar.gz
cd xmpppy-0.5.0rc1
sudo python setup.py install
Then download and setup MacFSEvents (inotify for OS X):
git clone https://github.com/malthe/macfsevents.git
cd macfsevents
sudo python setup.py install
With all the above setup you are ready to continue and set up dvcs-autosync.
#### Set up dvcs-autosync
This is the same as for the Linux client. Copy the example file from [here][10]
to `~/.autosync` and edit at least the following sections:
[autosync]
path = ~/secure
Change the path to your freshly created secure folder (`~/secure`).
[xmpp]
username = you@yourXMPPserver.tld
password = Your_Passw0rd
alsonotify = you@yourXMPPserver.tld
And change the XMPP account data. That's it. You can change more things, but
that is all explained in the config file.
When you have set it up, start dvcs-autosync from the terminal:
dvcs-autosync
Now you should see all your files being synced. When it has caught up with all
the files, mount the EncFS folder:
encfs ~/secure ~/share
When you look in the ~/share folder now, you have all your files. Don't forget
to add a cronjob for dvcs-autosync:
crontab -e
Then add:
*/5 * * * * dvcs-autosync
With this all set up, you have your own, truly secure encrypted file
synchronization service! Well done.
### Pitfalls of EncFS
EncFS does have a few disadvantages. For me they don't weight up to all the
advantages.
* It leaks file modification/creation dates.
* It leaks file size.
* Some filenames can be derived from the encrypted form.
You can read a very good article about EncFS [here][12]. It explains all the
possibilities, but also all the pitfalls.
[There has also been a extensive code audit of EncFS which resulted in some
issues. Read this mailing list post to find out.][13]
### Race conditions
With multiple edits of a file, on different devices, then the file whichever
dvcs-autosync commits first is used as "master copy". The others receive an XMPP
notification, and incorporate a (5 second) wait time. When not online, as far as
I've seen in the last three weeks of intensive usage, the when a file on the
server is newer it is overwritten with a pull.
If you have any questions, you can always contact me via email.
[1]: https://www.digitalocean.com/?refcode=7435ae6b8212
[2]: http://i.imgur.com/0L5GxCR.png?2
[3]: https://en.wikipedia.org/wiki/EncFS
[4]: https://github.com/rmayr/dvcs-autosync
[5]: http://clients.inceptionhosting.com/aff.php?aff=083
[6]: https://clients.inceptionhosting.com/aff.php?aff=083
[7]: http://www.rsync.net/resources/howto/ssh_keys.html
[8]: http://blog.kozubik.com/john_kozubik/2010/02/git-and-subversion-support-at-rsyncnet.html
[9]: https://github.com/rmayr/dvcs-autosync/blob/master/README
[10]: https://github.com/rmayr/dvcs-autosync/blob/master/.autosync-example
[11]: https://github.com/RaymiiOrg/df/blob/master/dvcs.sh
[12]: http://movingtofreedom.org/2007/02/21/howto-encfs-encrypted-file-system-in-ubuntu-and-fedora-gnu-linux/
[13]: http://sourceforge.net/mailarchive/message.php?msg_id=31849549
---
License:
All the text on this website is free as in freedom unless stated otherwise.
This means you can use it in any way you want, you can copy it, change it
the way you like and republish it, as long as you release the (modified)
content under the same license to give others the same freedoms you've got
and place my name and a link to this site with the article as source.
This site uses Google Analytics for statistics and Google Adwords for
advertisements. You are tracked and Google knows everything about you.
Use an adblocker like ublock-origin if you don't want it.
All the code on this website is licensed under the GNU GPL v3 license
unless already licensed under a license which does not allows this form
of licensing or if another license is stated on that page / in that software:
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
Just to be clear, the information on this website is for meant for educational
purposes and you use it at your own risk. I do not take responsibility if you
screw something up. Use common sense, do not 'rm -rf /' as root for example.
If you have any questions then do not hesitate to contact me.
See https://raymii.org/s/static/About.html for details.