Btsync on raspberry pi

sketch > Rpi (Raspberry pi)

Get involved! contact me: rpi (at] arcol [dot) hu

Bittorrent sync (btsync), aka. Why not git-annex-assistant?

I started with git-annex-assistant, but I think it is just not suited for seemless file sharing.

I would love to be prooved wrong, but I think it can not be made work on multiple raspberry pis.

System overview

In the most basic setup a TV connected raspberry pi with XBMC and Btsync installed with a big harddrive required.

You can have multiple devices (work computer, laptop 1, laptop 2) and the raspberry pi.
The raspberry pi is the central backup unit, supposed to be connected 24/7, it can always serve you the files whichever laptop you turn on.

Mine has a 2.5" (laptop) harddrive with 1TB capacity plugged to the raspberry pi. It is big enough for a laptop backup, and also has some room for future improvements.

More reliable setup

One of the design goal is to install at least 3 raspberry pi at different locations (one at home, one at your mom's place, one at your grandma's place, etc). All the three raspberry pi is identical, they synchronize also between themselves, and can serve you 3x the speed of a single raspberry pi.

So basically it is at least as reliable as a commercial (drxxbxx account), ready at any time at your full internet speed serving your beloved files.

It is more than a file synchronizer

As the raspberry pis are installed with XBMC, you can play files from the harddrive straight away.
Some usecases:

  • You arrived from vacation and you want to share photos with your mom, you put it in your btsync folder, and while you are driving to your mom's place it is already synchronized ready to be watched
  • You want to share videos with your grandmother, you simply put in btsync folder, and shows up automagically at your grandma's TV.
  • Your laptop is always backed up.

Preferred file layout

  1. You label the harddrive as "btsync" when formatting to ext4 filesystem
  2. It is mounted on raspberry pi (raspbmc) as /media/btsync
  3. on the harddrive you stick to this layout
/media/btsync/  # <-- harddrive mounting point, because it is labeled 'btsync'
             /swapfile # A 4GB swapfile for additional memory 
                       # (required by btsync to run flawlessly)
             /btsync/  # <-- btsync main directory, everything is synced inside it
                    /movies   # <-- your vacation movies
                    /series   # <-- your video series about a theme 
                              #     (like bicycle learning)
                    /photos   # <-- your vacation photos to be shared
                    /music    # your guitar plays or musics downloaded from opsound.org
                    /PARTITION_NAME # your laptop's partition name, like 
                              # /mnt/dat on laptop 1, your main backup target 
                              # (mine is 60GB)
###########  btsync related directories: ########
             /.btsync # <-- btsync puts its temporary files, .db files here
             /btsync/config_raspi #<-- every config required by raspberry pi (for new install)                
                    /.SyncArchive # deleted files from btsync repository goes here for 30days.
                    /.SyncIgnore  # files to be ignored like .DS_Store, Thumbs.db, etc
                    /.SyncID      # 32 bytes long ID, it is required by btsync
                                  # if not there, the harddrive is not recognized by btsync

Caution aka BIG FAT WARNING

!!! WARNING !!!
Never ever put your git repositories directly under btsync. 
Only copy it over there on your local machine.
One of git's drawbacks are the millions of minuscule files, 
it is really easy to screwup by a synchronizer program.

I have a 'projects' dir under desktop folder 
(/home/your_username/Desktop/projects) where 
I put everything what I work on (everything!).
Eg. multiple git repos, my ramblings, everything. 

Once a week I copy the 'projects' folder over btsync folder 
(todo: could be automatized).
!!! WARNING !!!

RELIABILITY aka. not enough ram

The raspberry pi has 512MB of ram, from which you can assign 128MB for GPU (seamless video playback).

So you end up with 384MB of ram (in reality only 374, some rounding errors here and there).

Btsync requires a fixed amount of RAM 3-400bytes/file when syncing. So theoretical maximum is around 1 million file. But in reality it is much lower, for me 119617 synced file (still not finished) eats 640MB of ram on raspberry pi.

If you do not enable swapfile, then reliability is absolutely zero.

SWAPping kills performance, no?

On raspberry pi, not really.
It is a super-slow cpu, so using swapfile, the CPU still never iowaits, what would normally do on a 10 year old desktop cpu.

Here is a proof (partial top output):

%Cpu(s): 97.1 us,  2.6 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem:    383712 total,   371804 used,    11908 free,    20560 buffers
KiB Swap: 16777200 total,   449100 used, 16328100 free,    12720 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
 3151 root      20   0  639m 214m 1280 S  96.2 57.1   3591:46 btsync

This means 97.1% cpu is used, 0.0% iowaits (wa), and btsync eats 96.2% cpu.

Some metric data

The raspberry pi (xbmc also running, so it gets only 40-60% of cpu time) can btsync index 1 file per second (for small files), and the speed required for this is about 4-8kiB/s (laptop -> raspi), and 3-5KiB/s (raspi -> laptop).
So almost no speed required.
For big files (10MB and above), the speed can go up to 800KiB/s - 1MB/s, but not above.
(For comparison scp file transfer is 1.2-1.4MB/s)

From this, you can have a pretty good idea, what initial synchronization time you can expect.

A real world example:
My laptop has a 86.2GiB data (181854 files).
I started synchronize it 30th May 2014,
and still 66GiB is needed to synchronize two days later (1st June 2014)
and it is at 106863 files.

Update 20th June 2014
Laptop: 82.8 GB in 157749 files
Raspberry Pi: 34.8 GB in 118259 files

Pretty slow, but hopefully only initially is it that crap.
I suspect it is really Raspberry pi CPU bound, rather then network.

Update 7th July 2014
It finished!!!
Laptop: 87.0 GB in 157970 files
Raspberry Pi: 87.0 GB in 157970 files

Now on, it is full speed. It was full speed (900kB/s - 1.1MB/s) like the past 3 weeks or so. But I was away with a pretty crap internet connection (10kiB/s upspeed), and the laptop was switched off overnight.

The memory usage of the raspberry pi is not that high, as the beginning:

             total       used       free     shared    buffers     cached
Mem:        383712     372008      11704          0       4788      30848
-/+ buffers/cache:     336372      47340
Swap:      4194300      46440    4147860

the relevant line of top output:

3144 root      20   0  284m 207m 1964 S  79.5 55.4   3538:32 btsync

Update 7th July 2014
I killed dropbox, and migrated over btsync completely.
That means, I migrated my scrapbook over btsync, now it looks like this:
93.5 GB in 297481 files

So it is still 5.9GB files to be uploaded. Also I figured out the btsync speed.
Each file upload is like this: first second 5KiB/s, second second 14KiB/s, then it starts to ramp up to 1.1MiB/s.
The problem starts when you have 10000 files each of it under 10KiB.
So the upload speed is around 7-10KiB/s only.

Linux/udev bugs

BUG1:
I also prepared (workarounded buggy linux behaviour), when you unplug the drive.
Unfortunately there is no way to permit only two application (btsync, xbmc) swapping, and none of the other programs. So in case of hard drive disconnect only these two application would crashes. Right now it is pretty random behaviour.

If you plug-unplug in short period (less then 10 minutes), then there were no swapping, so you should be fine.

BUG2:
Udev is also buggy, there is no event which triggers after the mounting happened. The last possible event is just before the mounting point is created.

BUG3:
If you name your udev rule as 100-my-mount-rules. It is triggered after rule 10, and before rule 2. The best is to name your rule as 992, 993, 994, etc.

Udev rules

When you plug in the prepared (labelled 'btsync', formatted ext4) 1TB usb hard drive,
it magically launch a script which adds the swap file and start the btsync.

cat /etc/udev/rules.d/992-my-mount.rules 
ACTION=="add", SUBSYSTEM=="block", ENV{ID_TYPE}=="disk", ENV{ID_FS_LABEL}=="btsync", RUN+="/home/pi/scripts/my-mount-script.sh"
ACTION=="remove", SUBSYSTEM=="block", ENV{ID_TYPE}=="disk", ENV{ID_FS_LABEL}=="btsync", RUN+="/home/pi/scripts/my-umount-script.sh"

Mount/unmount scripts

There is no such rule in udev, which triggers *after* mounting happened.
So by the time the script is running there are still no mounting points in the system (/media/btsync).
If you wait inside the script (sleep 2), the udev does not finish until the script has not quit.
The default timout is 60sec.

To WORKAROUND this complicated and buggy behaviour, we create a daemon
inside the script, which launch in the background, does not prevent
udev finishing mounting. And after in less then 2 second udev does finish the mounting.

The script is my-mount-script.sh, the helper daemon is my-mount-script-helper-daemon.sh.

pi@raspberrypi ~ $ cat scripts/my-mount-script.sh 
#! /bin/sh

# [2202.846354] [3027] 
# udev_device_update_db: created db file '/run/udev/data/b8:1' for 
# '/devices/platform/bcm2708_usb/usb1/1-1/1-1.3/1-1.3.5/1-1.3.5:1.0/host4/target4:0:0/4:0:0:0/block/sda/sda1'
# [2202.858284] [3027] udev_monitor_send_device: passed -1 bytes to netlink monitor 0x197fcb0

# BIG FAT WARNING:
# Unfortunately after this script finished by udev 
# (either the script quits or udev kills the script after timeout(60 sec).
# The actual mountpoint (/media/btsync) will be created. So this script is just too early executed.
# No other udev rules exists which triggers *AFTER* the mountpoint is created.

# WORKAROUND
# 1. We create a helper script and launch it as a deamon from this script, 
#    so udev can go ahead and finish the mounting.
# 2. We dummy-wait from the helper daemon.

LOGFILE="/home/pi/udev.log"
SWAPFILE="/media/btsync/swapfile"

# First say hello, that we have inserted a disk
echo " " >> $LOGFILE
echo "USB disk labelled 'btsync' inserted: `date +%Y-%m-%d\ %H:%M:%S\ CEST`" >> $LOGFILE

# swapfile created at /media/btsync/swapfile using this command: (1024*4096 = 4GB)
# sudo dd if=/dev/zero of=./swapfile bs=1024 count=4194304
# sudo chmod 666 ./swapfile
# mkswap ./swapfile

# kill btsync
killall btsync

# Switch swap off through dphys-swapfile
echo "dphys-swapfile swapoff"
dphys-swapfile swapoff

# Make a new configuration file :
echo "CONF_SWAPSIZE=4096" > /etc/dphys-swapfile
echo "CONF_SWAPFILE=/media/btsync/swapfile \n" >> /etc/dphys-swapfile

# Launch the helper daemon, which finishes the job, after Sir. udev finishes its shit.

echo "Launching helper daemon" >> $LOGFILE
nohup /home/pi/scripts/my-mount-script-helper-daemon.sh 0<&- &>> $LOGFILE &

Helper daemon:

pi@raspberrypi ~ $ cat scripts/my-mount-script-helper-daemon.sh 
#! /bin/sh

# This is the HELPER DAEMON because of udev.
# For WARNING see the main (my-mount-script.sh) file.

LOGFILE="/home/pi/udev.log"
SWAPFILE="/media/btsync/swapfile"
echo "This is the helper daemon" >>$LOGFILE

SLEEPCOUNT=0
while [ ! -f "$SWAPFILE" ]
do
  echo "$SWAPFILE not present, we are sleeping for 2 sec ($SLEEPCOUNT in total)" >>$LOGFILE    
  SLEEPCOUNT=$((SLEEPCOUNT+2)) 
  sleep 2
done
echo "ls /media/btsync" >> $LOGFILE
echo `ls /media/btsync` >>$LOGFILE

# Switch swap on again, this time with the 4GB file
# dphys-swapfile swapon
# DONT use this stupid dphys-swapfile shell script, it 
# can not pass arguments to swapon command
# and we want to assign priority to new swaps

# just to be on the safe side
swapoff /media/btsync/swapfile 
# start with swap (does the same as dphys-swapfile, except priority assignment)

SWAPPRIORITY=`cat /proc/swaps |tail -1 |cut -f5`
if [ $SWAPPRIORITY -gt 99 ]
then
  SWAPPRIORITY=$((SWAPPRIORITY + 10))
else
  SWAPPRIORITY=100
fi

echo "swapon -p $SWAPPRIORITY /media/btsync/swapfile" >>$LOGFILE 
swapon -p $SWAPPRIORITY /media/btsync/swapfile 

SWAPCURRENT=`cat /proc/swaps |tail -1 |cut -f3`
SWAPCURRENTFULL=`cat /proc/swaps |tail -1`

if [ $SWAPCURRENT -gt 500000 ]
then
  # Start btsync
  echo "btsync is starting, swapsize: $SWAPCURRENT" >> $LOGFILE
  /etc/init.d/btsync_initd start
else
  # Kill btsync
  echo "killall btsync (not enough swap space)" >> $LOGFILE
  echo "swapcurrent is $SWAPCURRENT" >> $LOGFILE
  echo "full swap output: " >> $LOGFILE
  echo "$SWAPCURRENTFULL" >> $LOGFILE
  killall btsync

fi

# Restore the old configuration file, in case of power outage :
echo "CONF_SWAPSIZE=100" > /etc/dphys-swapfile

echo "we HAPPILY quit :)))" >> $LOGFILE

Suspend/resume btsync

killall -STOP btsync # suspend btsync
killall -CONT btsync # resume btsync

# The same goes for xbmc:
killall -STOP xbmc.bin # suspend XBMC
killall -CONT xbmc.bin # resume XBMC

How to check progress

$ top
# then short process by memory by pressing 'Shift+m' (M) buttons

$ free
And watch the overall memory usage, example:
             total       used       free     shared    buffers     cached
Mem:        383712     374304       9408          0       3624      91668
-/+ buffers/cache:     279012     104700
Swap:     16777200     526972   16250228

You see? It is currently using 901MB in total

# this command lists the last 100 files changed inside a directory. 
# Takes about 10mins to complete.
pi@raspberrypi /media/btsync/btsync $ ls -ArRt | tail -n 100

# this command lists all directories' size.
$ for i in filmek filmek_sorozatok/ dat/*; do du -sk $i ; done

DynDNS

crontab wget -O /dev/null -q

/var /tmp

Move it elsewhere then the SD card

http://www.cyberciti.biz/faq/howto-mount-tmp-as-separate-filesystem-with-noexec-nosuid-nodev/

http://mywayonlinux.blogspot.hu/2014/05/raspberry-pi-arch-linux-arm.html

Forced restart

uptime:
17:42:47 up 17 days, 4:54, 2 users, load average: 5.87, 4.91, 3.49

XBMC completely froze the video, no way to restart it, kill or kill the whole Xorg.
Even shutdown command fails to finish.
Thank God, btsync continues where it left off. Does not start from the very beginning.

Possible bug/Todo

Somehow the swapfile is created *before* the mounting occured (only checked to /media/btsync directory, and not the actual mounting point), and the swap file filed the sdcard (root partition):

/media/btsync/swapfile <-- normal file on the root (/) filesystem
/media/btsync1 <-- the external hard drive was mounted here, 
                          because the /media/btsync directory already existed.

Needs some more investigation.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-NonCommercial-NoDerivs 3.0 License