-
Notifications
You must be signed in to change notification settings - Fork 0
Getting Started
- One or more x86-64 machines.
- Linux kernel 2.6.27 or later
- glibc 2.9 or later
- The corosync and corosync lib package.
- QEMU 0.13 or later
- GNU Autotools
- pkg-config
- corosync devel package
- git (when compiling from source repo)
- liburcu
- optional:fuse-devel (for sheepfs)
Nearly every modern Linux distribution has x86_64 corosync binaries pre-built available via their repositories. We recommend you use these packages if they are available on your distribution.
For debian package based systems:
$ sudo aptitude install corosync libcorosync-dev
For RPM package based systems:
$ sudo yum install corosynclib-devel
For EL6 (RHEL, CentOS, SL, etc), the provided version of corosync is too old and you must install corosync from source. The following command will remove any existing copies of corosync.
$ sudo yum remove corosync corosynclib corosynclib-devel
If your distribution doesn't provide the corosync packages, or you prefer to compile from source:
$ git clone git://github.com/corosync/corosync.git $ cd corosync $ git checkout -b flatiron origin/flatiron $ ./autogen.sh $ ./configure --enable-nss $ sudo make install
See also: Corosync Config
QEMU 0.13 provides built-in support for sheepdog devices. Some distributions provide pre-built versions of this newer version of QEMU. If your distribution has an older version of QEMU or you prefer to compile from source, retrieve the latest QEMU and compile:
$ git clone git://git.sv.gnu.org/qemu.git $ cd qemu $ ./configure $ sudo make install $ cd ..
$ git clone git://github.com/collie/sheepdog.git $ cd sheepdog $ ./autogen.sh $ ./configure $ sudo make install $ cd ..
Please note, sheepdog supports a "make rpm" target which will generate an rpm package that can be installed on the local machine. To use this installation method, use the following instructions:
$ git clone git://github.com/collie/sheepdog.git $ cd sheepdog $ ./autogen.sh $ ./configure $ make rpm $ sudo rpm -ivh x86_64/sheepdog-0.* $ cd ..
- To launch the sheepdog daemon, either start it using '/etc/init.d/sheepdog start' or launch the sheepdog daemon by directly calling 'sheepdog <store_dir>' on each machine of the cluster. Make sure that TCP port 7000 is open for sheepdog communication.
Note: At this time, the sheepdog init.d script is hard-coded to use /var/lib/sheepdog as the store directory. If you wish to use another directory, you will need to always start sheepdog using the 'sheep <store_dir>' call.
$ sudo /etc/init.d/sheepdog start
Or;
$ sudo sheep /var/lib/sheepdog
The /var/lib/sheepdog will be where sheepdog will store objects. The directory must be on a filesystem with an xattr support. In case of ext3 or ext4, you need to add 'user_xattr' to the mount options.
$ sudo mount -o remount,user_xattr /var/lib/sheepdog
- Format sheepdog cluster
$ collie cluster format --copies=3
It is enough to run this command on just one machine. "--copies" specifies the number of default data redundancy. In this case, the replicated data will be stored on three machines.
Sheepdog has the back-end store to store the data object persistently on the node basis. Currently you have two choices, one is called Simple Store (default), the other Farm. Farm is supposed to provide advanced features such as cluster-wide snapshot, faster recovery, data de-duplication(not implemented yet) and so on.
You can choose to use Farm as the back-end store by following command:
$ collie cluster format -b farm --copies=3
- Check cluster state
Following list shows that Sheepdog is running on 32 nodes.
$ collie node list Idx Node id (FNV-1a) - Host:Port ------------------------------------------------ 0 0308164db75cff7e - 10.68.13.15:7000 1 03104d8b4315c8e4 - 10.68.13.1:7000 2 0ab18c565bc14aea - 10.68.13.3:7000 3 0c0d27f0ac395f5d - 10.68.13.16:7000 4 127ee4802991f308 - 10.68.13.13:7000 5 135ff2beab2a9809 - 10.68.14.5:7000 6 17bd6240eab65870 - 10.68.14.4:7000 7 1cf35757cbf47d7b - 10.68.13.10:7000 8 1df9580b8960a992 - 10.68.13.11:7000 9 29307d3fa5a04f78 - 10.68.14.12:7000 10 29dcb3474e31d4f3 - 10.68.14.15:7000 11 29e089c98dd2a144 - 10.68.14.16:7000 12 2a118b7e2738f479 - 10.68.13.4:7000 13 3d6aea26ba79d75f - 10.68.13.6:7000 14 42f9444ead801767 - 10.68.14.11:7000 15 562c6f38283d09fe - 10.68.14.2:7000 16 5dd5e540cca1556a - 10.68.14.6:7000 17 6c12a5d10f10e291 - 10.68.14.13:7000 18 6dae1d955ca72d96 - 10.68.13.7:7000 19 711db0f5fa40b412 - 10.68.14.14:7000 20 7c6b95212ee7c085 - 10.68.14.9:7000 21 7d010c31bf11df73 - 10.68.13.2:7000 22 82c43e908b1f3f01 - 10.68.13.12:7000 23 931d2de0aaf61cf5 - 10.68.13.8:7000 24 961d9d391e6021e7 - 10.68.13.14:7000 25 9a3ef6fa1081026c - 10.68.13.9:7000 26 b0b3d300fed8bc26 - 10.68.14.10:7000 27 b0f08fb98c8f5edc - 10.68.14.8:7000 28 b9cc316dc5aba880 - 10.68.13.5:7000 29 d9eda1ec29c2eeeb - 10.68.14.7:7000 30 e53cebb2617c86fd - 10.68.14.1:7000 31 ea46913c4999ccdf - 10.68.14.3:7000
- Create a 256 GB virtual machine image of Alice.
$ qemu-img create sheepdog:Alice 256G # default as raw image $ qemu-img create -f qcow2 sheepdog:Alice 256G # you can enjoy all the qcow2 features
- You can also convert from existing KVM images to Sheepdog ones.
$ qemu-img convert -t directsync ~/amd64.raw sheepdog:Bob
- See Sheepdog images by the following command.
$ collie vdi list name id size used shared creation time object id -------------------------------------------------------------------- Bob 0 2.0 GB 1.6 GB 0.0 MB 2010-03-23 16:16 80000 Alice 0 256 GB 0.0 MB 0.0 MB 2010-03-23 16:16 40000
- Boot the virtual machine with sheep daemon started on local node
$ qemu-system-x86_64 sheepdog:Alice
- Boot the virtual machine without sheep daemon started on local node
$ qemu-system-x86_64 sheepdog:192.168.10.2:7000:Alice
This suppose you have one sheep started up on the node with the IP 192.168.10.2 and port 7000
- Sheepdog supports local cache called object cache. Object cache caches data and vdi objects on the local node. It is at higher level than backend store. This extra cache layer translate gateway requests into local requests, largely reducing the network traffic and highly improve the IO performance. Dirty objects will be flushed to the cluster storage by 'sync' request from guest OS. You have to run latest QEMU to work with object cache. To enable the object cache, you can start QEMU by following command:
$ qemu-system-x86_64 -drive file=sheepdog:Alice,cache=writeback
- Create a snapshot
$ qemu-img snapshot -c name sheepdog:Alice
- After getting snapshot, a new virtual machine images are added as a not-current image.
$ collie vdi list name id size used shared creation time object id -------------------------------------------------------------------- Bob 0 2.0 GB 1.6 GB 0.0 MB 2010-03-23 16:16 80000 Alice 0 256 GB 0.0 MB 0.0 MB 2010-03-23 16:21 c0000 s Alice 1 256 GB 0.0 MB 0.0 MB 2010-03-23 16:16 40000
- You can boot from the snapshot image by specifying snapshot id
$ qemu-system-x86_64 sheepdog:Alice:1
- Create a Charlie's image as a clone of Alice's image.
$ qemu-img create -b sheepdog:Alice:1 sheepdog:Charlie
- Charlie's image is added to the virtual machine list.
$ collie vdi list name id size used shared creation time object id -------------------------------------------------------------------- Bob 0 2.0 GB 1.6 GB 0.0 MB 2010-03-23 16:16 80000 Alice 0 256 GB 0.0 MB 0.0 MB 2010-03-23 16:21 c0000 s Alice 1 256 GB 0.0 MB 0.0 MB 2010-03-23 16:16 40000 Charlie 0 256 GB 0.0 MB 0.0 MB 2010-03-23 16:23 100000
- Run the shutdown command on the one of the cluster machines.
$ collie cluster shutdown
This command stops all the sheep processes on the cluster.
- Debian squeeze amd64