Openfiler High Availability NAS Server

The 1.1 release of Openfiler is the first release with full cluster and replication support. Previous versions and releases do not have cluster support and there are no plans to backport support to older releases.

The Open Source Distributed Replicated Block Device (DRBD) software is used to provide a mechanism of copying individual disk data blocks from a live server to a secondary server usually located at a remote site for disaster-recovery purposes. DRBD is the equivalent of RAID 1 except that the data blocks are sent over a network to disks located in a remote server. DRBD can work over VPN links for added security if a dedicated secure link is not available between the primary active server and secondary passive server. DRBD is *not* a filesystem and does not support simultaneous access to disk blocks by the primary and secondary nodes.

DRBD works along the premise that only one server or node will be accessing the disk blocks at any given point in time. The node primary node, the node with read and write access to the DRBD block device, is the source and the secondary node is the target. And data written to the source is sent over a dedicated network link to the target and written to the local disk on the target before being written to the local disk on the source and the write acknowledged to the application performing the write operation. Data reads are however done locally from the primay node. In the event of a failure of the primary node – even as catastrophic as total destruction from fire, the data is still safe and available to serve out from the secondary node.

DRBD is an abstraction layer that sits between the local disks and upper level volume managers and filesystems. It is a form of virtualisation, where the local disks are no longer accessed directly by the file system or volume manager. Rather operations are intercepted by the DRBD layer and depending on what operation is to be performed, either one of three possible replication protocols is used to redirect the operation or the operation is passed through to the local disk. If the operation is a read request, the DRBD layer accesses local disk, reads the required blocks and sends the data back up to the requesting applicaiton. If the operation is a write request one of the following protocols is used to redirect the write:

To gain a deeper understand of DRBD and its modus operandi, please visit www.drbd.org.

Heartbeat is an Open Source application that provides a mechanism for monitoring resources in a cluster and manages the process of automatically failing over these resources between nodes participating in the cluster. It is a “cluster manager” for all intents and purposes. A resource is any service, application or system component that needs to be kept continuously available to clients accessing them regardless of the state, healthy or otherwise, of the host/node providing the resource. Examples of resources that can be managed by heartbeat include file services such as smb/nfs/ftp; filesystems such as ext3; IP addresses. The aptly named hearbeat sends a network ping between nodes in the cluster at predetermined intervals to ensure that the nodes are “alive” and in a serviceable state. Should a ping not return within an expected timeframe and subsequent pings fail up to a limited threshold, it is assumed the unresponsive node is “dead” and a prescribed chain of events is initiated to acquire the resources that were hitherto being provided by the now failed node.

In an Openfiler replicated cluster, heartbeat is used to manage all the file services, the LVM volumes and filesystems as well as the state of the replicated block device, DRBD. In the event of a failure of the primary node providing services to clients on the network, heartbeat will take care of ensuring service and resource bringup on the secondary node and can even be configured to automatically switch services and resources back to the original primary node when that node has been brought back to a serviceable state. Heartbeat ensures that resource bringup upon a failover instance is coherent and as transparent as possible to the client machine. In many cases, the client will even be oblivious to the fact that services have been failed over. At most, during a failover operation, the client machine will experience a short blip where access to data is suspended.

To learn more about Heartbeat, please visit www.linux-ha.org.

Two identical server machines are required to act as primary and secondary nodes in the cluster. While it is possible to have systems with different specificaitons perform these roles, it is not recommended practice. At the very least the disk and network device configurations should be identical. It is possible to take advantage of network device bonding to increase the network bandwidth between the nodes in the cluster and thereby increase overall write performance. If possible to achieve, a dedicated network link, bypassing network hubs and switches, between the two nodes is preferred. A minimum of two network interfaces is required for a cluster setup. Better performance and resiliency will be achieved however, with five network interfaces. Two network interfaces, bonded, are used for the replication link between the primary and secondary nodes, two network links, again bonded, are used for service provision and the fifth link is dedicated for heartbeat pings.

To configure the hardware and network, perform the following steps;

To install the cluster software, perform the following steps;

To configure a secure link between the two cluster nodes, perform the following steps;

At least two DRBD block devices are required. One for the cluster metadata and one for application data. The following steps must be performed identically on both nodes. To configure DRBD, please proceed as follows:

Heartbeat makes use of three configuration files. The /etc/ha.d/ha.cf and /etc/ha.d/authkeys files are configured by the administrator manually and the /etc/ha.d/haresources file is managed by Openfiler. The steps to configure heartbeat must be performed identically on both nodes. To configure heartbeat, proceed as follows;

To start the DRBD service, proceed as follows;

To create the volume group for Openfiler data partitions, proceed as follows;

The following steps should be performed in final preparation for cluster operation;

This is the final step in the process;

Trobleshooting 

================================ 

IF it doesn’t boot properly edit the grub then  

 remove .snmp from grub

===================================

Installing openfiler into into USB Device

http://www.openfiler.com/community/forums/viewtopic.php?pid=4265#p4265
It is possible, following the following steps:
– disable your internal HD’s, its not really needed but to be sure not to mess up very handy. I had Grub take out my internal HD of my laptop 😉
– boot from the latest Openfiler iso (2.2 respin 2)
– at the boot prompt, for textmode, type: expert text
  type, for graphical: expert
(This way anaconda will let you choose your usb stick to install to)
– Install as usual up and until the reboot message.
– reboot
– be surprised, it does boot but ends in a kernel panic, don’t panic 😉
– the initrd doesn’t know about USB storage so at the end of the boot process in the ramdisk it will switch to a blackhole instead of a filesystem on disk.
– the short way, fix is on the following URL:
http://simonf.com/usb/

The slightly longer and edited version follows, with my comments between ():

mount /dev/sda3 /mnt/sysimage
mount /dev/sda1 /mnt/sysimage/boot
chroot /mnt/sysimage
(your /dev/sdaX might vary depending on your installation choices!)

(I haven’t tried the mkinitrd stuff but the alternative does work.)

cp /boot/initrd-2.X.X.img /tmp/initrd.gz
gunzip /tmp/initrd.gz
mkdir /tmp/a
cd /tmp/a
cpio -i < /tmp/initrd
vi init
(find the line with ‘insmod /lib/sd_mod.ko’)
( insert the following beneath it)
insmod /lib/sr_mod.ko
insmod /lib/ehci-hcd.ko
insmod /lib/uhci-hcd.ko
sleep 5
insmod /lib/usb-storage.ko
sleep 8
(save)
cd /lib/modules/`uname -r`/kernel/drivers
cp usb/storage/usb-storage.o /tmp/a/lib
cp usb/host/ehci-hcd.ko /tmp/a/lib
cp usb/host/uhci-hcd.ko /tmp/a/lib
cd /tmp
find . | cpio -c -o | gzip -9 > /boot/usbinird.img
(you could also replace your original initrd)
(edit /boot/grub/menu.lst to have a new boot option with your custom usbinitrd.img)
(this needs to be done for every kernel update!!! but could be automated with a script)
(To be honest I would like to see this appear in the official rPath distro since it is so simple to get it to work, when you know what you’re need looking for ;-))

Further digging around with Google revealed that you can add a few things to make life either easier for yourself or your USB stick’s life.
For example move /tmp, /var/lock, /var/log, /var/run, /var/tmp to tmpfs
further edit /etc/fstab to include the flag noatime with your /boot and / mountpoints and to preserve history it is also possible through two init scripts to copy those /var directories to tmpfs on startup and back to usbstick on shutdown.
I’ll include my fstab en preinit script and later my preshutdown script.
LABEL=/                 /                       ext3    defaults,noatime 1 1
LABEL=/boot             /boot                   ext3    defaults,noatime 1 2
/dev/devpts             /dev/pts                devpts  gid=5,mode=620  0 0
/dev/shm                /dev/shm                tmpfs   defaults        0 0
/dev/proc               /proc                   proc    defaults        0 0
/dev/sys                /sys                    sysfs   defaults        0 0
# LABEL=SWAP-sda3         swap                    swap    defaults        0 0
tmpfs             /tmp             tmpfs     defaults,noatime 0 0
tmpfs             /var/lock         tmpfs     defaults,noatime 0 0
tmpfs             /var/log         tmpfs     defaults,noatime 0 0
tmpfs             /var/run         tmpfs     defaults,noatime 0 0
tmpfs             /var/tmp         tmpfs     defaults,noatime 0 0

Further this is my preinit script
#!/bin/bash

echo “creating tmpfs in /var/log”
mount -n -t tmpfs tmpfs /var/log
echo “copy /var/log”
cp -a /var/log_persistent/* /var/log

mount -n -t tmpfs tmpfs /var/lock
echo “fixing /var/lock structure”
cp -a /var/lock_persistent/* /var/lock

mount -n -t tmpfs tmpfs /var/run
echo “fixing /var/run structure”
cp -a /var/run_persistent/* /var/run

echo “done, resume normal operations….”

made a link to it:

chmod 755 /sbin/preinit
cd /etc/rc3.d
ln -s /sbin/preinit S00preinit

If you want it in other runlevels as well then add it in rc1.d and rc2.d etc..
I made the following directories in /var:
/var/run_persistent
/var/log_persistent
/var/lock_persistent
and booted without the tmpfs entries in fstab and then copied the files from /var/log to /var/log_persistent and then activated tmpfs in fstab after which it worked.
What I need todo is add the script to copy everything back on shutdown and/or add a cronjob copying everything back every hour or so.

Hopefully it is clear what needs to be done, if not let me know through the forum.

Joop

This is what got it working for me EXCEPT the only problem part of this post was:

cd /tmp
find . | cpio -c -o | gzip -9 > /boot/usbinird.img

Should be:
cd /tmp/a
find . | cpio -c -o | gzip -9 > /boot/usbinird.img

I just got mine booting on a 2GB Udrive on the 64bit version of Openfiler 2.2

Also, I manually partitioned on the anaconda installer:
75 MB – /boot *forced primary ext2
100 MB – swap
rest MB – / ext3