 | |||||||||||
| |||||||||||
 |
|||||||||||
| |
| ||||||||||||||||
| ShaoLin HA Cluster 1.0 Installation Manual and Operations Guide | ||
|---|---|---|
| Prev | Chapter 1. Introduction to ShaoLin HA Cluster | Next |
ShaoLin HA Cluster is a middle-ware install into your Linux OS. The software consists of a number of components, this include some management process and kernel modules. The software is loaded in the very beginning stage of system boot. ShaoLin HA Cluster enables two computers to share a single set of hard drives or a storage subsystem simultaneously. The challenge of simultaneous access to a single storage is the issue of data synchronization and corruption. However, total storage sharing enables another computer to reuse data from a failed server. The technique of enabling two computers to share one storage is to make sure there is always exactly one writer while the other computers are the readers of the shared storage.
Two machines use a shared storage system. When one machine boots up first, that machine will be the ACTIVE NODE. When the second machine boots up and detects there is an ACTIVE NODE by listening from the heartbeat messages of the ACTIVE NODE, the second machine becomes the STANDBY NODE. The STANDBY NODE then monitors the status of the ACTIVE NODE using a specialized heartbeat protocol. If any unrecoverable error is detected, the STANDBY NODE will failover the complete system which will restart or shutdown the ACTIVE NODE and switch the STANDBY NODE to ACTIVE NODE.
Linux has a special feature called initial RAM disk (initrd). The initrd image is loaded right after the start of the Linux kernel. The initrd allow you to start anything prior real system initialization. Sometimes it is used to load system drivers such as file system drivers and host bus adapter(HBA) drivers, before the kernel really mount the filesystems and run initialization over it. The loading of the Linux kernel and the initrd only performs a read-only access to the boot disks, and since the initrd is a RAM disk image, it will not produce and change of physical data reside on the storage system. ShaoLin HA Cluster make use of the initrd feature of Linux, by generating a special initrd image which contains the ShaoLin HA Cluster software components. This allow the cluster services to be started before the system initialization or System V init.
The selection of Linux kernel and initrd images are done by the boot loader. After installed with ShaoLin HA Cluster to your system. An entry will be added to your boot loader (either LILO or Grub, see Section 10.1.1 for more information) . When booted with the cluster enabled initrd, the cluster service will be started. It first checks the status and existence of the other cluster. If the other cluster exists and is working properly. ShaoLin HA Cluster immediately turns the server into a hot standby state, entering a process which continuously monitors the active cluster. If the other cluster fails or error is detected from the other cluster, ShaoLin HA Cluster will activate the STANDBY NODE(the machine which the cluster process it is running on) and take over the entire system by continue the boot process of the Linux OS, which later reinitialize all services. Since both machines uses the same set of disks, it ensures a consistent application configurations and data over two machines without the need of extra synchronization. This technique is known as the System Image Oriented clustering.
ShaoLin HA Cluster makes the STANDBY NODE in an uninitialized hot standby state. If a heartbeat channel timeouts or failure is reported. The STANDBY NODE will take over the entire system by shutdown or reboot the ACTIVE NODE. The ACTIVE NODE will try to flush all its I/O buffers to disks before failover to avoid data lost. After the ACTIVE NODE is complete offline, then the STANDBY NODE will reuse all the resource of the previous ACTIVE NODE, reinitialize to the desired system running level together restart all the application services. The time required to take over depends on the time of system initialization and application start time. Usually it will be faster on a high-speed machine, and slower in machines which has a large shared storage which might result in long file system checks in case of file system corruption.
The heartbeat operation between cluster nodes are done by the Kernel heartbeat daemon (KHBD). The design of KHBD is to make sure heartbeat messages are sent reliability other serial port of Ethernet by the Linux kernel. KHBD runs at kernel space which make sure the Linux kernel of the ACTIVE NODE is correctly monitored by the STANDBY NODE. The kernel heartbeat technology also ensures the heartbeat message is sent to the STANDBY NODE correctly in terms of time without delay, since the task scheduler of the kernel heartbeat is driven by the system real-time clock. This assures the heartbeat timing is correct and reliable. KHBD also ensure itself is crash safe, even application crashes occurs or even the system is being hacked, it ensures heartbeat will still operate correctly and reliable in all cases.
The ShaoLin HA Cluster Manager is responsible for all user space communications with the kernel. It also receives all heartbeat messages from the other cluster, to check the overall healthiness of the other cluster. It also provides a remote access server interface for remote control of the clusters.
The application availability control feature is used in the ACTIVE NODE only. It is used to monitor a specific application and tries to recover it if error occurs. If it finds the application contains unrecoverable errors, it will signal the STANDBY NODE to failover the system. This is useful to recover application crashed due to software bugs or possibly other application errors.
The basic ShaoLin HA Cluster software enables failure detection of operating system and hardware. To specifically monitor an application's behavior, customization is necessary. The Cluster Manager has a modularized interface for communication with individual Service Monitoring Agents (SMA). Special SMAs can be installed into the system to monitor a specific service (e.g. MySQL, Apache etc.). SMA is loaded by Cluster Manager and is used to monitor the desired service provided by the SMA. SMA is useful to detect DOS attacks, recover overloaded applications, and recover application errors. The use of SMA ensures the serviceability of the application and is suggested to use by mission critical servers. Please see ShaoLin HA Cluster SMA information page for information about supported applications.
A client software as the user interface for controlling the clusters at local and remotely at the console. It can communicate with Cluster Manager using a socket interface either with Unix socket or TCP/IP. It provides a menu driver interface for users to interactively monitor and control the clusters in real-time through network or at the console. By using Cluster Configurator which is a web-based configuration interface in Webmin, it is also possible to control the clusters through a web browser.
