Cluster Administration Tool
Table of Contents
Introduction
ht_cluster is a simple tool that can be used to automate the execution of tasks on a large cluster of machines. It is the recommended tool for administering a Hypertable cluster and is the tool that is referenced throughout the documentation.
ht_cluster was developed as a replacement for Capistrano, a Ruby-based remote server automation tool. It is based on the same concepts as Capistrano, but improves on it in a number of ways, most notably usability. The biggest problem with Capistrano and the primary motivation for writing ht_cluster is Capistrano's dependence on Ruby and a number of Ruby gems. These external dependencies can be problematic on some systems and a stumbling block to getting it installed. ht_cluster, on the other hand, has no external dependencies and works "right out of the box".
The tool is driven off of a configuration file called cluster.def, the cluster definition file. This file contains definitions for roles and tasks. A role is a named set of machines that will all perform some similar task. A task is a script that can launch a daemon server and/or run an arbitrary set of programs. Each task is associated with a set of roles. When a task is executed, it is executed on the set of machines defined by the set of roles with which it is associated. To execute a task, the name of the task is provided as the first argument to ht_cluster followed by any optional task arguments:
ht cluster <task-name> [<args> …]
NOTE: As can be seen by the above command line, the ht_cluster program should be launched with the ht wrapper script, which makes appropriate changes to the environment before launching the tool. This script is invoked as ht <tool> [args…]. The ht script will prepend ht_ to the <tool> argument to locate the executable file, which is why it is invoked as ht cluster (i.e. without an intervening underscore character).
Cluster Definition Syntax
The cluster definition file, named cluster.def by default, contains the definitions of roles and tasks. ht_cluster looks for cluster.def first in the current working directory and if not found, looks for it in the directory ../conf, relative to the directory containing the ht_cluster executable file. The location of the cluster definition file can also be supplied directly by passing an –f <filename> argument to ht_cluster.
The language of the cluster definition file is almost entirely Bash, which is one of the advantages it has over the Capistrano, whose language is a mixture of Ruby and Bash. The cluster definition file is essentially just a Bash script with a few additional constructs for defining tasks and roles. The additional constructs are described below.
Roles
A role is a named set of machines that will run some similar service or perform some similar task. For example, in the Hypertable cluster.def file there is a role called slave, which defines the set of machines that will run the RangeServer process. The following is the BNF for a role definition statement:
role: <role-name> <hostspec>
The <role-name> argument is an identifier that is the name of the role and must be a valid Bash identifier (e.g. consisting of alphanumeric and underscore characters and not beginning with a numeric character). See Host Specification Pattern for a description of the <hostspec> argument. The following is an example role definition:
role: slave test[00-99]
Role definitions may be split across multiple lines and the end of the definition is signalled by the beginning of a new role definition or an empty line. Role definitions translate into Bash variable definitions where the variable name is the name of the role prefixed with "ROLE_". For example, the above role definition can be referenced in subsequent parts of the cluster definition file with ${ROLE_slave}.
Tasks
Tasks are essentially just Bash functions that are associated with a set of roles. The following is the BNF for a task definition statement:
<task-comment-block>
task: <take-name> [ roles: <role-list> ] {
<task-body>
}
The <task-comment-block> is a Bash comment block (e.g. lines starting with the ‘#’ character. The text up to the first ‘.’ character in the comment is taken as the short description for the task and should be less than 50 characters. The text after the first ‘.’ character in the comment is taken as the extended description of the task and can be of arbitrary length. The <task-name> argument is an identifier that is the name of the task and must be a valid Bash function identifier (e.g. consisting of alphanumeric and underscore characters and not beginning with a numeric character). The <role-list> argument in the optional roles: clause is a comma-separated list of role names to which the task applies. If no roles: clause is provided then the task applies to all roles. The <task-body> is similar to a Bash function body and can contain a mix of valid bash statements, task invokations, and ssh: statements which are described below.
The ssh: statement can appear within a task definition body and directs the task to perform the statements within its body on all the machines defined by the roles associated with the task. The following is the BNF for the ssh: statement:
ssh: [<ssh-options>] {
<ssh-body>
}
The <ssh-body> is a Bash script that will get executed, in parallel, on all machines defined by the set of roles associated with the task. The optional <ssh-options> argument is a space-separated list of options that control the behavior of the ssh command. Currently, the only supported option is random-start-delay=<milliseconds>. This directs the ssh tool to randomly wait <milliseconds> milliseconds before running the script on each machine (to avoid a potential thundering herd problem). The following are examples of task definitions:
Task Example 1
# Start slave processes. Starts slave processes by starting the FS broker and
# RangeServers from the current installation. The processes are started with a
# random start delay of 5000 milliseconds.
task: start_slaves roles: slave {
ssh: random-start-delay=5000 {
${INSTALL_PREFIX}/current/bin/start-fsbroker.sh ${FS} ${CONFIG}
${INSTALL_PREFIX}/current/bin/start-rangeserver.sh ${CONFIG}
}
}
Task Example 2
# Distribute installation. This task writes the value of the variable
# \$HADOOP_DISTRO (${HADOOP_DISTRO}) into the conf/hadoop-distro file of the
# Hypertable \$HYPERTABLE_VERSION (${HYPERTABLE_VERSION}) installation and then
# runs the rsync_installation task.
task: dist roles: source {
if [ ! -f ${INSTALL_DIR}/conf/hadoop-distro ] ||
[ "`cat ${INSTALL_DIR}/conf/hadoop-distro`" != "${HADOOP_DISTRO}" ]; then
ssh: {
echo ${HADOOP_DISTRO} > ${INSTALL_DIR}/conf/hadoop-distro
}
fi
rsync_installation
}
In Task Eample 2, rsync_installation is the name of another task to be executed as part of the dist task. Tasks compile down into Bash functions, so the rsync_installation statement translates directly into a Bash function call.
Include
Another advantage of the ht_cluster tool over Capistrano is that the cluster definition statements can be split into separate files that can be updated independently. This is accomplished with the include: statement that has the following BNF:
include: <pathname>
The <pathname> argument is either a relative or absolute pathname of another cluster definition file that is to be included as part of the definition. If a relative path name is supplied, it is relative to the directory of the parent file (i.e. the file containing the include: statement). All of the definitions in the parent file or any previously included cluster definition file, up to the include: statement, are visible to the included file. This allows deployment-specific definitions (e.g. roles) to be defined in a parent file and deployment-agnostic definitions (e.g. tasks) to be defined in an included file which can be upgraded independently. Comparing this with Capistrano, to update task definitions, one had to manually construct a new Capistrano definition file (Capfile) by cutting and pasting the existing role definitions with the newly updated task definitons. The following is an example cluster.def file that illustrates the use of the include: statement.
INSTALL_PREFIX=/opt/hypertable HYPERTABLE_VERSION=0.9.8.4 PACKAGE_FILE=/root/packages/hypertable-0.9.8.4-linux-x86_64.tar.gz FS=hadoop HADOOP_DISTRO=cdh4 ORIGIN_CONFIG_FILE=/root/hypertable.cfg PROMPT_CLEAN=true role: source test00 role: master test[00-02] role: hyperspace test[00-02] role: slave test[03-99] role: thriftbroker role: spare include: "core.tasks"
The core.tasks file contains all of the task definition statements and can be updated with bug fixes and enhancements without requiring any modification to the cluster.def file.
Interactive Mode
Arbitrary shell commands or cluster definition tasks can be run remotely on any set of machines using the syntax described below. To run ht_cluster in interactive mode, run it without any arguments. Type 'quit' or 'exit' to exit the command interpreter.
$ ht cluster cluster>
Running a shell command on all roles
To run a shell command on all hosts specified in all role definitons in the cluster definition file, just enter the command at the cluster> prompt, for example:
cluster> echo "Hello, World!" [test00] Hello World! [test01] Hello World! [test02] Hello World! …
Running a shell command on a specific set of hosts
To run a shell command on a specific set of hosts, precede the command with on <hostspec> at the cluster> prompt, where <hostspec> is a host specification pattern. For example:
cluster> on test[02-04] echo "Hello, World!" [test02] Hello World! [test03] Hello World! [test04] Hello World!
Running a shell command on a specific set of roles
To run a shell command on a specific set of roles, precede the command with with <role-list> at the cluster> prompt, where <role-list> is a comma-separated list of role names. For example:
cluster> with master echo "Hello, World!" [test00] Hello World!
Running a task
To run a task from the interactive command prompt, just type the ‘!’ character followed by the task name and optional arguments. To run the task on a specific set of hosts, include an on <hostspec> clause immediately after the task name (but before the task arguments). For example:
cluster> !start_hyperspace cluster> !start_hyperspace on test02
Built-in Tasks
ht_cluster provides some built-in tasks. These tasks are described below.
show_variables
The show_variables task displays all of the global variables defined in the cluster definition file, including the implicit variables generated from the role definitions (e.g. ROLE_<role>). The following shell transcript shows an example of running the show_variables task.
$ ht cluster show_variables FS=hadoop HADOOP_DISTRO=cdh4 HYPERTABLE_VERSION=0.9.8.4 INSTALL_DIR=/opt/hypertable/0.9.8.4 INSTALL_PREFIX=/opt/hypertable ORIGIN_CONFIG_FILE=/root/hypertable.cfg PACKAGE_FILE=/root/packages/hypertable-0.9.8.4-linux-x86_64.tar.gz PROMPT_CLEAN=true ROLE_hyperspace=test[00-02] ROLE_master=test[00-02] ROLE_slave=test[03-99] - test37 ROLE_source=test00 ROLE_spare= ROLE_thriftbroker= RSYNC=rsync -av -e 'ssh -o StrictHostKeyChecking=no' RS_DUMP_FILE=/tmp/rsdump.txt THRIFTBROKER_ARGS=
with
The with task provides a way to run a shell command on the hosts of a specific role or set of roles. The following shell transcript shows an example of running the with task to run the hostname command on the master and slave roles.
$ ht cluster with master,slave hostname [test00] test00.admin.hypertable.com [test01] test01.admin.hypertable.com [test02] test02.admin.hypertable.com [test03] test03.admin.hypertable.com [test04] test04.admin.hypertable.com …
Task Help
The ht_cluster tool accepts a –T (or -–tasks) argument that will display all of the known task names along with their brief description. The following shell transcript shows an example of running ht_cluster with the –T argument.
$ ht cluster –T TASK DESCRIPTION ============================= ================================================= destroy ..................... Destroy database removing all tables destroy_hyperspace .......... Destroy Hyperspace state destroy_masters ............. Destroy master state destroy_slaves .............. Destroy slave state dist ........................ Distribute installation fhsize ...................... FHS-ize the installation install_origin_config ....... Install origin configuration file install_package ............. Install package on all machines kill ........................ Kill all Hypertable processes push_config ................. Push config file out to all machines rangeserver_dump ............ Dump RangeServer statistics restart_ganglia ............. Restart Ganglia rsync_config_dir ............ Rsync config dir from source machine to all rsync_installation .......... Rsync installation dir from source machine to all run_test .................... Run tests run_test_dispatcher ......... Start test dispatcher set_current ................. Set the symbolic link 'current' to point to $HYPER set_hadoop_distro ........... Set Hadoop distro start ....................... Start all Hypertable processes start_database .............. Start primary Hypertable processes start_ganglia ............... Start Ganglia start_gmetad ................ Start Ganglia gmetad start_gmond ................. Start Ganglia gmond start_hyperspace ............ Start hyperspace start_masters ............... Start master processes start_monitoring ............ Start monitoring server start_slaves ................ Start slave processes start_test_clients .......... Start test clients start_thriftbrokers ......... Start ThriftBrokers start_thriftbrokers_only .... Start ThriftBrokers on ThriftBroker-only servers start_thriftbrokers_primary . Start ThriftBrokers on primary servers stop ........................ Stop all Hypertable processes stop_database ............... Stop primary database processes stop_fsbrokers .............. Stop FS brokers stop_ganglia ................ Stop Ganglia stop_gmetad ................. Stop Ganglia gmetad stop_gmond .................. Stop Ganglia gmond stop_hyperspace ............. Stop Hyperspace stop_masters ................ Stop masters stop_monitoring ............. Stop monitoring servers stop_slaves ................. Stop slaves stop_test ................... Stop tests stop_thriftbrokers .......... Stop ThriftBrokers stop_thriftbrokers_only ..... Stop ThriftBroker on ThriftBroker-only servers stop_thriftbrokers_primary .. Stop ThriftBroker on primary servers upgrade ..................... Verify and upgrade installation upgrade_installation ........ Upgrade installation verify_upgrade .............. Verify upgrade
To get the long description of a task the –e <taskname> argument may be supplied, for example:
$ ht cluster –e dist dist ==== Distribute installation. This task writes the value of the variable $HADOOP_DISTRO (cdh4) into the conf/hadoop-distro file of the Hypertable $HYPERTABLE_VERSION (0.9.8.4) installation and then runs the rsync_installation task. -- ROLES: source
Internals
The ht_cluster tools is driven off a cluster configuration file that is mostly a bash script with a few additional constructs for defining tasks and roles and for including other configuration files. When ht_cluster runs for the first time (and each time the configuration file changes), it will compile the configuration file into a bash script and will store it in a file with the following path name:
~/.cluster + <absolute-path-to-cluster-def-file> + .sh
To get an idea of what the compiled configuration looks like, take a look at the above file, or run ht_cluster with the –-display-script argument, which will cause the script to be written to the terminal. The generated script is fairly voluminous, so we won’t reproduce it in its entirety here. Some of the important sections are covered below.
Header
The script starts out with a comment header such as the one shown below.
#!/bin/bash # # version: 0.9.8.4 (v0.9.8.4-0-gdebbc65-dirty) # dependency: /opt/hypertable/current/conf/core.tasks
The version: and dependency: comments are used by ht_cluster to determine if the configuration needs to be re-compiled. If the version of ht_cluster being run does not match the version: comment, or if the modification timestamp of any of the files listed in the dependency: comments is newer than the modification timestamp of the compiled script, it will recompile the configuration file.
Variable definitions
All global variable declarations are translated to allow them to be overridden by the environment. For example:
INSTALL_PREFIX=${INSTALL_PREFIX:-/opt/hypertable}
HYPERTABLE_VERSION=${HYPERTABLE_VERSION:-0.9.8.4}
PACKAGE_FILE=${PACKAGE_FILE:-/root/packages/hypertable-0.9.8.4-linux-x86_64.tar.gz}
FS=${FS:-hadoop}
HADOOP_DISTRO=${HADOOP_DISTRO:-cdh4}
ORIGIN_CONFIG_FILE=${ORIGIN_CONFIG_FILE:-/root/hypertable.cfg}
PROMPT_CLEAN=${PROMPT_CLEAN:-true}
Role definitions
Role definition statements are translated into simple variable definitions, where the variable name is the name of the role prefixed with ROLE_. For example:
ROLE_source="test00" ROLE_master="test[00-02]" ROLE_hyperspace="test[00-02]" ROLE_slave="test[03-99] - test37" ROLE_thriftbroker="" ROLE_spare=""
Task definition
Tasks compile directly into Bash functions. For example, the dist task shown in Task Example 2 of the Cluster Definition Syntax -> Tasks section compiles into a Bash function that looks something like the following:
dist () {
local _SSH_HOSTS="(${ROLE_source})"
if [ $# -gt 0 ] && [ $1 == "on" ]; then
shift
if [ $# -eq 0 ]; then
echo "Missing host specification in 'on' argument"
exit 1
else
_SSH_HOSTS="$1"
shift
fi
fi
echo "dist $@"
if [ ! -f ${INSTALL_DIR}/conf/hadoop-distro ] ||
[ "`cat ${INSTALL_DIR}/conf/hadoop-distro`" != "${HADOOP_DISTRO}" ]; then
/opt/hypertable/current/bin/ht ssh " ${_SSH_HOSTS}" "echo ${HADOOP_DISTRO} > ${INSTALL_DIR}/conf/hadoop-distro"
if [ $? -ne 0 ]; then
exit 1
fi
fi
rsync_installation
}
Notice how the ssh: statement compiles into an invocation of the ht_ssh tool and uses the _SSH_HOSTS local variable as the host specification. The _SSH_HOSTS variable is initialized to the role variables derived from the roles: clause of the task: statement but can be overridden by passing an on <hostspec> argument to the task. Also notice how the invocation of the rsync_installation task is just a call to the corresponding Bash function.
Command-line Argument Parsing
The compiled Bash script also includes logic to parse command line arguments for task names and invokes the corresponding task Bash function. For example:
if [ $1 == "clean_hyperspace" ]; then shift clean_hyperspace $@ elif [ $1 == "clean_masters" ]; then shift clean_masters $@ elif [ $1 == "clean_slaves" ]; then shift clean_slaves $@ elif [ $1 == "cleandb" ]; then shift cleandb $@ …
This is how ht_cluster runs tasks. It simply launches the Bash script, passing the task name along with its arguments as command line parameters to the script.
