New Cluster Administration Tool - ht_cluster

12.01.2014  |  Usability

As of the 0.9.8.4 release we’ve introduced a new cluster administration tool called ht_cluster.  The tool provides a simple way to perform common administrative tasks such as starting and stopping Hypertable and upgrading to a new version.  This tool replaces Capistrano, which was the recommended cluster administration tool that served us well for many years.  ht_cluster is based on the same role and task concepts as Capistrano, where roles are named sets of hosts (e.g. master, slave) and tasks are shell scripts associated with roles that run in parallel on all hosts in the associated roles.  While conceptually very similar, ht_cluster improves upon Capistrano in a number of ways, most notably usability.  The following list describes the ways in which ht_cluster improves upon Capistrano.

• No Dependencies – No dependency on Ruby or any Ruby gems.  All of the dependencies are included in the Hypertable package, so ht_cluster works “right out of the box”.
• Single Language – The cluster configuration file is written entirely in Bash, whereas with Capistrano the language is a mixture of Ruby and Bash, making it difficult to mentally jump back-and-forth between languages.
• Maximum Performance – ht_cluster is implemented with libssh 0.6.3 and connection establishment and command execution happens asynchronously and in parallel, which provides for maximum parallelism and performance.
• Host Specification Pattern – Hosts can be described using a concise and flexible host specification pattern.  Several hundred hosts can be specified with a pattern such as test[000-299].  Hostnames can also be subtracted from a pattern which allows dead hosts to be easily removed from the configuration, for example:  test[000-299] - test137.  Capistrano, on the other hand, requires each host name to be listed individually.
• Stable Output – ht_cluster buffers command output and writes the complete output for each host in order of host name.  If the command output is stable, then the output of ht_cluster is stable, making it useful as a testing tool.  Capistrano, on the other hand, interleaves the output of the commands as then run in parallel on each host, making the output unstable.
• Upgradability – The deployment-specific cluster configuration (roles) and the common cluster configuration (tasks) are stored in separate files.  This allows the tasks that are common to all deployments to be upgraded independently and automatically.  With Capistrano, upgrading common task definitions requires manual cutting and pasting of the old configuration file, along with the task definitions from the new one, in order to create an updated configuration file.
• Random Start Delay – ht_cluster provides a random-start-delay option to allow tasks to be started on each host after a random delay.  This feature can be used to avoid thundering herd problems that can occur when a service is started on a large number of hosts simultaneously.
• Command History – ht_cluster uses the history feature of libedit, the command-line editing library that it uses to read interactive commands.  Through the use of this feature, a historical record is kept of recent commands that can be navigated with the arrow keys (or Emacs-style navigation keys).  The command history is stored in the file ~/.cluster_history, making the command history available across invocations of the tool.
• Task Arguments – Task definitions in ht_cluster can accept arguments.  Since tasks compile down into Bash functions, it’s easy to write tasks that accept and manipulate any number of arguments.
• Run Task on Any Host – By default, a task will run on the hosts defined by the roles with which it is associated.  However, ht_cluster also allows tasks to be run on an arbitrary set of hosts by providing an on <hostspec> argument to the task.  For example, this allows new machines to be added to a cluster and bootstrapped with a single command:
  
  ht cluster start_slaves on test37,test41

For details on how to configure ht_cluster and use it to administer a Hypertable cluster, see Hadoop Installation.  For comprehensive documentation of the ht_cluster tool see ht_cluster.

Posted By:  Doug Judd