htck

The htck program is used to check the integrity (metadata consistency) of a Hypertable database.  If an integrity problem is detected, the tool can also be used to repair it.

Preparing to Run

You must perform the following steps prior to running htck:

  1. Stop all client activity (esp. insert activity).
     
  2. Run ht wait-until-quiesced to wait for Hypertable to get into an idle state.  For example:
    $ ht wait-until-quiesced
    16d27cbad8dd528dd6745fef491dfb4f metadata.tsv
    16d27cbad8dd528dd6745fef491dfb4f metadata.tsv
    

    As a side-effect, the ht wait-until-quiesced command will generate a metadata.tsv file in the current working directory that contains a dump of the sys/METADATA table after the system enters the idle state.  This file will be used as input to the htck program.

  3. Stop Hypertable

    $ ht cluster stop

Once Hypertable has stopped completely, the next step is to restart Hyperspace and start an FS broker on the machine from which you plan to run htck.  For example:

$ ht cluster start_hyperspace
$ ht start-fsbroker hadoop  # or local, ...

Checking Integrity

To check for integrity problems, run htck as follows, using the metadata.tsv file that was generated by the ht wait-until-quiesced command.

$ ht htck check metadata.tsv

If htck detects no integrity problems, it will generate output that looks something like the following (entry counts may differ):

METADATA entries: 100977
RSML range entities: 100977
RSML dangling entities: 0
METADATA overlaps:  0
METADATA gaps:  0
METADATA incomplete:  0
RSML dangling fix-ups: 0
METADATA shrunk ranges:  0
Unassigned locations: 0
Unassigned start row: 0
Unassigned entities: 0
Load acknowledge fix-ups: 0

If htck does detect an integrity problem, it will produce output that describes the problems that it encountered.  For example:

METADATA entries: 100976
RSML range entities: 100977
RSML dangling entities: 1
Error: GAP in metadata range found
  last end row:  2/81:ur7H6vYc7tekOLBCte+aPsGA==_388659_2097371132442
  next start row: 2/81:ur7HdsMGe7Z/wApbmVezq28g_502360009_2604183617513
METADATA overlaps:  0
METADATA gaps:  1
METADATA incomplete:  0
RSML dangling fix-ups: 0
METADATA shrunk ranges:  0
RSML dangling entries:
  2/81[ur7H6vYc7tekOLBCte+aPsGA==_388659_2097371132442..ur7HdsMGe7Z/wApbmVezq28g_502360009_2604183617513] at rs211
Unassigned locations: 0
Unassigned start row: 0
Unassigned entities: 0
Load acknowledge fix-ups: 0

Repairing Integrity

To repair integrity problems that were discovered with htck check, you can run htck again, but replace the word check with repair.  For example:

$ ht htck repair metadata.tsv

The above command will generate a file metadata.repair.tsv in the current working directory.  This file will need to be loaded via LOAD DATA INFILE as soon as the system is brought back up.  For example:

$ ht cluster start

After a few seconds, the system should have loaded the sys/METADATA ranges, at which point you can load the metadata.repair.tsv file (e.g. in another terminal window):

$ echo "USE sys; LOAD DATA INFILE 'metadata.repair.tsv' INTO TABLE METADATA;" | ht shell --batch

At this point, once the system comes up fully, it should be in a consistent state.  To double-check, rerun the ht wait-until-quiesced command followed by htck check and verify that no integrity problems are detected.