Roadmap to Hypertable 1.0
07.23.2012 | Release Status
With the release of Hypertable version 0.9.6.0 I thought I would take some time to describe where we are in terms of the Hypertable 1.0 release and what work is remaining. We had intended to make the next Hypertable release our beta release. However, it’s been four months since the release of 0.9.5.6 and since the beta release is not quite ready to go, we decided to do one last alpha release and call it 0.9.6.0. In this release we’ve put in a considerable effort to fix a number of stability issues that have affected prior releases.
0.9.6.0 Stability Improvements for HDFS deployments
The biggest source of instability for Hypertable deployments running on top of HDFS has do with the unclean shutdown of either the Master or RangeServer. Upon restart after this situation has ocurred, the RangeServer (or Master) can fail to come up with an error message similar to the following in its log file:
1342810317 ERROR Hypertable.RangeServer : verify_backup (/root/src/hypertable/src/cc/Hypertable/Lib/MetaLogReader.cc:131): MetaLog file '/hypertable/servers/rs12/log/rsml/0' has length 0 < backup file '/opt/hypertable/0.9.5.6 /run/log_backup/rsml/rs12/0' length 11376
This problem was due to a misunderstanding on our part of the HDFS API semantics. Whenever the Master or RangeServer writes data to any of its log files, it makes a call to
FSDataOutputStream.sync() to ensure that the data makes it in to the filesystem and is persistent. However, after making this call, a call to the
FileStatus.getLen() does not return the correct value.
FileStatus.getLen() only returns the correct file length if the file was properly closed. HDFS provides an alternate API,
DFSClient.DFSDataInputStream.getVisibleLength(), that returns the actual length of the file regardless of whether or not it was closed properly. We’ve since modified the HDFSBroker length method to use this alternate API which has solved the problem with negligible performance impact.
There have been a number of other important, though more rare, issues that have been fixed in the 0.9.6.0 release. A brief description of each of these fixes can be found in the Hypertable 0.9.6.0 Release Notes.
0.9.7.0 “beta” Release – Automatic RangeServer Failover
The main feature that has been holding up our beta release is automatic RangeServer failover. Currently, if a machine running a RangeServer dies, the portion of the data set managed by that machine will be unavailable until an operator intervenes and does one of two things:
- Bring the machine back online
- Replace the machine with a new one and assign it the same IP address and/or hostname
The 0.9.7.0 “beta” release will include automatic RangeServer failover in which the Master will detect when a RangeServer dies and will orchestrate the recovery of that RangeServer. Recovery will involve re-assigning the ranges managed by the failed RangeServer to other servers and enlisting the help of other RangeServers in the system to replay the failed server’s commit log in parallel.
We have been laying the groundwork for automatic RangeServer failover for five years and have been actively working on it full-time for the past nine months. The difficult part of this work is not so much the basic failover logic, but handling all of the failure scenarios. For example, after a RangeServer fails and recovery is initiated, the system needs to properly handle the case where one or more of the recovery participants fail. We’ve put a tremendous effort into testing for and handling all of the various failure scenarios and have over seventy tests to verify correctness and prevent regressions.
We plan to release Hypertable 0.9.7.0 with automatic RangeServer failover next month.
On to Hypertable 1.0 …
Once the beta release is out, we will have an approximately two month beta period in which we will resolve any remaining issues and then release version 1.0. Thanks to everyone who has helped us get to this point and you can rest assured that we're working very hard to make 1.0 available as soon as possible.
Posted By: Doug Judd, CEO, Hypertable Inc.
Here's what other people had to say
Excellent blog here! Also your web site loads up very fast! What host are you using? Can I get your affiliate link to your host? I wish my website loaded up as fast as yours lol
I use t-gel which helps control them and the conditioner smothers them where they are slower so easier to get them. Took weeks but best way I found for self checking
and doing yourself.
Thank you for using my Guide and if it work for you that makes me happy
Thank you for your information.
Hi there, awesome site. I thought the topics you posted on were very interesting. I tried to add your RSS to my feed reader and it a few. take a look at it, hopefully
I can add you and follow.
Sharing my very first blog post … so glad to have finally entered the world of the blogger! Hope you enjoy it ; )
about CPI not being sole criterion of suscces so I will not comment on that again. I will bring another view to it. The real cost of a IIT education or anything in life is Opportunity cost. As 12th class student, we all are unable to understand it, but it becomes more predominant in later years of life. I support IITs for implementing the 1 year drop criterion and it comes from my personal experiences. My story: Didn’t appear for any engineering exam in 12th passing year. Deemed Engineering too expensive and no one in my circle (seniors/teachers) had cleared JEE. In 2000, appeared for JEE screening without much preparation. Started preparation and same year cleared Bihar Engineering but didn’t get admission, because I want to go for JEE. Cleared JEE in 2001, but got my degree with 1 yr extension. Doing well now in Job. The Cost : 2 yr lost, in which I could have easily done M.Tech from IIT.My Best Friend, Saurav : Cleared Bihar Engineering in 2000 and went to NIT Trichy. Passed out in 2004 2 yrs before me with good grades. Went to IIM later.Saurav’s Elder Brother Gaurav: Brilliant chap. Was 3rd topper of his college with 80 plus % marks in 12th in Bihar board. Couldn’t clear JEE in 2 attempts so went to HBTI Kanpur. Kept preparing for JEE in HBTI also and skipped his 1st year exam for JEE. Couldn’t clear again. He was a topper throughout his B.Tech in HBTI but because of his mistake graduated in 5 yrs. (The longing for JEE).My Another Batchmate, Shailesh : The most brilliant of our batch. Cleared JEE with 1 yr drop but about 1500 rank, took admission to IIT Kgp, and prepared for JEE again this time went to IITD with only slightly better rank. His CPI in Kgp also suffered. Lost a year because of preparation.My Another Coaching Mate, Gopal : cleared JEE with me at rank 1300, went to IIT Kgp, Dropped out and went to FITJEE Delhi, prepared again and got rank of 85. Came in IITK CSE as my junior. Lost 1 yr because of JEE. My Another Friend, Deepak : Cleared Bihar Engineering in 2001, couldn’t clear JEE,so went to BIT Sindri in Civil, did M.Tech. from IIT Roorkee in 2007 and now employed with Airport Authority of IndiaMy Younger Brother : Cleared Bihar Engineering after 1 yr drop. But I suggested him drop it and keep on preparing for JEE 2006. IIT implemented 1 yr drop rule. That plus some other issues and he got depressed and did really bad in AIEEE also. He vowed not to study and I had a hard time persuading him. Finally got admitted him to a Tier 3 college in BCA after 3 yr drop. Now he is pursuing MCA from NIT Trichy. He studies is delayed by 4 yrs because of all wrong choices. In reality, I have seen more examples of my younger brother, Gaurav and Shailesh. In south I see more examples of Saurav and Deepak. The real cost of IIT is opportunity cost. The time lost in preparation which is irrecoverable. I sometimes think, was going to IIT the best decision I made. No doubt it taught me a lot and I have the IIT Brand. But the cost on family resources was really worth IIT at that time. I was happy that IIT implemented the 1 yr drop rule. There was a time, when IITs were only way to get a sure shot job but that has changed now. My brother’s classmate opted for a BCA from St Xaviers Ranchi and she got hired by Wipro in her final yr from campus. She is doing M.Tech from BITS Pilani now in 4 yrs. (while on job). The new JEE may miss some of the anecdotal examples and some diversity, but JEE is all about selecting the best based on statistics and statistics can’t go on chasing the LONG TAIL of Data. http://kbnexot.com woknakthmh [link=http://rpcbccbnpio.com]rpcbccbnpio[/link]
81cVery nice and touchy acritle and comments.IIT and JEE are synonymous only seemingly. The two are entirely different things. Lets analyse:IIT-JEE:Firstly What is IIT it is just a college similar to myriad other colleges in the country. Whats the difference? There are facilities even better than IITs in many other colleges. Can anyone list facilities what IITs provide which are not available elsewhere and that maximum student at IITs use?? How many IITians are there who have REALLY USED those so called facilities (which I personally dont even know) in the IITs and students in other colleges have not and hence less privileged??? So just IIT is nothing great in itself.JEE: what makes IIT a brand what people see is due to the JEE. IITs are about their flagship product which is there students through JEE. It is students who make IIT the place it is. Remove JEE and IIT will not be IIT anymore.I can challenge Govt with some experiment: Let the JEE be there without IIT. Let the students qualified by JEE not go to IITs but let them go to some underprivileged college in some remote area with minimum facility required for an engineering college (only JEE qualified students should be admitted there) and compare then the IITian produced and JEEian produced.You will SEE.It is really important what product you want from a system. System should be designed according to the desired output not the input .!! Inputs are to be screened according to the System requirement.Let there be lakh of seats at IIT if you can and if that satisfies you but why do YOU want JEE to be replaced??If you want a TON more of cereal to be strained you DONT make the hole of strainer bigger but you need to make more strainers of same sort OR strain for more time. How can you meddle with Straining Mechanism if you want same quality of strained cereal?? As simple as that. http://hvzapiwbw.com kxjyae [link=http://qyzusy.com]qyzusy[/link]
[Disclosure: I used to work at Amazon, but not on Dynamo or Cart, and have no non-public information about those smtyess.]I think this is based on a flawed application of metrics from other organizations that do not match Amazon’s actual business needs.Sure, you can build a centralized system with a critical core that has near-zero downtime usually at great operational cost but there’s not a bank in the world that’s actually 99.999% available to customers over the internet. Most retail banks I’ve used take several hours of outages *per month* for maintenance on their web services.Amazon wants its smtyess not just to keep running but to keep taking orders from millions of people all over the world. This means that they are concerned with the reliability not just of a storage server, but everything needed to connect it to its application servers and to customers. They have found that the cost-effective way to do this is to distribute every component across geographically separated datacenters. Amazon has remained available through real and simulated datacenter-wide outages ranging from power/cooling failures to floods, fire, and hurricanes. No Amazon system lives within a single building, much less a single network switch or rack. The availability numbers you see externally for Amazon services are the combined downtime of every component in a large, networked system, and they also reflect tradeoffs made under a different cost-benefit model than a financial institution.Finally, although the formally provable aspects of Dynamo’s eventual consistency guarantee may be vague, any team operating a distributed system will study and understand the actual operational characteristics in practice, under both normal conditions and various failure modes. Some smtyess may have realistic failures that lead to days or hours of inconsistency (in which case the team has deliberately chosen this and will write client software to be aware of it); others might be tuned to achieve consistency within milliseconds under normal operation and to set off alarms within seconds after a failure. I’ve never known any that would be used in circumstances where human lifetimes are a relevant time period.
c0bThe fact an entrance exam is being so ovlreaeuvd that people are letting a rank in an exam or the exam itself define them or the IITs should be a clear indicator that something needs to change here if one is to shape the IITs as centers of academic excellence. Mr. Sibal thinks by rooting out the tumor itself can help make a necessary change,perhaps he does not realize that it is malignant.Since the US is being so referenced here in this blog no University admissions process here overvalues a Math/Science/robotics olympiad(leave alone think of them as indicators of innovation ) or a perfect SAT score or a >4.0 GPA or the number of Ap/College courses you finished or how early you finished them. If your resume indicates you achieved all that and more by overlooking school acitivities that is a sound reason to reject that resume. So one has to be well rounded and has to be able to do justice to whatever opportunities(most of all school) that came in one’s way. This is why though many come from very small farming town schools where there are not as many opportunities as say a magnet school in a city they still stand a fair chance of making it to the best Universities if they are able to demonstrate how they excelled in using whatever opportunities taht came their way. It is a different matter that many from those kind of schools may not want to seek higher education but that again is a choice they make.If there is anything to be learned from the US model it is the higher ethical standards when pursuing or defining academic excellence. People do not usually adopt shortcuts here like coaching tutoring to get better grades because those are meant for kids who are not able to keep up so are taken as indicators that a kid needs help just to keep up.As for repeatedly citing Outliers as evidence hardly anyone in good academic standing gives that book so much importance. People know those kind of books (which examine factors of success/failure superficially ) are meant for nothing more, than to generate talkshow material.( Tiger mom is another example).Everyone knows the value and importance of hardwork/discipline but there are many more important factors like maintaining high ethical standards in not cutting corners or looking for shortcuts and bypasses and respecting the choice of the individual in determining how to shape his life. It is not every genius who thinks that getting published or cited repeatedly or getting recognition/money is an indicator of success. Many would rather let their work be accepted by the world when it is ready for accepting it but have full faith in the success of their pursuit of knowledge. Does not mean they are failures if they choose to stay out of the limelight and refuse to run in a rat race.
We are using 0.9.6.5 version of HT and waiting for 0.9.7.0 “beta” Release – Automatic RangeServer Failover. I hope everything will be good with new version.