Comments on: Migrating HBase: In the Trenches http://blog.mozilla.org/data/2011/02/04/migrating-hbase-in-the-trenches/ Mozilla metrics team technical articles Sat, 01 Oct 2011 18:32:33 +0000 hourly 1 http://wordpress.org/?v=3.5.1 By: Tom Goren http://blog.mozilla.org/data/2011/02/04/migrating-hbase-in-the-trenches/comment-page-1/#comment-3589 Tom Goren Sat, 01 Oct 2011 18:32:33 +0000 http://blog.mozilla.org/data/?p=348#comment-3589 You guys did an awesome job with this.
You are welcome to check out my solution as well on my blog.
I was a little hesitant to copy hbase data straight from the hdfs due to the same data consistency worries you stated as well when planning.
Instead I went a little roundabout, and while I’m sure your solution out performs mine by far, my approach seems to require a little less manual intervention.
Also you can divide the table into time stamp based chunks fairly easily, and batch the process.

Anyhow thanks a lot! just my humble contribution, hope it helps somebody (we had to migrate the exact same route 0.20.x to CDH3).

]]> By: » HBase Backup Options HBase.info -- All things about HBase http://blog.mozilla.org/data/2011/02/04/migrating-hbase-in-the-trenches/comment-page-1/#comment-2970 » HBase Backup Options HBase.info -- All things about HBase Thu, 23 Jun 2011 12:10:17 +0000 http://blog.mozilla.org/data/?p=348#comment-2970 [...] 由于Dictcp做集群复制存在数据不一致的问题，Mozilla的开发人员开发了一个Backup工具，具体情况请参考他们的这篇Migrating HBase in the Trenches。 [...]

]]> By: Xavier http://blog.mozilla.org/data/2011/02/04/migrating-hbase-in-the-trenches/comment-page-1/#comment-2925 Xavier Fri, 27 May 2011 15:44:22 +0000 http://blog.mozilla.org/data/?p=348#comment-2925 Hi Matthias,

I updated the link. I moved the repo to our mozilla-metrics github organization a couple of weeks ago. You don’t actually have to fix anything in .META. HBase will figure that out on its own. But you do need to copy it. As I alluded to in the post, to minimize your downtime you can use Backup to make a “dirty” non-functioning copy of the data first. Then during your downtime you’ll only need to copy the files that have changed.

Feel free to join our IRC channel irc.mozilla.org #metrics if you need any further clarification.

Cheers,

Xavier

]]> By: Matthias http://blog.mozilla.org/data/2011/02/04/migrating-hbase-in-the-trenches/comment-page-1/#comment-2924 Matthias Fri, 27 May 2011 15:23:52 +0000 http://blog.mozilla.org/data/?p=348#comment-2924 Hi Xavier,
we are also trying to copy hbase data over to new cluster. 0.20.4 to cdh3u0(0.90.1). Since we dont have that much data, we could stop hbase for a short while to perform distcp. However I am not sure how to fix meta information referencing old regionservers once data moves to new cluster. Did you come accross that problem when trying distcp?

Thanks for your help,
Matthias

ps. the link to the backup utility does not work any more. Is this tool still available?

]]> By: HBase Backup Options « Sematext Blog http://blog.mozilla.org/data/2011/02/04/migrating-hbase-in-the-trenches/comment-page-1/#comment-2663 HBase Backup Options « Sematext Blog Fri, 11 Mar 2011 16:57:10 +0000 http://blog.mozilla.org/data/?p=348#comment-2663 [...] came up with their own Backup tool.  They’ve described the tool and its use in the popular Migrating HBase in the Trenches [...]

]]> By: Links 7/2/2011: FOSDEM 2011 Closing, GNOME 3 Test Day | Techrights http://blog.mozilla.org/data/2011/02/04/migrating-hbase-in-the-trenches/comment-page-1/#comment-2573 Links 7/2/2011: FOSDEM 2011 Closing, GNOME 3 Test Day | Techrights Mon, 07 Feb 2011 13:48:33 +0000 http://blog.mozilla.org/data/?p=348#comment-2573 [...] Migrating HBase: In the Trenches We recently had a situation where we needed to copy a lot of HBase data while migrating from our old datacenter to our new one. The old cluster was running Cloudera’s CDH2 with HBase 0.20.6 and the new one is running CDH3b3. Usually I would use Hadoop’s distcp utility for such a job. As it turned out we were unable to use distcp while HBase was still running on the source cluster. Part of the reason for this is that the HFTP will throw XML errors due to HBase modifying files (particularly the case if HBase removes a directory). And to transfer our entire dataset at the time was going to take well over a day. This presented a serious problem because we couldn’t accept that kind of downtime. We were also about 75% full in the source cluster so doing HBase export was out as well. Thus I created a utility called Backup. [...]

]]> By: Tweets that mention Blog of Data » Blog Archive » Migrating HBase: In the Trenches -- Topsy.com http://blog.mozilla.org/data/2011/02/04/migrating-hbase-in-the-trenches/comment-page-1/#comment-2565 Tweets that mention Blog of Data » Blog Archive » Migrating HBase: In the Trenches -- Topsy.com Sat, 05 Feb 2011 05:35:30 +0000 http://blog.mozilla.org/data/?p=348#comment-2565 [...] This post was mentioned on Twitter by Mozilla News, Planet Mozilla and anurag, Xavier Stevens. Xavier Stevens said: My blog post about our recent HBase migration along with some code to do backup sync's – http://t.co/5p3Libq [...]

]]>