Comments for Blog of Data

Comment on Migrating HBase: In the Trenches by Tom Goren

Tom Goren — Sat, 01 Oct 2011 18:32:33 +0000

You guys did an awesome job with this. You are welcome to check out my solution as well on my blog. I was a little hesitant to copy hbase data straight from the hdfs due to the same data consistency worries you stated as well when planning. Instead I went a little roundabout, and while I'm sure your solution out performs mine by far, my approach seems to require a little less manual intervention. Also you can divide the table into time stamp based chunks fairly easily, and batch the process. Anyhow thanks a lot! just my humble contribution, hope it helps somebody (we had to migrate the exact same route 0.20.x to CDH3).

Comment on Flume, Hive and realtime indexing via ElasticSearch by Understanding DNT Adoption within Firefox « Blog of Metrics

Understanding DNT Adoption within Firefox « Blog of Metrics — Thu, 08 Sep 2011 16:56:07 +0000

[...] guy lives and dies by the data. In late 2010, the metrics team gave a small talk on how we collect log data (click here for the video ppt). While that project has gone multiple iterations over time, the [...]

Comment on How glow.mozilla.org gets its data by HBase 的应用案例 | 我要去桂林_田春峰的互联网生活

HBase 的应用案例 | 我要去桂林_田春峰的互联网生活 — Sat, 02 Jul 2011 12:10:41 +0000

[...] The first version of Glow used several technologies including SQLStream and HBase to process download request logs in real time and make them available for the site to display. If you are interested in learning the technical details behind Glow read this article from the Mozilla Metrics team, which has links to the code repositories: http://blog.mozilla.org/data/2011/03/22/how-glow-mozilla-org-gets-its-data/ [...]

Comment on Migrating HBase: In the Trenches by » HBase Backup Options HBase.info -- All things about HBase

» HBase Backup Options HBase.info -- All things about HBase — Thu, 23 Jun 2011 12:10:17 +0000

[...] 由于Dictcp做集群复制存在数据不一致的问题，Mozilla的开发人员开发了一个Backup工具，具体情况请参考他们的这篇Migrating HBase in the Trenches。 [...]

Comment on Migrating HBase: In the Trenches by Xavier

Xavier — Fri, 27 May 2011 15:44:22 +0000

Hi Matthias,

I updated the link. I moved the repo to our mozilla-metrics github organization a couple of weeks ago. You don’t actually have to fix anything in .META. HBase will figure that out on its own. But you do need to copy it. As I alluded to in the post, to minimize your downtime you can use Backup to make a “dirty” non-functioning copy of the data first. Then during your downtime you’ll only need to copy the files that have changed.

Feel free to join our IRC channel irc.mozilla.org #metrics if you need any further clarification.

Cheers,

Xavier

Comment on Migrating HBase: In the Trenches by Matthias

Matthias — Fri, 27 May 2011 15:23:52 +0000

Hi Xavier,
we are also trying to copy hbase data over to new cluster. 0.20.4 to cdh3u0(0.90.1). Since we dont have that much data, we could stop hbase for a short while to perform distcp. However I am not sure how to fix meta information referencing old regionservers once data moves to new cluster. Did you come accross that problem when trying distcp?

Thanks for your help,
Matthias

ps. the link to the backup utility does not work any more. Is this tool still available?

Comment on Riak and Cassandra and HBase, oh my! by Cassandra/Riak/Dynamo Optimistic Concurrency Control

Cassandra/Riak/Dynamo Optimistic Concurrency Control — Fri, 06 May 2011 02:36:10 +0000

[...] along with installed user base, the Dynamo clones seem to have something else. They are truly web scale. I say this because, unlike virtually all other NoSQL implementations, the Dynamo-based [...]

Comment on Flume, Hive and realtime indexing via ElasticSearch by Rich Kroll

Rich Kroll — Tue, 12 Apr 2011 02:26:26 +0000

@deinspanjer I’ve done some experimenting and thought you may be interested in some of the findings. I am interested in using ElasticSearch in a similar way as you outlined in this post but really wanted to have ES live in it’s own decorator and leverage a fan out sink to write to both ES and Hive. This would allow for additional decorators over ES or have (think batching).

What I ended up doing was to add a UUID/GUID to the flume event when it is created. This is later used as the ID in ES when indexing the log event and allows for an event to be idempotent. The drawback to this design is that each event failure causes an additional write to ES. As there is no queue in front of ES, this could be a problem in the event of a large bulk of failed messages, but could be mitigated with other decorators.

Best,
Rich

Comment on How glow.mozilla.org gets its data by Links: Googles Python Anfängerkurs, Firefox 4 Visualisierungen, Data Science Toolkit, RSS Tuning, McLuhan – tobiaskut.de - Open Source | Content Management | Redaktion

Fri, 08 Apr 2011 07:01:58 +0000

[...] Firefox hat zum Launch von Firefox 4 eine sehr beeindruckende Visualisierung ins Netz gestellt. Unter http://glow.mozilla.org lassen sich die Downloads in Echtzeit verfolgen. Jeder Punkt steht für einen Download, am unteren Bildschirmrand zeigt ein Balkendiagramm die Downloads im Minutentakt und wer es ganz genau wissen will hat über das Kreisdiagramm unten links die Möglichkeit die Daten von Kontinenten auf Stadtebene runterzubrechen. Das ist Datenvisualisierung! Details zur technischen Umsetzung stehen im übrigen hier: How glow.mozilla.org gets its data [...]

Comment on How glow.mozilla.org gets its data by HBase支持的 Firefox4 下载统计页 | laura's site

HBase支持的 Firefox4 下载统计页 | laura's site — Wed, 06 Apr 2011 03:32:46 +0000

[...] 　　英文原文：How glow.mozilla.org gets its data [...]