Comments on: Flume, Hive and realtime indexing via ElasticSearch

By: Understanding DNT Adoption within Firefox « Blog of Metrics

Understanding DNT Adoption within Firefox « Blog of Metrics — Thu, 08 Sep 2011 16:56:07 +0000

[...] guy lives and dies by the data. In late 2010, the metrics team gave a small talk on how we collect log data (click here for the video ppt). While that project has gone multiple iterations over time, the [...]

By: Rich Kroll

Rich Kroll — Tue, 12 Apr 2011 02:26:26 +0000

@deinspanjer I’ve done some experimenting and thought you may be interested in some of the findings. I am interested in using ElasticSearch in a similar way as you outlined in this post but really wanted to have ES live in it’s own decorator and leverage a fan out sink to write to both ES and Hive. This would allow for additional decorators over ES or have (think batching).

What I ended up doing was to add a UUID/GUID to the flume event when it is created. This is later used as the ID in ES when indexing the log event and allows for an event to be idempotent. The drawback to this design is that each event failure causes an additional write to ES. As there is no queue in front of ES, this could be a problem in the event of a large bulk of failed messages, but could be mitigated with other decorators.

Best,
Rich

By: Orange Factor 2: The redesign | 3.1415926535897932384626433…

Orange Factor 2: The redesign | 3.1415926535897932384626433… — Tue, 08 Mar 2011 17:29:08 +0000

[...] when you star an orange, the data is pushed into our database. This database is hosted by the metrics team and stores log files and tbpl data. Actually we pull from that directly and can calculate by test [...]

By: deinspanjer

deinspanjer — Tue, 04 Jan 2011 13:35:16 +0000

There is some documentation, but I don’t have the link handy at the moment. The A-team will be putting a front-end on this that should be accessible to anyone.
The data source is tinderbox logs, and if the UI is too specific then we can certainly look at providing access to the data in elasticsearch.

By: Axel Hecht

Axel Hecht — Tue, 04 Jan 2011 12:47:31 +0000

Is there information on what exactly you store, and how folks beyond WOO could get to that/benefit from it?

By: aphadke

aphadke — Sun, 02 Jan 2011 18:07:28 +0000

Jonathan,
Thanks for the compliment! Excited to see what Flume has to offer in 2011 :-)

-anurag

By: Jonathan Hsieh

Jonathan Hsieh — Sun, 02 Jan 2011 17:10:00 +0000

Great post guys! We’re working on making the Flume’s agents/collectors topologies more flexible to make doing what you have done easier in the future!

By: deinspanjer

deinspanjer — Sun, 02 Jan 2011 15:50:19 +0000

We have done a bit of exploration with fanout, but currently, fanout is incompatible with DFO. If the DFO gets an ack before any failures, it will not handle failures from any other sinks. Conversely, if the very last sink to report in is a failure, then it will resend to all the sinks.

I wanted to try to patch this, but we didn’t have enough time in our schedule, and Cloudera devs mentioned that there was some re-architecture work to be done in that area so I was afraid that we might be patching soon-to-be dead code.

By: Rich Kroll

Rich Kroll — Sun, 02 Jan 2011 13:23:04 +0000

Thanks for the great writeup! I was just wondering if you have used the fanout collector instead of combining the two sinks in one?

By: Tweets that mention Blog of Data » Blog Archive » Flume, Hive and realtime indexing via ElasticSearch -- Topsy.com

Fri, 31 Dec 2010 04:41:56 +0000

[...] This post was mentioned on Twitter by Planet Mozilla, anurag, anurag and others. anurag said: CORRECTED Flume, Hive and ElasticSearch at Mozilla – http://bit.ly/flume-es-hive WP cache seems to have an old copy, sorry for the bad link [...]