{"id":298,"date":"2013-04-30T16:56:52","date_gmt":"2013-04-30T16:56:52","guid":{"rendered":"http:\/\/blog.mozilla.org\/services\/?p=298"},"modified":"2015-06-20T00:45:32","modified_gmt":"2015-06-20T00:45:32","slug":"introducing-heka","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/services\/2013\/04\/30\/introducing-heka\/","title":{"rendered":"Introducing Heka"},"content":{"rendered":"<p>We here on the Mozilla Services team are happy to announce our first beta release (v0.2b1) of <a title=\"Heka\" href=\"http:\/\/hekad.readthedocs.org\/\">Heka<\/a>, a tool for high performance data gathering, analysis, monitoring, and reporting. Heka&#8217;s main component is <a title=\"hekad\" href=\"http:\/\/hekad.readthedocs.org\/\">hekad<\/a>, a lightweight daemon program that can run on nearly any host machine which does the following:<\/p>\n<ul>\n<li>Gathers data through reading and parsing log files, monitoring server health, and\/or accepting client network connections using any of a wide variety of protocols (syslog, statsd, http, heka, etc.).<\/li>\n<li>Converts the acquired data into a standardized internal representation with a consistent metadata envelope to support effective handling and processing by the rest of the Heka system.<\/li>\n<li>Evaluates message contents and metadata against a set of routing rules and determines all of the processing filters and external endpoints to which a message should be delivered.<\/li>\n<li>Processes message contents in-flight, to perform aggregation, sliding-window event processing and monitoring, extraction of structured data from unstructured data (e.g. parsing log file output text to generate numeric stats data and\/or more processing-friendly data structures), and generation of new messages as reporting output.<\/li>\n<li>Delivers any received or internally generated message data to an external location. Data might be written to a database, a time series db, a file system, or a network service, including an upstream hekad instance for further processing and\/or aggregation.<\/li>\n<\/ul>\n<p>Heka is written in <a title=\"Go\" href=\"http:\/\/golang.org\/\">Go<\/a>, which has proven well-suited to building a data pipeline that is both flexible and fast; initial testing shows a single hekad instance is capable of receiving and routing over 10 gigabits per second of message data. We&#8217;ve also borrowed and extended some great ideas from <a title=\"Logstash\" href=\"http:\/\/www.logstash.net\/\">Logstash<\/a> and have built Heka as a plugin-based system. Developers can build custom Input, Decoder, Filter (i.e. data-processing), and Output plugins to extend functionality quickly and easily.<\/p>\n<p>All four of the plugin types can be implemented in Go, but managing these plugins requires editing the config file and restarting and, if you&#8217;re introducing new plugins, even recompiling the hekad binary. Heka provides another option, however, by allowing for &#8220;Sandboxed Filters,&#8221; written in <a title=\"Lua\" href=\"http:\/\/lua.org\/\">Lua<\/a> instead of Go. They can be added to and removed from a running Heka instance without the need to edit the config or restart the server. Heka also provides some Lua APIs that Sandboxed Filters can use for managing a circular buffer of time-series data, and for generating ad-hoc graph reports (such as the following example) that will show up on Heka&#8217;s reporting dashboard:<\/p>\n<p style=\"text-align: center;\"><a href=\"http:\/\/blog.mozilla.org\/services\/files\/2013\/04\/56.png\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-full wp-image-299\" src=\"http:\/\/blog.mozilla.org\/services\/files\/2013\/04\/56.png\" alt=\"56\" width=\"1257\" height=\"596\" srcset=\"https:\/\/blog.mozilla.org\/services\/files\/2013\/04\/56.png 1257w, https:\/\/blog.mozilla.org\/services\/files\/2013\/04\/56-300x142.png 300w, https:\/\/blog.mozilla.org\/services\/files\/2013\/04\/56-1024x485.png 1024w\" sizes=\"(max-width: 1257px) 100vw, 1257px\" \/><\/a><\/p>\n<p>Heka is a new technology. We&#8217;re running it in production in a few places inside Mozilla, but it&#8217;s still a bit rough around the edges. Like everything Mozilla produces, however, it&#8217;s open source, so we&#8217;re releasing early and often to make it available to interested developers (contributors \/ pull requests welcome!) and early adopters. Here&#8217;s a list of resources for those who&#8217;d like to learn more:<\/p>\n<p>&nbsp;<\/p>\n<ul>\n<li><a title=\"Heka v0.2 binaries\" href=\"https:\/\/docs.services.mozilla.com\/_static\/binaries\/hekad-0.2\/\">Heka v0.2 binaries<\/a><\/li>\n<li><a title=\"Heka presentation\" href=\"https:\/\/people.mozilla.com\/~rmiller\/heka-intro-2013-03\/\">Intro to Heka<\/a> HTML presentation slides<\/li>\n<li><a title=\"Heka documentation\" href=\"http:\/\/hekad.readthedocs.org\/\">Heka project documentation<\/a><\/li>\n<li><a title=\"Heka github project\" href=\"https:\/\/github.com\/mozilla-services\/heka\">Heka github project<\/a><\/li>\n<li><a title=\"Heka mailing list\" href=\"https:\/\/mail.mozilla.org\/listinfo\/heka\">Heka mailing list<\/a> (for developer support)<\/li>\n<li>IRC: #heka channel on irc.mozilla.org<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>We here on the Mozilla Services team are happy to announce our first beta release (v0.2b1) of Heka, a tool for high performance data gathering, analysis, monitoring, and reporting. Heka&#8217;s main component is hekad, a lightweight daemon program that can &hellip; <a class=\"go\" href=\"https:\/\/blog.mozilla.org\/services\/2013\/04\/30\/introducing-heka\/\">Continue reading<\/a><\/p>\n","protected":false},"author":598,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30944],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/services\/wp-json\/wp\/v2\/posts\/298"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/services\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/services\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/services\/wp-json\/wp\/v2\/users\/598"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/services\/wp-json\/wp\/v2\/comments?post=298"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/services\/wp-json\/wp\/v2\/posts\/298\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/services\/wp-json\/wp\/v2\/media?parent=298"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/services\/wp-json\/wp\/v2\/categories?post=298"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/services\/wp-json\/wp\/v2\/tags?post=298"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}