Those of us here on Mozilla Service’s Heka team were very pleased by the positive response and interest generated by our initial announcement about the project. And we’re even more pleased by the fact that some of you out there have decided to help out, contributing doc tweaks, bug fixes, and, in some cases, completely new plugins back to the Heka core. All the activity has kept us inspired, and we’ve landed a huge number of fixes and improvements ourselves since then. We’re happy to be rolling these out in a new Heka 0.3 release.

A full list of what’s new in this release can be found in the changelog, but here are some of the bigger features:

• ElasticSearch output: We had just decided that we wanted to write Heka message data out to ElasticSearch (so we could search through our data using a Kibana dashboard) when we received a pull request from Tanguy Leroux providing exactly that. The screenshot below is of a Kibana dashboard. It is displaying a histogram of the 10 (anonymized) Firefox Sync users who received the most 503 HTTP response codes over a specific period of time, extracted by Heka from our load balancer log files.
• Restartable plugins: It is now possible to specify any Heka input, filter, or output plugin as restartable, so it will reinitialize itself and start over when encountering an error. This is especially useful for plugins that require persistent connections to external services, as it allows them to reconnect. You can also set them to back off exponentially up to a user-defined cap, or add some timing jitter to prevent several reconnection attempts from happening simultaneously.
• Resume-from-location log file parsing: When shutting down, LogfileInput will note where it stopped parsing a log file, and will try to pick up from the same location when it restarts.
• Nagios output: If you use Nagios for monitoring, you can now use the NagiosOutput plugin to generate notifications triggered by Heka messages. Combine this with the ability to do arbitrary data processing in Heka’s dynamic Lua filters, and it becomes very easy to set up ad-hoc notifications for specific targeted events.
• Improved text parsing: We’ve moved the regular expression match group capturing functionality out of the router and into a decoder, so it won’t slow down routing of messages that don’t use capture groups. We also managed to add some timezone-shifting functionality, for cases where a non-UTC time zone is used but not specified in the timestamps.
• HTTP input: Thanks to an initial effort by David Delassus, we’ve now got an HttpInput plugin that will make HTTP requests and turn the resulting response bodies into Heka messages. You’ll need a custom Lua filter to parse the results and extract useful data, at least until the helpful decoders that we have under development are ready to take over that job for you.
• Cloudwatch input & output: We’ve added plugins to get data out of and into Amazon’s Cloudwatch metrics service. They’re not in the Heka core, but they’re in the Mozilla Services repository of custom Heka plugins and are available in the released binaries.
• New mailing list: There’s a new, dedicated Heka mailing list for announcements about changes to configuration options, Heka behavior, and anything else that might impact running Heka servers. Anyone interested in Heka should check it out!

As you can see, that’s a lot of progress. Big thanks to the Heka team and everyone who sent in patches, bug reports, and suggestions – keep them coming!

Heka is improving rapidly, but it’s still best suited for early adopters at this point. If you’re interested in rolling your sleeves up and digging in, please feel free to check out the binaries, the source code, and the documentation. And don’t forget to join the mailing list, and to drop in to the #heka channel on irc.mozilla.org to ask questions or share your experiences.