JS Probes

Have you ever had your browser mysteriously stall periodically and wondered “what the f#@$! is it doing?!!” Or perhaps you’re working on something, say the garbage collector, and you’d like to see what effect your changes are having. Or maybe even write a little analysis that postprocesses some sort of trace of what is going on, and figures out what the optimal pattern of actions would be. (“If I’d thrown this big chunk of data out of the cache here, then I would’ve had room for all of these little things that got evicted instead, and would have had way fewer misses…”)

The usual way to do things like this is to manually add some instrumentation code (probably just logging a bunch of events) and postprocess the results. This works fine, but it has a few drawbacks: (1) you have to figure out where to insert your instrumentation, often in unfamiliar code; (2) you’ll need to recompile, possibly several times; (3) the logs can get very large very quickly; and (4) you’ll probably end up writing a very special-purpose postprocessor that (5) dumps stuff to a text file that only you know how to interpret, and even you will only remember what it all means for a week or two. The next time you need to do something similar, you’ll find that all of your instrumentation code is severely bitrotted and misses some paths that have been added in the meantime, so you’ll start everything over from scratch.

Well, tough luck. Sometimes those are just facts of life and you’ll need to suck it up. Quit whining, dammit.

But many times, the events of interest (or more precisely, “probe points”) are of general interest. If you can manage to slip them into the code and so get other developers to maintain them for you as they make changes, then everyone can rely on those probes being in roughly the right place permanently. That’s #1 above, and depending on how they’re implemented there’s a good chance you won’t even need to recompile, so that’s #2.

I’ve done an implementation of these sorts of probes in the SpiderMonkey Javascript engine. There are probe points like “a GC is starting (and it’s local to one compartment)”, “the heap has been resized”, and “javascript function F is being called/is returning.” Some of these are straightforward to place into the code — the start of a GC wasn’t hard to figure out, for example. Some weren’t so straightforward, such as JS function calls (they might seem simple, but what if you’re running JITted? Which JIT? Are you still running JITted by the time you return from the function?) I’ve delivered the probe information to various backends — anything from Windows’ ETW (blog post forthcoming whenever I manage to implement the start/stop functionality), to dtrace/systemtap (another blog post, probably coming sooner since I recently scraped together a demo), to a simple callback mechanism (see JS_SetFunctionCallback on MDN) and other special-purpose ones that only care about a small subset of probes.

#3 (log it all vs online handling) ventures into religious territory. It is easiest to mindlessly log everything of interest and postprocess it. But what if you want realtime updates? Or if you want to track different information depending on what you learn from other probe points? Or what if the volume of your log writing interferes with whatever you’re trying to measure (eg disk I/O)? Or maybe you need to track some sort of state in order to give the probes meaning. (GC when idle => good. Avoidable GC when the user is waiting => bad.)

Those arguments are what led to the creation of tools like DTrace and Systemtap. Both give you a scripting environment that can aggregate information from probes as they fire, control exactly what information gets tracked as things are happening, and can be attached/detached at any time. They’re pretty cool, and invaluable once you get familiar with them. They’re also extremely system-dependent and generally require root access or special builds or kernel debuginfo or something, which ends up meaning that you often can’t just hand off analysis scripts to other people and have those people get some use out of them. And even you may not be able to take them to another environment.

Still, they deal pretty well with #4 (avoiding one-use, special-purpose processors), at least for environments matching the one they were written for. And if they can draw from statically-inserted probe points (the type I was talking about above), they can actually be pretty general. #5 is still a killer, though — at least the way I write systemtap scripts, they all end up with idiosyncratic ways of dumping out the results of some particular analysis, and nobody else is going to get much enlightenment without studying the script for a while first.

What if we could do better? What if we could insert these static probes, but rather than feeding the information to some niche tool that is usable by only a handful of people, we make the data available to a plain old Firefox addon? You could collect, aggregate, summarize, mutilate, fold, spindle, or crush the data directly in JS code. Then we could let addon authors go crazy with visualizations and analysis libraries. That’d be cool, right?

Graph GC behavior. Warn the user when slow or suspicious stuff is happening. Figure out what’s going on during long event handlers. Graph the percentage of time spent in different subsystems. Correlate performance/trace data with user-meaningful actions. Make a flight-recording of various metrics and let the user walk through history. Your ideas here.

Ok, so I tricked you. I’m not going to tell you how to do any of that. This blog post is a tease, an advertisement for the work that Brian Burg did this summer during his Mozilla internship. If you’re interested, he’ll be giving his internship final presentation tomorrow (today when you’re reading this, or perhaps yesterday or last month for those of you who have fallen behind on your Planet reading.) That’s 1:30PM PDT on Thursday, September 22 at the Mountain View Mozilla headquarters, and I’m 97.2% sure it will be broadcast over Air Mozilla as well. And taped, I think? (Sadly, I can’t find where those are archived. Somebody please tell me and I’ll update this post.) There will be a demo. With pretty pictures! And he’ll be writing it up on his own blog Real Soon Now. I’m not going to say any more for now — I’d get it wrong anyway.

Update: Argh! I got the date wrong! It’s not Wednesday, September 21 as I originally wrote. It’s today, Thursday, September 22. Sorry for the confusion!

Tags: , ,

4 comments

  1. Do you mean the 22? Because you did not post this until Wednesday evening.

  2. Frank Ch. Eigler

    Looking forward to Brian’s talk.

    FWIW, “They’re also extremely system-dependent and generally require root access or special builds or kernel debuginfo or something,” doesn’t describe modern systemtap well for purposes of instrumenting your own user-space program. For these, no root, nor kernel-debuginfo is needed. Check out unprivileged-mode support on recent fedora/rhel.

  3. Uh oh, I got busted. Sorry, this is probably FUD to some degree, partly due to my own incompetence, so let me say what this was based on.

    System-dependence: stock Fedora and RedHat have utrace, stock Ubuntu does not. Utrace-enabled kernels exist for Ubuntu, but you have to find and install them (or build your own.) As far as I know, the stock Android kernel does not have utrace enabled. And of course, neither OS X nor Windows can run systemtap, though OS X has dtrace.

    root access: this is probably just me, but whenever I try to run stap on anything as a regular user (in the stapusr group), it tries to contact a nonexistent compile server and fails. Oh… I probably need to specifically start up a server or something, I guess? But doesn’t that still require root access? (Maybe this is because I’m running the systemtap 1.6 that I compiled, so I don’t get startup stuff?) Regardless, it seems unlikely that a typical Linux addon user will have a systemtap server running.

    Kernel debuginfo or build environment: ok, you got me there. Most of the stap scripts I’ve written use a combination of user-space probes and kernel probes, but that’s an unfair comparison to the JS Probes stuff (which simply can’t use kernel probes at all.)

    Also, in its current prototypical stage, JS Probes doesn’t have a low-overhead way of inserting probes into userspace. Brian is faking it for now with full function calls that queue up data to another thread. Eventually, I’d like to piggyback on the sdt stuff for Linux. For Windows, Kevin Gadd has an initial implementation of the basic functionality needed. So at the moment, we’re talking smoke & mirrors.

    Oh, and don’t get me wrong — systemtap is awesome. I plan on using it for a bunch of detailed analysis tasks on my desktop (and hopefully on Android soon.) I just don’t think it’s going to be useful infrastructure for building general-usage addons that eat probe information.

    Though it’d be nice to have as an optional backend source for much richer information… be able to write a systemtap script that calls an API to feed information to the addon, perhaps. That’d be cool, especially since stap can digest things down to a small enough size that it wouldn’t interfere with the task being monitored.

  4. Frank Ch. Eigler

    Thanks a lot for the extra information. The unprivileged code does indeed require a local (or avahi-nearby) compile server installation (and a one-time root operation to authorize its use in this way).

    “I just don’t think it’s going to be useful infrastructure for building general-usage addons that eat probe information.”

    You are right, but it’s not quite as far as you might have thought. Plus, in a couple of months, we may start to see results on a new pure-userspace backend, which should take care of most of the above concerns.

    (In the mean time, you might want to investigate other technologies like LTTNG UST and gdb tracepoints.)

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.