mozilla-central automated landing proposal

sfink    Uncategorized     automation, mozilla, planet

This was originally a post to the monster thread “Data and commit rules” on dev-planning, which descended from the even bigger thread “Proposing a tree rule change for mozilla-central”. But it’s really an independent proposal, implementable with or without the changes discussed in those threads. It is most like Ehsan’s automated landing proposal but takes a somewhat different approach.

• Create a mozilla-pending tree. All pushes are queued up here. Each gets its own build, but no build starts until the preceding push’s build is complete and successful (the tests don’t need to succeed, nor even start.) Or maybe mostly complete, if we have some slow builds.
• Pushers have to watch their own results, though anyone can star on their behalf.
• Any failures are sent to the pusher, via firebot on IRC, email, instant messaging, registered mail, carrier pigeon, trained rat, and psychic medium (in extreme circumstances.)
• When starring, you have to explicitly say whether the result is known-intermittent, questionable, or other. (Other means the push was bad.)
• When any push “finishes” — all expected results have been seen — then it is eligible to proceed. Meaning, if all results are green or starred known-intermittent, its patches are automatically pushed to mozilla-central.
• Any questionable result is automatically retried once, but no matter what the outcome of the new job is, all results still have to be starred as known-intermittent for the push to go to mozilla-central.
• Any bad results (build failures or results starred as failing) cause the push to be automatically backed out and all jobs for later pushes canceled. The push is evicted from the queue, all later pushes are requeued, and the process restarts at the top.
• When all results are in, a completion notification is sent to the pusher with the number of remaining unmarked failures

Silly 20-minute Gimped-up example:

1. Good1 and Good2 are queued up, followed by a bad push Bad1
2. The builds trickle in. Good1 and Good2 both have a pair of intermittent oranges.
3. The pusher, or someone, stars the intermittent oranges and Good1 and Good2 are pushed to mozilla-central
4. The oranges on Bad1 turn out to be real. They are starred as failures, and the push is rolled back.
5. All builds for Good3 and Good4 are discarded. (Notice how they have fewer results in the 3rd line?)
6. Good3 gets an unknown orange. The test is retriggered.
7. Bad1 gets fixed and pushed back onto the queue.
8. Good3′s orange turns out to be intermittent, so it is starred. That is the trigger for landing it on mozilla-central (assuming all jobs are done.)

To deal with needs-clobber, you can set that as a flag on a push when queueing it up. (Possibly on your second try, when you discover that it needs it.)

mozilla-central doesn’t actually need to do builds, since it only gets exact tree versions that have already passed through a full cycle.

On a perf regression, you have to queue up a backout through the same mechanism, and your life kinda sucks for a while and you’ll probably have to be very friendly with the Try server.

Project branch merges go through the same pipeline. I’d be tempted to allow them to jump the queue.

You would normally pull from mozilla-pending only to queue up landings. For development, you’d pull mozilla-central.

Alternatively, mozilla-central would pull directly from the relevant changeset on mozilla-pending, meaning it would get all of the backouts in its history. But then you could use mozilla-pending directly. (You’d be at the mercy of pending failures, which would cause you to rebase on top of the resulting backouts. But that’s not substantially different from the alternative, where you have perf regression-triggered backouts and other people’s changes to contend with.) Upon further reflection, I think I like this better than making mozilla-central’s history artificially clean.

The major danger I see here is that the queue can grow arbitrarily. But you have a collective incentive for everyone in the queue to scrutinize the failures up at the front of the queue, so the length should be self-limiting even if people aren’t watching their own pushes very well. (Which gets harder to do in this model, since you never know when your turn will come up, and you’re guaranteed to have to wait a whole build cycle.)

You’d probably also want a way to step out of the queue when you discover a problem yourself.

Did I just recreate Ehsan’s long-term proposal? No. For one, this one doesn’t depend on fixing the intermittent orange problem first, though it does gain from it. (More good pushes go through without waiting on human intervention.)

But Ehsan’s proposal is sort of like a separate channel into mozilla-central, using the try server and automated merges to detect bit-rotting. This proposal relies on being the only path to mozilla-central, so there’s no opportunity for bitrot.

What’s the justification for this? Well, if you play fast and loose with assumptions, it’s the optimal algorithm for landing a collection of unproven changes. If all changes are good, you trivially get almost the best pipelining of tests (the best would be spawning builds immediately). With a bad change, you have to assume that all results after that point are useless, so you have no new information to use to decide between the remaining changes. There are faster algorithms that would try appending pushes in parallel, but they get more complicated and burn way more infrastructural resources. (Having two mozilla-pendings that merge into one mozilla-mergedpending before feeding into mozilla-central might be vaguely reasonable, but that’s already more than my brain can encompass and would probably make perf regressions suck too hard…)

Side question: how many non-intermittent failures happen on Windows PGO builds that would not happen on (faster) Windows non-PGO builds?

Wading through history

sfink    Uncategorized     hg, mercurial, mozilla, mq, mqext, planet

Recently — well, actually, by now it wasn’t recently at all — I received a review request for a patch to JSD. It fixed an intermittent crash when using Firebug on a page that went into an endless stack-eating loop. A couple of people had worked on reproducing it, and the exact conditions were a little flaky, so I first tried it out myself. Kaboom! Yay!

So I imported the patch just to verify that it fixed the problem. Before compiling with it, I updated my tree to the latest version. Why? I don’t know. Just because it’s what I usually do. It seemed like a good idea at the time.

Only it wasn’t. It was a really, really dumb idea. I was changing two variables while trying to test one of them, and I got what I deserved: it stopped crashing after the patch, but when digging in to verify that it really was behaving as intended, I discovered it still wasn’t crashing.

This was just before the All Hands, and although I poked at it every few days, I didn’t make any headway: the patch seemed good, but I really wanted to confirm that it fixed the crash. (There were reasons why I was a little skeptical, but it’s not really relevant here.)

Eventually, when I had some time to think about it properly, I realized the best thing to do would be to revert to the older version that crashed for me. But how to find it?

One way would be to binary search nightlies. But I happened to be on a poor network connection, and downloading nightlies was insanely slow.

Also, I thought I should be able to do better. I run with an mq extension (mq = Mercurial Queues) that commits my patch queue on any change. Get it at git://github.com/hotsphink/mqext.git (I really should switch to bitbucket, rather than pointlessly restricting my audience to people who are minimally comfortable with both git and hg.) So all I had to do was to go back to the point where I imported the patch from bugzilla.

Finding the right moment was easy: ‘hg log –mq’ showed me all the changes made to my patch queue, one of which was commented “IMPORT: bz://643360″ (an autogenerated comment courtesy of mqext.)  That was changeset 026ac43e9114. Yay!

But that changeset is for my patch queue, not my source repo. Fortunately, mq stores ‘parent’ fields in patch files that give the source repo changeset id that a patch was applied on top of. I’ll skip a number of failed attempts to track through this, and just give my final recipe:

1. (already described) hg log –mq to find the appropriate changeset in the patch queue repo.
2. cd to .hg/patches and run hg cat -r changeset series. This is because you need to know the names of the patch files in order to look at them — or specifically, the name of the first patch file, because it’s the only one whose parent will still be in the source repo. All other patches’ parents will be the source repo with mq patches applied to them, and will have been stripped out of the repo due to intervening actions. Because hg (or rather, mq) is not interested in preserving history.
3. hg cat -r firstpatchname and look for the “# Parent changeset” line.
4. cd back to your source repo and fetch that revision however you want — update to it, or clone a repo with it, or whatever.

I’m guessing this little recipe isn’t going to be useful to very many people, but I wanted to write it out for myself. So phbbbtt!!!

Work Configuration

sfink    Uncategorized     hg, mercurial, mozilla, planet

Inspired by Nicholas Nethercote’s description of how he sets up his tracemonkey work environment, I thought I’d describe my work configuration and how it differs from njn’s.

Like Nick, I work almost entirely off of the tracemonkey tree these days, and mostly within js/src. I don’t use the js shell all that much compared to the full browser, though, so I tend to do things with the whole tree.

working repositories

Similar to Nick, I have a ~/src/ directory populated with clones of the tracemonkey repo. I have one, “TM-upstream/”, that follows the upstream tracemonkey repository. In fact, I use cron to pull updates hourly. The rest are created as clones of TM-upstream, or sometimes of each other. I vary in how I create these. Some are created via ‘hg clone TM-upstream TM-whatever’, although for whatever reason I usually do ‘cp -rlp TM-upstream TM-whatever’ and then edit TM-whatever/.hg/hgrc to change the ‘default’ path to TM-upstream. The ‘cp’ method is faster, but the end result is pretty much the same. Sometimes I copy the mq subdirectory (.hg/patches) from the repo I’m cloning, sometimes I create a new one from scratch. And sometimes I don’t use one at all.

Oh, and with emacs I had to do

  (setq vc-make-backup-files t)

to make it break hardlinks when modifying files. Breaking hardlinks is normally the default, but it seems like vc mode has a different default that is really really bad if you’re using ‘cp -rlp’ to clone your repos.

All of my (tracemonkey-based) repos start with “TM-”, probably because I use my src/ subdirectory for checkouts of various other projects (bugzilla-tweaks, archer-mozilla, archer, firebug, addon-sdk, etc.). Not all of those are hg-based; I have several git repos and even an svn checkout or two. For the Mozilla tree, I tend to only actively use one or two repos at a time; the rest are for dormant unfinished work.

I made a shell function ‘pullup’ that does ‘(cd $(hg path default) && hg pull)’, which goes to the default upstream repo (probably TM-upstream, unless this is a clone of a clone) and updates its objects. (Note the lack of a -u; I don’t want to update the working directory for the upstream repo without a good reason.) To update my working repo, I’ll ‘hg qpush -a’ to apply as many patches as I can, then probably ‘hg qpop’ to pop off the last one because it failed. (I tend to have a small pile of heavily bitrotted patches lurking around at the end of my series file.) Then I’ll do ‘pullup’ to update the upstream repo and ‘hg pull –rebase’ to merge the changes into my patch queue. My ~/.hgrc sets my merge tool to kdiff3, so any conflicts will pop up the visual merge editor.

I push changes directly from my working repo by using

  hg qpop
  hg show | head
  hg qref -e # if needed

to fix up the commit messages, then qpush everything back on that I’m committing. (I tend to break up my commits into at least 2 pieces, so I usually push more than one change at a time.) Then I do ‘hg qfinish -a’, do my last round of testing, and ‘hg push tracemonkey’ (tracemonkey is set in the [paths] section of my ~/.hgrc).

I don’t bother to run ‘hg outgoing’, because I only commit patches that I’m about to push. I suppose if I were collaborating with someone else, I might get some extra crud that I’d need to worry about, but so far I’ve always done that through patches imported into my patch queue.

object directories

I place my object directories underneath the source directory, so that I can use hg commands while my working directory is underneath the object directory. I mostly use plain ‘~/src/TM-whatever/obj’, which is almost always a debug build. If I need an opt build, it’ll be ‘obj-opt’ in place of ‘obj’. Rarely, I’ll make ‘obj-somethingelse’ for special purposes.

Prefixing things with ‘obj’ helps when moving stuff between machines, because I can do

  rsync -av --exclude='/obj*' TM-whatever desthost:/some/where

building

When underneath obj/js/src, I’ll just run ‘make’ or ‘make -j16′ or whatever to rebuild (even when testing with the browser, because my mozconfig always has ‘ac_add_options –enable-shared-js’ so rebuilding here is enough. In fact, I tend to forget to remove it when making opt builds for performance testing.)

I also tend to modify things in js/jsd and js/src/xpconnect/src, so I have a special makefile that does a minimal rebuild for those:

ROOT := $(shell hg root)

all:
 $(MAKE) -C $(ROOT)/obj/js/src
 $(MAKE) -C $(ROOT)/obj/js/jsd
 $(MAKE) -C $(ROOT)/obj/js/src/xpconnect/src
 $(MAKE) -C $(ROOT)/obj/layout/build
 $(MAKE) -C $(ROOT)/obj/toolkit/library

I have that saved as ~/mf, and I have a shell alias ‘mk’ that does ‘make -f ~/mf’. So I’ll make my changes, then run ‘mk -k -j12′ or whatever. (I don’t know why I bother to give numbers to my -j options, since I use distcc’s hosts syntax for limiting concurrent jobs anyway.)

Even lazier, I have my emacs set up to pick the right make command depending on what directory I’m in (please excuse my weak elisp-fu):

; Customizations based on the current buffer's path

(defun get-hg-dir (path)
 (if (equal path "/")
 nil
 (if (file-exists-p (expand-file-name ".hg" path))
 (expand-file-name ".hg" path)
 (get-hg-dir (directory-file-name (file-name-directory path))))))

; For Mozilla source:
;  - if within an hg-controlled directory, set the compile-command to
;      make -f ~/mf...
;    which will do a fairly minimal rebuild of the whole tree
;  - unless we're also underneath js/src, in which case, just do a make
;    within the JS area
(defun custom-compile-hook ()
 (let ((path (buffer-file-name))
 (dir (directory-file-name (file-name-directory (buffer-file-name)))))
 (if (not (null (get-hg-dir path)))
 (if (string-match "js/src" dir)
 (set (make-local-variable 'compile-command)
 (concat "make -C " (expand-file-name (concat dir "/../../obj/js/src")) " -k"))
 (set (make-local-variable 'compile-command)
 (concat "make -f ~/mf -k -j12"))))))

(add-hook 'find-file-hook 'custom-compile-hook)

I have my F12 key bound to ‘compile, so I just hit F12, check that the command is right, then press enter to build. One problem I have is that our build output is much too verbose, so I don’t notice warnings very well. I keep meaning to shut it up (probably by only printing the file being compiled unless there are errors/warnings), but I haven’t gotten around to it.

compiling: distcc and ccache

I rely heavily on distcc for my builds. I do almost all of my Mozilla work on a single laptop machine, though occasionally I’ll reboot it into Windows to suffer through something there, or use one of my two desktops (one home, one work). My work desktop is quite beefy. My home desktop is less so, but still good enough to speed up builds dramatically. I run a cron job on my laptop to autodetect where I am and switch my ~/.distcc/hosts symlink to the appropriate hosts file, which contains “localhost finkdesk/12″ at work and “localhost 192.168.1.99/7″ at home. The /12 and /7 are the max number of concurrent jobs distcc will trigger; I set it lower on my home machine to keep from bogging it down with contending jobs, though honestly I haven’t benchmarked to see what the right numbers are.

About half the time, I’ll have distccmon-gnome running to monitor where the jobs are going to. It’s a quick way to spot when I’m sending things to the wrong place (eg when I’m VPNed into the work network and finkdesk is reachable; if I accidentally send things there, distcc will slow everything down because the network time way outweighs the compilation speedups.) Or, more often, that something’s messed up and all builds are going to localhost. Or that I’m only getting a single job at a time because I forgot to use -j again.

I also use ccache at all times, but I don’t do anything nonstandard with it. Just be sure to set CCACHE_PREFIX=distcc and allow it to get big with ‘ccache -M’.

linking: gold

When I’m working outside of js/src proper, I also like to use the gold linker in place of the default binutils bfd linker. I’m on Fedora 14, so to switch to gold I do

  cd /etc/alternatives
  rm ld
  ln -s /usr/bin/ld.gold ld

(and to switch back, link to ld.bfd). gold takes my minimal links from 30 seconds to about 10 seconds, which is really nice. Unfortunately, I frequently have to switch back to ld.bfd due to incompatibilities. elfhack and valgrind are the usual offenders. Update: According to jseward, valgrind >= 3.6.0 should work fine. Yay! (I currently have 3.5.0).

patch queue

While they’re in my mq, all of my patches are labeled with the bug number and a brief description. When I’m reshuffling changes between my various patches, I create temporary patches whose names are prefixed with “M-” (for Merge) to indicate that I’m planning on qfolding them into some other existing patch. I also use “T-” for temporary patches (debugging printouts or whatnot). It helps to see the state of everything with a glance at my ‘hg qseries -v’ output (which, due to aliases and defaults, I actually spell ‘hg series’).

Very recently, I’ve started using ‘hg qcrecord’ to split up and reorganize patches, and I’m loving it. It’s the same basic story, though — I use it to create temporary to-be-merged patches that I qfold later. I tend to do

  hg qref -X '*'
  hg qcrecord

quite a bit to move stuff out of the current patch (well, the current patch + the current changes on top of it).

disk space

Finally, I also try to occasionally go through all my TM-* directories and run ‘hg relink’ to rediscover what can be hardlinked. It takes a while, so I really ought to cron it. It tends to recover surprisingly large amounts of disk space.

Complete and total tangent:

My underinformed, overopininated take on this is that hg’s disk structures are wrong. As I understand it, the wasted space comes from: (1) you clone a repo, which creates a bunch of hardlinks, using very little space; (2) you periodically update the base repo, breaking many of the hardlinks; then (3) you update the derived repo with those changes. hg doesn’t figure out that it can re-link the object files — which is understandable, since it would need to know for a given file that not only are the latest versions identical, but also that the complete set of revisions between the two repos is identical.

It doesn’t seem that hard for it to figure this out. But even if it did, any local change in the derived repository is going to prevent sharing anyway. That’s what bugs me. Conceptually, hg’s object store is a big pile of byte strings, one for every revision of every file, and each tagged with (and looked up by) its checksum. There’s an optimization that all the revs of a single file can be stored compactly as a set of deltas rather than storing a full (compressed) copy of every rev, but that really ought to be an optimization, not a fundamental data structure. If you ditched the optimization entirely and kept a full copy of every rev, you could trivially share a repo across all of your checkouts. (You could even share a repo with completely unrelated projects, though that’d be more likely to hurt than help.) I would find this much nicer.

Actually, it’s not just that all the versions of a file need to be stored within one filesystem file. hg seems to want the set of versions within a filesystem file to mean something. I would rather have that information (the set of known revisions) stored within a checkout, so that extra revs would be harmless. Then you don’t need to lose the optimization; you can still stuff all revisions into one file, even revisions from completely unrelated branches. You’d even have flexibility to use multiple filesystem files for a single source file, if it has a bunch of revisions that you want rapid access to. (So file1 contains revA + a few deltas, file2 has revB only, file3 has revC + a few deltas, etc. Think images.)

I think I’m probably describing git’s data structures here. If so, it seems like git has it right. Checkouts should have their own state, history, etc., but feed off of a chaotic assortment of checksummed data wads that are optimized for whatever you want to optimize for. It gives much more flexibility.

You shouldn’t even really need to have all revisions stored locally, if you know of a place on the network where you can find old/unrelated revisions when you want them. If you ever ask to jump back 3 years, then sure, it’d take a while to pull down the needed data, but most of the time you’d save lots of disk space for stuff you’re never going to ask for anyway. (And if it bothers you, you can always pull it all down.)

Or maybe I’m wrong about how hg does things.

Whew

Ok, that was long. Thanks for making it this far.  Let me know what I got wrong or what I’m doing stupidly. Preferably with a description of your vastly better way of doing it!

Updating UUIDs (the wrong way)

sfink    Uncategorized     idl, mozilla, uuid

When you change an IDL interface, you have to update its uuid. Simple enough, just grab a new uuid and stick it in the .idl file. Easy enough, right?

I’ve been working on JSD (the JavaScript debugger interface) recently, and its IDL file contains 18 different interfaces. So say I add a method to jsdIScript. What interfaces do I need to update? From empirical observations (as in, when I forget to do one I get yelled at by a reviewer), you have to update the uuid on the interface containing any added methods, any interfaces that use that interface within their definitions, and any interface that… inherits? extends? that interface. Recursively.

Update: No you don’t, says #developers, though currently the conversation is not finished and it seems like it may come down to something of a judgement call (something like “if you modify the vtable, update the uuid. If you change an interface used in a method in such a way that it might break a user, update the uuid.”) I’ve updated my script to use the more common rule (inheritance only), though if you want the stricter behavior you can get it with –mode=params.

Leaving the rest of this post here for now:

Solution 1: Manually go through and trace the dependencies, updating uuids as you go.

Bleckth. Maybe someone else can manage to do that properly, but I’m an airhead, and I’d never get that exactly right.

Solution 2: Nuke ‘em all and let God sort ‘em out. (As in, update every single uuid in the file.)

Easy, but inelegant and I sometimes wonder if knowing whether an interface changed might actually matter to someone. Besides, this could result in me getting yelled at by a reviewer, and that’s what I depend on for figuring out the rules (see “empirical observations”, above.)

Solution 3: automate solution 1.

…so I did.

Give it a .idl file to chew on and one or more interfaces that you know you’ve changed, and it’ll chase through the dependencies for you. Assuming I got the rules right. It spits out a new file with brand-new uuids on all affected interfaces, and even spews to stderr the set of interfaces it’s updating and why:

% update-uuids jsdIDebuggerService.idl jsdIContext >/dev/null
uuids to update:
 jsdIContext because it was given on command line
   3e5c934d-6863-4d81-96f5-76a3b962fc2b -> 24ad10b2-8b4f-49f6-9236-f0ecaed0e19a
 jsdIStackFrame because its body contains jsdIContext
   7c95422c-7579-4a6f-8ef7-e5b391552ee5 -> 4c8c5902-77e8-4a9d-99f8-0ae6f0c58eec
 jsdIContextEnumerator because its body contains jsdIContext
   57d18286-550c-4ca9-ac33-56f12ebba91e -> d102ff63-59ea-4ed7-86d5-490c9e9b6b5a
 jsdICallHook because its body contains jsdIStackFrame
   3eff1314-7ae3-4cf8-833b-c33c24a55633 -> 150610f5-89cd-4f76-b960-06447471eb00
 jsdIExecutionHook because its body contains jsdIStackFrame
   3a722496-9d78-4f0a-a797-293d9e8cb8d2 -> cd3bfe98-c8a3-4c98-91d1-c7e2e79c396c
 jsdIDebuggerService because its body contains jsdIContextEnumerator
   aa232c7f-855f-4488-a92c-6f89adc668cc -> 75ab47da-2400-4efe-bb5e-745dceba4e06

(Sorry, no examples of inheritance-triggered changes there.)

It’s a quickie parse-with-regexes Perl hack.

One major flaw — it only considers a single .idl file. If some other .idl file depends on an interface modified within your file, then this won’t tell you it needs to be updated. I’m pretty sure that no other IDLs depend on the JSD IDL, so I don’t care yet. If this would be useful to you and that’s a necessary feature, let me know and I’ll throw it in. It’s easy enough to implement as long as you provide the list of IDL files to consider.

The other major flaw is that this doesn’t update the uuids in header files, once again because JSD didn’t need it. That would be some more work, and I don’t even know if I have the rules right so I’m not going to bother unless someone asks me to and tells me this is the right thing in the first place.

If nobody comments here within a week or so telling me I’m completely wrong about how this stuff works, I’ll add a link to this script on MDC. Maybe. If I remember. (And it turns out, someone did tell me I’m completely wrong. Yay!)

Fun and Games With gdb

sfink    Uncategorized     gdb, mozilla, planet, tools

As with most developers, I have a love/hate relationship with gdb, and have built up a clunky set of tips, tricks, and workarounds that I commonly use. I thought I’d share some of mine here, in a typically disorganized fashion:

Breakpoint Command Lists

This may be well known to everyone but me, but gdb lets you run arbitrary canned expressions every time a breakpoint is hit. This is incredibly useful when you need to know the last time something changed before a crash or some other detectable event. The usual problem is that if you set a breakpoint or watchpoint or whatever, you’ll hit it over and over again before the “interesting” one happens, and it’ll take way too long and you’ll give up in disgust. So instead, try something like this:

  (gdb) b somefilename.cpp:1234
  Breakpoint 2 at ...
  (gdb) command 2
  > bt
  > cont
  > end
  (gdb) cont

Now, gdb will run normally, except whenever that breakpoint is hit it’ll dump out the full stack and then continue. When you crash or hit your other breakpoint or whatever, you can look backwards to the previous time you hit breakpoint 2 (hardware watchpoints are especially good with this) and see what the stack was then. If the breakpoint is hitting a lot, this may not be fast, but you can leave it running unattended all night if you have to.

I’ve tried getting clever with this, with mixed results: once, I wanted to know when a particular value changed. I could set a hardware watchpoint, but it was constantly getting changed by a function f() that I didn’t care about. It was some other mysterious source of mutation that was tripping me up. So I set the hardware watchpoint, and then set breakpoints on the beginning and end of f(). I then created a command list for the breakpoint on the start of f() that disabled the watchpoint and continued, and a command list for the breakpoint on the end of f() to re-enable the watchpoint.

Unfortunately, it didn’t work. I still don’t know why. Maybe there’s something weird about watchpoints. I didn’t have time to dig into it then. But you get the idea. (Hmm… if it was having trouble with enable/disable, perhaps I should have set and cleared a convenience variable $active and made the watchpoint conditional on that…)

Logging From gdb

Occasionally, I have some big wad of data that needs to be interpreted by a different program. Let’s say it’s in a char* variable with its length stored in another variable. My usual approach is to print out the contents using the ‘x’ or ‘p *foo @ 100′ or whatever, then use emacs or Perl to unescape. But for large wads of data, this is unworkable. (For example, I’ve done this with entire images.)

So write it out to a file instead:

  (gdb) p creat("/tmp/somefile.dat", 0777)
  $26 = 37
  (gdb) p write(37, s, len)
  $27 = 168
  (gdb) p close(37)
  $28 = 0

…and now you have a handy little file containing just your data and nothing else.

In conjunction with the previous trick, you can append wads of binary data to a trace file every time you hit a breakpoint. Fun!

Watchpoints

This may be outdated. gdb has a nice watchpoint facility, but it has a tendency to take you too literally — when you ask it to watch a particular expression, it often sets a software watchpoint on that expression, meaning it constantly re-evaluates the expression over and over and slows execution down to a crawl. If you know that the various pointers involved aren’t going to change (or if they change, you want the current one anyway), it’s often handy to take the address of the expression you want to watch and then watch the dereference of that:

  (gdb) p &fun->atom
  $27 = (JSAtom **) 0x7fffe3b6b778
  (gdb) watch *$27
  Hardware watchpoint 5: *$27

This doesn’t seem to be as necessary as it used to; it seems to do hardware watchpoints automatically in many more cases. But it still complains when the expression is no longer valid.

I notice (while writing this) that there is now a -l option for doing something like this automatically. I’ve never tried it, but it sounds like it’d do what I want.

Automagical debugging

I nearly always run gdb within emacs so it automatically loads and highlights the current line of code for me (and gives history, etc.) But say I want to run

  myprog -s -t lamppost -- monkey walrus -teeth-

I have to:

• run emacs
• M-x gdb
• type “myprog”<enter>
• type “run -s -t lamppost — monkey walrus -teeth-”

…and that means retyping it all or cutting & pasting, even though I probably just ran that command. For running Firefox under the debugger, it’s even worse, because on Linux ‘firefox’ is actually a shell script that sets up the environment and then invokes ‘firefox-bin’. There, you have to use the ‘-g’ option on the shell script to do the right thing. I also want to start up the Perl debugger under emacs for Perl scripts, and (not that you care) I used to work on a program at a previous company that had an embedded scripting engine, and I sometimes wanted to run gdb on the C++ generated binary, and sometimes the script debugger on the embedded scripts.

Phew. So I wrapped it all up in a ‘debug’ script that guesses what you want and twiddles the options to emacs, gdb, Perl, and/or the Firefox wrapper script to do the right thing. Usage:

  debug firefox -no-remote -P test about:blank

That will bring up an emacs window running gdb on firefox-bin with the environment all set up automagically. (It makes some extreme assumptions, such as guessing that if you give it a shell script then it’s the firefox wrapper shell script so it can use the -g and -d options.) It is surprisingly robust; if you use ‘debug’ in place of ‘gdb’ in commands that accept a debugger, it’ll often just work. For example, here’s a way to run a single directory’s worth of mochitests:

  cd obj/_tests/testing/mochitest
  python runtests.py --test-path=js/jsd/test

But it turns out runtests.py has a –debugger argument, so you can do:

  python runtests.py --test-path=js/jsd/test --debugger=gdb

and it’ll run the browser under gdb and give you a command-line prompt. So, to make it nicer:
  python runtests.py --test-path=js/jsd/test --debugger=debug

Voilà! You get emacs running gdb running firefox running your mochitests.
I just tossed a snapshot of the script onto http://people.mozilla.org/~sfink/uploads/debug. It has some of my environment hardcoded in it, but it should probably work for you unmodified anyway. (The main thing I have to keep tweaking is the –annotate option to gdb. Either gdb or emacs changed somewhat recently.)
Conditionals with strings
Say you want to set a breakpoint on a certain line when script->filename is your script, perhaps “http://lifewithducks.com/cheesetasting/reformat.js”. You could do
  cond 7 strcmp(script->filename, "http://lifewithducks.com/cheesetasting/reformat.js") == 0
but that’s (1) a lot of typing, and (2) slow because that breakpoint may get hit a lot and gdb has to do some ptrace-y context dance to invoke expressions every time. For the typing problem, strstr() is your friend:
  cond 7 strstr(script->filename, "reformat")
Specifically, strstr() returns the location of a substring within a larger string, or NULL if it doesn’t find it. So you just need to call it with a unique snippet that you’re looking for, and it’ll Do The Right Thing.
For the “slow” problem, er… well, that example is going to be really messy. Let’s simplify and pretend you’re interested in breaking when strcmp(s, “spatula”) == 0. Here’s my evil shortcut:
  cond 7 *(int*)s == *(int*)"spatula"
gdb seems to be able to handle that expression much, much faster. Ok, that sort of assumes that your ints are 4 bytes. But again assuming 4-byte ints, it’s equivalent to
  cond 7 *(int*)s == *(int*)"spat"
because it’s treating the literal “spatula” as an address of a character array, and *(int*)”spatula” is thus giving the 1st 4 bytes interpreted as an integer. Here, that’s 1952542835 aka 0×74617073 ({ ‘t’, ‘a’, ‘p’, ‘s’ }), because I’m little-endian. If you want more characters and you’re on 64-bit, cast to (long*) instead. Or use multiple pieces:
  cond 7 ((int*)s)[0] == ((int*)"spatulamurders")[0] && ((int*)s)[1] == ((int*)"spatulamurders")[1] && ...
Obviously, this is really not helping the typing problem. I didn’t say I’d solve both at once, ok?
Back to the original example, you can use the latter trick to pick out the unique piece of the string you’re interested in. Let’s say that “reformat.js” is what we really care about, we’re on 64-bit, and we’ll ignore the string-length problem:
  cond 7 ((long*)script->filename)[5] == *(long*)"eformat."
I don’t use this often, but it’s been useful when I’ve needed it.
Mozilla Customizations
The Mozilla source uses a bunch of datatypes that gdb doesn’t do very well with, from 16-bit characters in strings to big nasty structures that are mostly not interesting but clutter up the output. I use a modified version of this .gdbinit. I also highly, highly recommend Jim Blandy’s archer-mozilla gdb extensions written in Python. I’d actually like to stop using that .gdbinit, since it’s written directly in gdb’s “scripting” language, and is painfully slow for printing strings. It should be rewritten with the Python stuff.
When I was working on compartment-related problems, I took a stab at modifying jimb’s stuff to automatically display the compartment for strings and to pretty-print compartments with their principals. It was pretty handy, though I doubt you’d want it to be active all the time. I should look into controlling it with an option or something, I guess. If you’re interested, you can find my changes at bitbucket (but note that jimb’s repo is the real repo; I just pushed to bitbucket so I could point to it from this blog entry.)
You may need to recompile gdb from the archer gdb sources for this stuff to work; I’m not sure. See jimb’s blog post for details.
How About You?
Those are all the ones I can think of for now. I’d be interested in hearing about other people’s tricks. It seems like everyone uses gdb a bit differently, so there’s a lot of opportunity for cross-pollination.
 
			

tedium? no, delirium

sfink    Uncategorized     mozilla

Bleh. Went to work, started feeling crappy, came home. Wrapped myself in six layers — heavy blanket, heavy coat, medium coat, 3 shirts. Still shivering. Kids not too happy to have Dad out of action, and decided to fight. Really didn’t help.

Ah, but now is the fun part. Fever went up somewhere near 40C, whatever that is. Managed to pick up Celsius thermometer in Japan, and it always seems to be the one on hand. At least, the one that hasn’t been stuck up a kid’s butt. I know that 37.0C suspiciously maps to exactly 98.6F, but I just don’t have the “feel” for one degree C.

I will not publish this post.

Fever dropped, leading to the only fun part: delirium. Head still pounding, back aches, eyes ache, but brain is floating in some crazy juice and showing the effects. So, typing this up with eyes closed — hurts too much to have them open, but kinda hard to review. That’s ok, I will not post this.

This is my Mozilla blog. That doesn’t seem good. Delirium sould probably be kept on personal blog. Oh well, doesn’t matter, I will not post. So here, Mozilla related stuff, ok?

Man, it’s weird having autocorrecting fingers. I type things out, hitting backspce every 5th character or so, before my brain has a chance to realize the I screwed up.

Mozilla… FOTN seemed a joke when I first looked at it. Took me a while to realize that there was no depth testing going on, which made some weird impossible figures. Made me worry for the fate of WebGL if our demos are that crappy. Fortunately, tried on laptop and it was fine. Really needed some depth of field stuff still, but at least it was a totally reasonable flythrough. Maybe some shadowing too; those buildings were hard to pick apart. Textures weren’t the greatest, but what do I expect? I can’t do WebGL shit. OpenGL a little bit, though I still hurt over never getting those bitangents right. That’s right, bitch: bitangent, not binormal. Thanks for the needless learning curve.

I will not send this. Thou shalt not say “bitch” in a mozilla blog.

What’s up with memory tracking? Here’s what I did before: capture all new/delete/malloc/free/whatever. At end of run, or whenever, scan through all memory blocks, finding pointers to other blocks. Make a big graph. So far, I think this matches up with what njn (Nethercote — awesome name. Reminds me of netherworld. Maybe with a hat on the o to make it coast. The demonic shore.) where was I? Eyes closed not so good. njn has plans for something ilke this. When I did this last time, though, one of the most useful things was not dumping out summaries starting at… er… dominators? Fancy term for nothing. What’s more useful is to dig out a pointer you care about, then have an interactive shell or whatever to say “what does this pointer point to and what points to it?” Then you can take a little hike through the graph, assuming you have callsites and context for every memory block (page loaded or whatever.)

Head still hurts with eyes closed.

GC. Ok, I don’t know what people have gotten done, but there’s no way our system is a real base for interactive graphics. GC pauses kinda suck when you’re surfing, but tiny tiny pauses really really suck when you’re animating. And I kinda doubt our periodic callbacks have what we need to do real simulations. Sure, you can check the current time and advance appropriately, but people rewrite game engines over and over again to tweak timing. What’s good for the browser is not good for a game/simulation.

Still think we’ll need per-process GC policy. You can’t just delay GC until your animation is done. Can’t can’t can’t. It’s a big project, and saying “look! see? we can draw 3d models!” isn’t really going to take you down that road.

Heh. But what do I know? I rede buks n’ randomidiotontheweb advice. I haven’t done it either.

Man, the whole thing with Shaver and his shit-covered glasses is depressing. It reminds me of my own episodes. One day, everything’s fine, the next everything’s shit, and you realize that both are totally valid ways of looking at the world.

Will not post.

Delirium. Squeeze it while it lasts. Dear lazyweb: what’s a good gift for a one year old? Never mind, wrong lazyweb. My brain said red bicycle with built-in shower fixture. Probably not the best choice.

EVA. Isn’t that extravehicular a-something? Something about a spacewalk? Can’t know with closed eyes.

Opened eyes. Still typing into correct form. Got paranoid there. Might be better if I didn’t though. Not gonna send.

What’s up with our developer tools? Are we making the best possible console and the best possible inspector and the best possible debugger and they’ll be like an apple avocado saldad with some things soft and others hard and lumpy? Firebug at least pulls it all together. Somebody please tell me there’s a plan?

Hm. I like apple and avocado salad. Add yogurt, raisins, sunflower seeds. Weird, but it reallly works.

Weird. Wierd. My brother has been an ace speller all his life, and he still doesn’t get that right.

I will not post this.

Will I ever finish Taras’s slowcalls thing? I have no idea why I never manage to get back to that thing. It’s close, too, just need to find where in browser-land the other contexts are getting created. FunctionTimer going multithreaded isn’t going to help my progress, though, but I knew that when I saw the thread handling for the Timeline service and the lack in FunctionTimer.

I guess I just don’t understand it. It reports a random assortment of events. Is that useful? Dunno, I’m not the audience and I don’t understand my audience. Damn it.

Why oh why does working on Windows suck so bad? It’s really not a bad OS. Compiles would probably go way *faster* there if we didn’t go through the Unix translation layer, though I’m no happier about leaving my comfortable unixy environment. I think I may have fallen off the end of that sentence. Eyes hurt too much to check.

Just please no Ant. Ant is a great idea, in the wrong language, with the wrong assumptions, with the deeply wrong Java environment mucking everything up. Shoving bamboo chopsticks under fingernails? Sure. Ant? Pleasegodpleasegodno.

Global dependency graph good. Make the ipdl BS spit out its dependencies first, though, please.

Do I like the open office setup at Moz? Still not sure. Awful at first; too many interesting conversations on all sides, plus IRC to contend with. Really tough for my old brain. Man, that sucks. Somehow, I’m still thinking I’m the young kid, sailing ahead faster than anyone expects. But nope, now I’m OLD, want to admit it or not. I still know a little about a lot, but it’s not coming together anymore like it used to. I got lazy. Lazy. Lazy. Hindsight’s a bitch.

There’s that word again. No post while under the influence. My head hurts.

Why are we all rewriting the same tools? Speedtracer is great. Why did you have to screw it up by writing it in Java? Writing in Java is a penance, not an advantage.

Not that I’ll defend Javascript. Scoping is broken, dammit! I don’t want my arrays to be objects! You’re close — can’t you please please give me real continuations and coroutines? And all those ‘function (a,b) {…}’ things. For a fundamental language feature, that’s just ay too verbose! Can’t it just be ‘(a,b)->{…} or something? Maybe ‘{(a,b):…}’? I guess that’s my Perl roots showing through, but Perl is still a language where you can have an idea, type it in, and run it. It works. Yes, it has enormous warts, but it got a lot of stuff right. Python is great, but I always feel like there’s a nun hitting my knuckles with a ruler when I use it.

Oh. And iterating over stuff. ARGH!

Whee. Head is spinning. Delirium starting to fade. Last chance to do something stupid!

Can’t we all just get along? What’s the deal with perf vs systemtap? Andrew Sutherland does cool shit, even if his viualizations often don’t speak to me. At least he has them, unlike my hairballs of solid text like my ETW posts. There are just so, so many low-hanging fruit in the performance area, and there really isn’t that big of a difference between what everybody needs. Ok, there are half a dozen different environments that are really, really different, but why don’t we have way more joint work and standardization here? I want to know why my app stutters every 24 seconds. Game programmers want to know why their game stutters every 24 seconds. The state of the art of profilers is craptastic, and they’ve had the basic same raw data to draw from for decades.

Related though, sorta. it’s not about throughput. I want to know how often we make the user wait, for how long, and how often. If the user is reading the text, GC is free. Free! Zero cost. As long as we’re not still digging through our short and curlies when the user wants to scroll or switch tabs or whatever. So why don’t we have a notion of when the user is waiting? I don’t care if you have 78 GC pauses with a mean of 726ms and a max… whatever. That could be great or awful. User waited? Awful. User was off in another window? Great, as long as there isn’t a metric buttload of garbage awaiting him when he gets back.

Head still hurts. Legs better.

Hey, if I publish now nobody will read it. Gone by next week.

No. Will not publish.

1:30am. Probably not a good way to get well, except I’ve been in bed since 4pm. I should go read my book now. Kestrel, 3yo going on 4, was worried that I might get bored so he gave me “Baby Danced the Polka” to read in the middle of the night. Awesome kid. 2yo was just traumatized to see Dad moaning away in bed, and pissed off when I wouldn’t let him touch me.

Need heavyweight JS profiler. track every frickin op executed, with cycle counter before and after. JM+TM+interp. Add ‘em up, dump them into a shadow array the length of the bytecode. Sure, it’s slow, but gives you coverage and skewed relative timings of everything everything. Tells how many times each op was mjitted/traced/interped. Tie that to systemtap-generated log of all I/O. Roast lightly over the coals, smear with barbeque sauce. How do you spell that? Who cares, eyes are closed. Not gonna post.

Why no distcc for teh build slaves? Sure, it hurts scalability a little, but it gives latency. Latency is God. Just need to make sure distcc servers match exactly; I’ve been burned by 32bit vs 64bit and Fedora 13 vs 14. I suppose the network I/O might melt down the cluster. Mmmm… melty cluster. Marshmallows and chocolate. Ever notice that anything Trade Joe’s puts out — cereal, I mean — with “clusters” in the name or the picture is vile?

Will not post. Not not not not post.

What do web developers do? Not chew tobacco very much thee days, I guess. Seems to be dying out. I sorta kinda know from things like my never-finished audiobook ripper. It’s a great deal, checking out audiobooks from the public library. They just take too long to listen to before you have to return them.

Web developers. CSS, of course. Still think the selectors are kinda impoverished, but I suppsoe they’re hairy enough. Firebug does a good job of CSS, though somehow I always find myself doubting it even though it ends up being right. Wonder why that is.

Javascript. Often using toolkit, these days. jQuery “operate on all these” is awesome. Mootools actually sounds cooler, more what I like — solid underpinnings, build the decorations on top. But it’s toast, you can’t compete with something that just hammers front end front end front end unless it falls apart. And jQuery seems fairly solid.

Wait, back up. Javascript. Debuggers. All JS is probably written with web searches going on like crazy. Write some code, web search for how to do next piece, write some code, repeat. Step through with debugger. Get confused about what the CGI/server is doing and what the client is doing. Need record and replay capabilities.

HTML, CSS layout. Would this be better as a visual task? Gives me the creeps. But yeah, you’re making a visual output, so something’s gotta be visual.

Events. Whee, that’s a big one. What’s the event what’s the keycode where does it go? How should the user see it?

Mice. Nasty scurrying little things. 50 of them at once is just disgusting. One at a time is… well, kinda cute. Cockroaches not so much.

Really need to go to bed. Bed might be made out of plywood, but probably softer. No money in it. Even with today’s negligible interest rates. Should stockpile raisins instead. They keep, and you can eat them. How did money get started anyway? I give you a fish, you give me a piece of paper. I trade the piece of paper to someone else who wants one of your newborn puppies, he gives me some tulip bulbs in return. Hey asshole, who said you could give me that piece of paper? Why can’t you just give paper for everything, and if someone tries to redeem, just give them more paper?

Eyes are open. 2am. Time for bed.

update: yes, I posted this. But I control whether things make it to Planet via a tag, and I didn’t set it for this. That seemed like a comfortable middle ground at the time I wrote this post. I maybe should re-read it to decide whether to nuke it entirely. Fortunately, nobody reads my blog directly.

ETW: Event Tracing for Windows, part 2: field reporting

sfink    Uncategorized     etw, mozilla, planet, profiling, windows

(See all of my ETW posts)

One use of event tracing is gathering field reports: the user notices something going horribly wrong, and is willing to help out by providing useful information. ETW is very valuable for this use, since 90% of Firefox users are on Windows and ETW logging comes standard with Windows. So here’s a guide to submitting useful trace info. If you are reporting a performance-related bug, it would probably be useful if you could offer up a .etl file with your bug report. (But you probably shouldn’t directly attach it to the bug if it’s big, and it probably will be.)

I’m going to give the “universal” option first, which will work on Windows XP on up with no additional installs needed:

  logman start "NT Kernel Logger" -o browser.etl -p "Windows Kernel Trace" 0x237f -bs 1024 -nb 10 100 -ets
  run your test scenario...
  logman stop "NT Kernel Logger" -ets

This will produce a file “browser.etl” containing all kinds of system-level tracing data. If you’re uploading it somewhere, you might want to compress it first. These files compress pretty well. I’ll explain what all the wacky options mean in a later post.

The other option is to use xperf to control the tracing. It is part of the Windows Performance Toolkit (WPT), which itself ships with the Windows 7 SDK. You do not need to install the entire SDK to get WPT! (The SDK is pretty huge.) So if you are running Vista or Windows 7 (or later, presumably):

1. Download and run the Windows 7 SDK (click on the Download button).
2. Select just the Windows Performance Toolkit components
3. Install!

Here’s the important part where you select just the component you need, stolen shamelessly from a very good blog post giving precise instructions for doing exactly this.

If you are running the 64-bit version of Vista or Windows 7, you’ll need to do one more thing to get full functionality: run

REG ADD "HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management" -v DisablePagingExecutive -d 0x1 -t REG_DWORD -f

This will allow you to collect stack traces.

If you are running Windows XP, it’s a little more work. You can do it, but you will need to copy things from a machine with WPT already installed. Make sure you are copying from the same architecture (64-bit binaries won’t work on 32-bit Windows XP). Copy over just the files xperf.exe and perfctrl.dll into somewhere in your $PATH. I would link to a copy of those files if I knew it was legal. On your Windows 7 machine with the WPT installed, it will probably be somewhere like C:\Program Files\Microsoft Windows Performance Toolkit\xperf.exe and you can copy it to C:\Windows on your XP machine.

Now you can run it:

  xperf -on DiagEasy -stackwalk profile
  run your scenario...
  xperf -stop -d browser.etl

This will gather more or less the same information as the logman command above, except it’ll also grab stacktraces during the profile events. I think it may even cram some additional xperf-specific details into the trace log, which make it easier to analyze the file on another machine according to the ntdebugging guy(s?).

Either way, link to this .etl file in a bug report and it will be much easier to make sense of what’s going wrong. The xperf-produced .etl files are preferred, since they supposedly have more goodies in them, but if installing xperf is too much trouble then the logman-generated ones are still quite handy.

If you want to see what you’ve collected, run

  xperf browser.etl

I’ll talk about how to interpret what you see later. This article is all about capturing traces, not analyzing them.

So what are those commands doing? To really answer this question, I’d need to go into the details of the ETW architecture, which will be a followup blog post. So I’ll just give a short version for now:

The logman command starts up a tracing session aka logger using the predefined name “NT Kernel Logger”, and connects it up to receive events from the provider named “Windows Kernel Trace”. The 0x237f option is a bitmask of what flags aka keywords to include in the trace.

  logman query providers "Windows Kernel Trace"

shows the list of available keyword bits for that provider and the descriptions of what they cover. The trace session is logged to the file browser.etl, and when the session is stopped all final events are flushed to it. The buffer settings are also configured to something reasonable.

The xperf command starts up the same logger with the DiagEasy group of keywords, which turns out to be the same as the 0x237f from the logman command. The log file is not specified, so it will default to \kernel.etl but we’re not going to directly use that file this time. Then the xperf -d command stops the logger and merges its output into browser.etl. According to the help message, it also adds “image identification information required for safe symbol decoding”, which sounds like a good thing.

ETW: Event Tracing for Windows, Part 1: Intro

sfink    Uncategorized     etw, mozilla, planet, profiling, windows

(See all of my ETW posts)

A while back, Andreas Gal asked me to add some ETW (Event Tracing for Windows) functionality into SpiderMonkey, the Mozilla Javascript engine. Though I’m anything but a Windows guy, it seemed straightforward enough. And actually, implementing it wasn’t too bad. Figuring out what the heck I had just implemented, how it worked, and how to use it is another story. Or stories. I will be writing a series of blog posts about the various aspects I have discovered.

Note: you may want to skip ahead to later parts in this series. This one is going to be high-level and whiny. Later installments should be more technical and HOWTOish.

First, some context. ETW (Event Tracing for Windows) was introduced in Windows 2000 as a single API for handling a grab bag of tasks, which all sound about the same but turn out to require wildly different implementations. The “supported” tasks include:

• a developer (of an application, the kernel, or a driver) inserting logging statements for personal use
• alerts for administrators
• low-overhead offline performance profiling (start up your scenario, generate massive log files, stop the logging, chew through them and produce reports and visualizations)
• realtime performance monitoring and analysis (watch the pretty CPU usage graph go up and down, or “Augh! Nothing is responding! Wtf is my stupid computer doing right now?!”)
• using a separate monitoring machine to suck down data from a different machine to do any of the above
• tracking end-to-end performance of requests traversing multiple applications and servers
• tracing the exact behavior (files opened, disk offset patterns, …) of some application
• capturing problem reports from the field
• doing everything from either command line or a GUI

which is a lot to stuff into one architecture. The result is inconsistent and tangled, but more or less functional. (More for some uses, less for others.)

The GUIs are rather limited: some parts are just silly and don’t need to exist, other parts have really great stuff but they’re hard to find. They feel like early prototypes. The connection between what you’re capturing and what you end up seeing is cryptic. The GUIs I have discovered so far for viewing trace results are Xperfview, Event Viewer, Sawbuck (a Google project), and RPM (Reliability and Performance Monitor aka Perfmon). There are also GUIs for configuring trace capture.

The command line tools are a mess. Multiple generations are hanging around. You are suggested to only use the latest stuff (xperf), but the latest stuff requires a separate install and so is much less useful for field report capturing (hint: if you can give people something to cut & paste, they’ll grumble but do it and get it right. If you have to walk them through a GUI, they’ll be happy but probably get something wrong. If you tell them to go install something and then do either of the above, you won’t hear back from any but the most motivated.) The different generations have largely overlapping functionality, but completely different syntax — and most confusingly, terminology.

On the other hand, the underlying implementation of the trace recording seems nice. It can log straight to files, log into buffers, do circular logging, and trade off overhead/completeness/reliability (eg you can capture to per-CPU buffers to avoid locking overhead and thus miss fewer events, but things may get out of order). It has a bunch of tuning knobs that semantically make sense. The GUIs and command-line tools for controlling them, on the other hand, do not. You have great flexibility in dynamically enabling and disabling providers, updating parameters, starting/stopping/flushing logging sessions, associating providers with sessions, etc.

There is decent support for implementing your own providers, which is what I originally set out to do. It feels a lot more complex than it really ought to be, but all in all it really isn’t that bad. Well, ok, I say that but I still haven’t completely figured out how to use all of the bits that are generated from my ETW manifest. But that’s more of a tooling issue — the different tools use totally different mechanisms for making use of the metadata in the ETW manifest, and some tools just can’t do anything with it. (That unfortunately includes the latest greatest visualization tool, xperfview.)

Navigating through the maze of tools, options, and documentation is insanely hard from my naive point of view. Is this how things are in the Windows ecosystem? I’ve read through dozens of whitepapers, official-ish manuals, blog posts, forum threads, etc., and I feel like each one I read brushes the dirt off of one more spot in the picture, but piles more dirt back on the rest. Nothing is complete or even gives a complete high-level overview. There are several that try to, some of them very well-written and polished, but they still seem to have a lot of wishful thinking: “if you just switch over to this way of doing things, then it all works like this, so be happy” not mentioning that their chosen way of doing things is incomplete and you’ll end up digging through the older tools and getting completely lost as you try to accomplish the one last little piece you need. There is little user-facing consistency.

Which is not to say that I’m going to do any better, but I’ll try to stick to some very constrained usage scenarios.