Feb 11

Fun and Games With gdb

As with most developers, I have a love/hate relationship with gdb, and have built up a clunky set of tips, tricks, and workarounds that I commonly use. I thought I’d share some of mine here, in a typically disorganized fashion:

Breakpoint Command Lists

This may be well known to everyone but me, but gdb lets you run arbitrary canned expressions every time a breakpoint is hit. This is incredibly useful when you need to know the last time something changed before a crash or some other detectable event. The usual problem is that if you set a breakpoint or watchpoint or whatever, you’ll hit it over and over again before the “interesting” one happens, and it’ll take way too long and you’ll give up in disgust. So instead, try something like this:

  (gdb) b somefilename.cpp:1234
  Breakpoint 2 at ...
  (gdb) command 2
  > bt
  > cont
  > end
  (gdb) cont

Now, gdb will run normally, except whenever that breakpoint is hit it’ll dump out the full stack and then continue. When you crash or hit your other breakpoint or whatever, you can look backwards to the previous time you hit breakpoint 2 (hardware watchpoints are especially good with this) and see what the stack was then. If the breakpoint is hitting a lot, this may not be fast, but you can leave it running unattended all night if you have to.

I’ve tried getting clever with this, with mixed results: once, I wanted to know when a particular value changed. I could set a hardware watchpoint, but it was constantly getting changed by a function f() that I didn’t care about. It was some other mysterious source of mutation that was tripping me up. So I set the hardware watchpoint, and then set breakpoints on the beginning and end of f(). I then created a command list for the breakpoint on the start of f() that disabled the watchpoint and continued, and a command list for the breakpoint on the end of f() to re-enable the watchpoint.

Unfortunately, it didn’t work. I still don’t know why. Maybe there’s something weird about watchpoints. I didn’t have time to dig into it then. But you get the idea. (Hmm… if it was having trouble with enable/disable, perhaps I should have set and cleared a convenience variable $active and made the watchpoint conditional on that…)

Logging From gdb

Occasionally, I have some big wad of data that needs to be interpreted by a different program. Let’s say it’s in a char* variable with its length stored in another variable. My usual approach is to print out the contents using the ‘x’ or ‘p *foo @ 100’ or whatever, then use emacs or Perl to unescape. But for large wads of data, this is unworkable. (For example, I’ve done this with entire images.)

So write it out to a file instead:

  (gdb) p creat("/tmp/somefile.dat", 0777)
  $26 = 37
  (gdb) p write(37, s, len)
  $27 = 168
  (gdb) p close(37)
  $28 = 0

…and now you have a handy little file containing just your data and nothing else.

In conjunction with the previous trick, you can append wads of binary data to a trace file every time you hit a breakpoint. Fun!


This may be outdated. gdb has a nice watchpoint facility, but it has a tendency to take you too literally — when you ask it to watch a particular expression, it often sets a software watchpoint on that expression, meaning it constantly re-evaluates the expression over and over and slows execution down to a crawl. If you know that the various pointers involved aren’t going to change (or if they change, you want the current one anyway), it’s often handy to take the address of the expression you want to watch and then watch the dereference of that:

  (gdb) p &fun->atom
  $27 = (JSAtom **) 0x7fffe3b6b778
  (gdb) watch *$27
  Hardware watchpoint 5: *$27

This doesn’t seem to be as necessary as it used to; it seems to do hardware watchpoints automatically in many more cases. But it still complains when the expression is no longer valid.

I notice (while writing this) that there is now a -l option for doing something like this automatically. I’ve never tried it, but it sounds like it’d do what I want.

Automagical debugging

I nearly always run gdb within emacs so it automatically loads and highlights the current line of code for me (and gives history, etc.) But say I want to run

  myprog -s -t lamppost -- monkey walrus -teeth-

I have to:

  • run emacs
  • M-x gdb
  • type “myprog”<enter>
  • type “run -s -t lamppost — monkey walrus -teeth-“

…and that means retyping it all or cutting & pasting, even though I probably just ran that command. For running Firefox under the debugger, it’s even worse, because on Linux ‘firefox’ is actually a shell script that sets up the environment and then invokes ‘firefox-bin’. There, you have to use the ‘-g’ option on the shell script to do the right thing. I also want to start up the Perl debugger under emacs for Perl scripts, and (not that you care) I used to work on a program at a previous company that had an embedded scripting engine, and I sometimes wanted to run gdb on the C++ generated binary, and sometimes the script debugger on the embedded scripts.

Phew. So I wrapped it all up in a ‘debug’ script that guesses what you want and twiddles the options to emacs, gdb, Perl, and/or the Firefox wrapper script to do the right thing. Usage:

  debug firefox -no-remote -P test about:blank

That will bring up an emacs window running gdb on firefox-bin with the environment all set up automagically. (It makes some extreme assumptions, such as guessing that if you give it a shell script then it’s the firefox wrapper shell script so it can use the -g and -d options.) It is surprisingly robust; if you use ‘debug’ in place of ‘gdb’ in commands that accept a debugger, it’ll often just work. For example, here’s a way to run a single directory’s worth of mochitests:

  cd obj/_tests/testing/mochitest
  python runtests.py --test-path=js/jsd/test

But it turns out runtests.py has a –debugger argument, so you can do:

  python runtests.py --test-path=js/jsd/test --debugger=gdb

and it’ll run the browser under gdb and give you a command-line prompt. So, to make it nicer:

  python runtests.py --test-path=js/jsd/test --debugger=debug

Voilà! You get emacs running gdb running firefox running your mochitests.

I just tossed a snapshot of the script onto http://people.mozilla.org/~sfink/uploads/debug. It has some of my environment hardcoded in it, but it should probably work for you unmodified anyway. (The main thing I have to keep tweaking is the –annotate option to gdb. Either gdb or emacs changed somewhat recently.)

Conditionals with strings

Say you want to set a breakpoint on a certain line when script->filename is your script, perhaps “http://lifewithducks.com/cheesetasting/reformat.js”. You could do

  cond 7 strcmp(script->filename, "http://lifewithducks.com/cheesetasting/reformat.js") == 0

but that’s (1) a lot of typing, and (2) slow because that breakpoint may get hit a lot and gdb has to do some ptrace-y context dance to invoke expressions every time. For the typing problem, strstr() is your friend:

  cond 7 strstr(script->filename, "reformat")

Specifically, strstr() returns the location of a substring within a larger string, or NULL if it doesn’t find it. So you just need to call it with a unique snippet that you’re looking for, and it’ll Do The Right Thing.

For the “slow” problem, er… well, that example is going to be really messy. Let’s simplify and pretend you’re interested in breaking when strcmp(s, “spatula”) == 0. Here’s my evil shortcut:

  cond 7 *(int*)s == *(int*)"spatula"

gdb seems to be able to handle that expression much, much faster. Ok, that sort of assumes that your ints are 4 bytes. But again assuming 4-byte ints, it’s equivalent to

  cond 7 *(int*)s == *(int*)"spat"

because it’s treating the literal “spatula” as an address of a character array, and *(int*)”spatula” is thus giving the 1st 4 bytes interpreted as an integer. Here, that’s 1952542835 aka 0x74617073 ({ ‘t’, ‘a’, ‘p’, ‘s’ }), because I’m little-endian. If you want more characters and you’re on 64-bit, cast to (long*) instead. Or use multiple pieces:

  cond 7 ((int*)s)[0] == ((int*)"spatulamurders")[0] && ((int*)s)[1] == ((int*)"spatulamurders")[1] && ...

Obviously, this is really not helping the typing problem. I didn’t say I’d solve both at once, ok?

Back to the original example, you can use the latter trick to pick out the unique piece of the string you’re interested in. Let’s say that “reformat.js” is what we really care about, we’re on 64-bit, and we’ll ignore the string-length problem:

  cond 7 ((long*)script->filename)[5] == *(long*)"eformat."

I don’t use this often, but it’s been useful when I’ve needed it.

Mozilla Customizations

The Mozilla source uses a bunch of datatypes that gdb doesn’t do very well with, from 16-bit characters in strings to big nasty structures that are mostly not interesting but clutter up the output. I use a modified version of this .gdbinit. I also highly, highly recommend Jim Blandy’s archer-mozilla gdb extensions written in Python. I’d actually like to stop using that .gdbinit, since it’s written directly in gdb’s “scripting” language, and is painfully slow for printing strings. It should be rewritten with the Python stuff.

When I was working on compartment-related problems, I took a stab at modifying jimb’s stuff to automatically display the compartment for strings and to pretty-print compartments with their principals. It was pretty handy, though I doubt you’d want it to be active all the time. I should look into controlling it with an option or something, I guess. If you’re interested, you can find my changes at bitbucket (but note that jimb’s repo is the real repo; I just pushed to bitbucket so I could point to it from this blog entry.)

You may need to recompile gdb from the archer gdb sources for this stuff to work; I’m not sure. See jimb’s blog post for details.

How About You?

Those are all the ones I can think of for now. I’d be interested in hearing about other people’s tricks. It seems like everyone uses gdb a bit differently, so there’s a lot of opportunity for cross-pollination.