Got some helpful comments on my previous post.
madvise(WILLNEED) in ld.so
Frank Ch. Eigler pointed out that other people have noticed the madvise/mmap deficiency in the dynamic linker. Unfortunately those unfortunate people did not have the ability to flush caches or to measure page faults exactly. Linux has gotten a lot nicer for diagnosing this. Frank also filed a glibc bug. SuSE already ships with a madvise()ed dynamic linker.
Both Bradley Baetz and Frank pointed me at the FIPS-caused bug which causes prelink to get angry at Firefox. This isn’t the first time that practically-useless FIPS has gotten in the way of a higher performing Firefox.
The most common response to people who complain about run-time linker issue is “are you using prelink?”. So I did my best to assume that prelink would erase a bunch of the overhead I saw in the previous bug.
I got rid of the offending nss rule in the prelink configuration as the bug instructed; prelink started working on Firefox (according to LD_DEBUG=statistics). Unfortunately the IO pattern was identical to what I was seeing before.
Frank suggested that I try prelink -u to undo any prelinking. This caused a bunch more runtime-linker IO from which one can conclude that the overhead I am seeing is not solved by prelink. The is still a lot of IO that happens before any application code is executed, so the runtime linker is still my primary suspect.
Going to spend some quality time with SystemTap to try to figure out what the linker is doing.