{"id":340,"date":"2014-10-30T16:47:58","date_gmt":"2014-10-30T20:47:58","guid":{"rendered":"http:\/\/blog.mozilla.org\/nfroyd\/?p=340"},"modified":"2014-10-30T16:47:58","modified_gmt":"2014-10-30T20:47:58","slug":"porting-rr-to-x86-64","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/nfroyd\/2014\/10\/30\/porting-rr-to-x86-64\/","title":{"rendered":"porting rr to x86-64"},"content":{"rendered":"

(TL;DR: rr from git can record and replay 64-bit programs.\u00a0 Try it for yourself<\/a>!)<\/p>\n

Over the last several months, I’ve been devoting an ever-increasing amount of my time to making rr<\/a> able to trace x86-64 programs.\u00a0 I’ve learned a lot along the way and thought I’d lay out all the major pieces of work that needed to be done to make this happen.<\/p>\n

Before explaining the major pieces, it will be helpful to define some terms: the host<\/em> architecture is the architecture that the rr binary itself is compiled for.\u00a0 The target<\/em> architecture is the architecture of the binary that rr is tracing.\u00a0 These are often equivalent, but not necessarily so: you could be tracing a 64-bit binary with a 64-bit rr (host == target), but then the program starts to run a 32-bit subprocess, which rr also begins to trace (host != target).\u00a0 And you have to handle both cases in a single rr session, with a single rr binary.\u00a0 (64-bit rr doesn’t handle the host != target case quite yet, but all the infrastructure is there.)<\/p>\n

All of the pieces described below are not new ideas: the major programs you use for development (compiler, linker, debugger, etc.) all have done some variation of what I describe below.\u00a0 However, it’s not every day that one takes a program written without any awareness of host\/target distinctions and endows it with the necessary awareness.<\/p>\n

Quite often, a program written exclusively for 32-bit hosts has issues when trying to compile for 64-bit hosts, and rr was no exception in this regard.\u00a0 Making the code 64-bit clean by fixing all the places that triggered compiler warnings on x86-64, but not on i386, was probably the easiest part of the whole porting effort.\u00a0 Format strings were a big part of this: writing %llx<\/tt> when you wanted to print a uint64_t<\/tt>, for instance, which assumes that uint64_t<\/tt> is implemented as unsigned long long<\/tt> (not necessarily true on 64-bit hosts).\u00a0 There were several places where long<\/tt> was used instead of uint32_t<\/tt>.\u00a0 And there were even places that triggered signed\/unsigned comparison warnings on 64-bit platforms only.\u00a0 (Exercise for the reader: construct code where this happens before looking at the solution<\/a>.)<\/p>\n

Once all the host issues are dealt with, removing all the places where rr assumed semantics or conventions of the x86 architecture was the next step.\u00a0 In short, all of the code assumed host == target: we were compiled on x86, so that must be the architecture of the program we’re debugging.\u00a0 How many places actually assumed this, though?\u00a0 Consider what the very simplified pseudo-code of the rr main recording loop looks like:<\/p>\n

while (true) {\r\n  wait for the tracee to make a syscall\r\n  grab the registers at the point of the syscall\r\n  extract the syscall number from the registers (1)\r\n  switch (syscall_number) {\r\n    case SYS_read: (2)\r\n      extract pointer to the data read from the registers (3)\r\n      record contents of data\r\n      break;\r\n    case SYS_clock_gettime:\r\n      extract pointers for argument structures from the registers\r\n      record contents of those argument structures (4)\r\n      break;\r\n    case SYS_mmap: (5)\r\n      ...\r\n    case SYS_mmap2: (6)\r\n      ...\r\n    case SYS_clone: (7)\r\n      ...\r\n    ...\r\n    default:\r\n      complain about an unhandled syscall\r\n  }\r\n  let the tracee resume\r\n}\r\n<\/pre>\n
Every line marked with a number at the end indicates a different instance where host and target differences come into play and\/or the code might have assumed x86 semantics.\u00a0 (And the numbering above is not exhaustive!)\u00a0 Taking them in order:<\/p>\n
    \n
  1. You can obtain the registers of your target with a single ptrace<\/tt> call, but the layout of those registers depends on your target.\u00a0 ptrace<\/tt> returns the registers as a struct user_regs<\/tt>, which differs between targets; the syscall number location obviously differs between different layouts of struct user_regs<\/tt>.<\/li>\n
  2. The constant SYS_read<\/tt> refers to the syscall number for read on the host.\u00a0 If you want to identify the syscall number for the target, you’ll need to do something different.<\/li>\n
  3. This instance is a continuation of #1: syscall arguments are passed in different registers for each target, and the locations of those registers differ in size and location between different layouts of struct user_regs<\/tt>.<\/li>\n
  4. SYS_clock_gettime<\/tt> takes a pointer to a struct timespec<\/tt>.\u00a0 How much data should we read from that pointer for recording purposes?\u00a0 We can’t just use sizeof(struct timespec)<\/tt>, since that’s the size for the host, not the target.<\/li>\n
  5. Like SYS_read<\/tt>, SYS_mmap<\/tt> refers to the syscall number for mmap<\/tt> on the host, so we need to do something similar to SYS_read<\/tt> here.\u00a0 But just because two different architectures have a SYS_mmap<\/tt>, it doesn’t mean that the calling conventions for those syscalls at the kernel level are identical.\u00a0 (This distinction applies to several other syscalls as well.)\u00a0 SYS_mmap<\/tt> on x86 takes a single pointer argument, pointing to a structure that contains the syscall’s arguments.\u00a0 The x86-64 version takes its arguments in registers.\u00a0 We have to extract arguments appropriately for each calling convention.<\/li>\n
  6. SYS_mmap2<\/tt> only exists on x86; x86-64 has no such syscall.\u00a0 So we have to handle host-only syscalls or target-only syscalls in addition to things like SYS_read<\/tt>.<\/li>\n
  7. SYS_clone<\/tt> has four (!) different argument orderings at the kernel level, depending on the architecture, and x86 and x86-64 of course use different argument orderings.\u00a0 You must take these target differences into account when extracting arguments.\u00a0 SYS_clone<\/tt> implementations also differ in how they treat the tls<\/tt> parameter, and those differences have to be handled as well.<\/li>\n<\/ol>\n
    So, depending on the architecture of our target, we want to use different constants, different structures, and do different things depending on calling conventions or other semantic differences.<\/p>\n
    The approach rr uses is that the Registers<\/tt> of every rr Task<\/tt> (rr’s name for an operating system thread) has an architecture, along with a few other things like recorded events.\u00a0 Every structure for which the host\/target distinction matters has an arch()<\/tt> accessor.\u00a0 Additionally, we define some per-architecture classes.\u00a0 Each class contains definitions for important kernel types and structures, along with enumerations for syscalls and various constants.<\/p>\n
    Then we try to let C++ templates do most of the heavy lifting.\u00a0 In code, it looks something like this:<\/p>\n
    enum SupportedArch {\r\n  x86,\r\n  x86_64,\r\n};\r\n\r\nclass X86Arch {\r\n  \/* many typedefs, structures, enums, and constants defined... *\/\r\n};\r\n\r\nclass X64Arch {\r\n  \/* many typedefs, structures, enums, and constants defined... *\/\r\n};\r\n\r\n#define RR_ARCH_FUNCTION(f, arch, args...) \\\r\n  switch (arch) { \\\r\n    default: \\\r\n      assert(0 && \"Unknown architecture\"); \\\r\n    case x86: \\\r\n      return f<X86Arch>(args); \\\r\n    case x86_64: \\\r\n      return f<X64Arch>(args); \\\r\n  }\r\n\r\nclass Registers {\r\npublic:\r\n  SupportedArch arch() const { ... }\r\n\r\n  intptr_t syscallno() const {\r\n    switch (arch()) {\r\n      case x86:\r\n        return u.x86.eax;\r\n      case x86_64:\r\n        return u.x64.rax;\r\n    }\r\n  }\r\n\r\n  \/\/ And so on for argN accessors and so forth...\r\n\r\nprivate:\r\n  union RegisterUnion {\r\n    X86Arch::user_regs x86;\r\n    X64Arch::user_regs x64;\r\n  } u.\r\n};\r\n\r\ntemplate <typename Arch>\r\nstatic void process_syscall_arch(Task* t, int syscall_number) {\r\n  switch (syscall_number) {\r\n    case Arch::read:\r\n      remote_ptr buf = t->regs().arg2();\r\n      \/\/ do stuff with buf\r\n      break;\r\n    case Arch::clock_gettime:\r\n      \/\/ We ensure Arch::timespec is defined with the appropriate types so it\r\n      \/\/ is exactly the size |struct timespec| would be on the target arch.\r\n      remote_ptr tp = t->regs().arg2();\r\n      \/\/ do stuff with tp\r\n      break;\r\n    case Arch::mmap:\r\n      switch (Arch::mmap_argument_semantics) {\r\n        case Arch::MmapRegisterArguments:\r\n          \/\/ x86-64\r\n          break;\r\n        case Arch::MmapStructArguments:\r\n          \/\/ x86\r\n          break;\r\n      }\r\n      break;\r\n    case Arch::mmap2:\r\n      \/\/ Arch::mmap2 is always defined, but is a negative number on architectures\r\n      \/\/ where SYS_mmap2 isn't defined.\r\n      \/\/ do stuff\r\n      break;\r\n    case Arch::clone:\r\n      switch (Arch::clone_argument_ordering) {\r\n        case Arch::FlagsStackParentTLSChild:\r\n          \/\/ x86\r\n          break;\r\n        case Arch::FlagsStackParentChildTLS:\r\n          \/\/ x86-64\r\n          break;\r\n      }\r\n      break;\r\n    ...\r\n  }\r\n}\r\n\r\nvoid process_syscall(Task* t, int syscall_number) {\r\n  int syscall_number = t->regs().syscallno();\r\n  RR_ARCH_FUNCTION(process_syscall_arch, t->arch(), t, syscall_number);\r\n}\r\n<\/pre>\n
    The definitions of X86Arch<\/tt> and X64Arch<\/tt> also contain static_assert<\/tt>s to try and ensure that we’ve defined structures correctly for at least the host architecture.\u00a0 And even now the definitions of the structures aren’t completely bulletproof; I don’t think the X86Arch<\/tt> definitions of some structures are robust on a 64-bit host because of differences in structure field alignment between 32-bit and 64-bit, for instance.\u00a0 So that’s still something to fix in rr.<\/p>\n
    Templates handle the bulk of target-specific code in rr.\u00a0 There are a couple of places where we need to care about how the target implements mmap<\/tt> and other syscalls which aren’t amenable to templates (or, at least, we didn’t use them; it’s probably possible<\/em> to (ab)use templates for these purposes), and so we have code like:<\/p>\n
    Task* t = ...\r\nif (has_mmap2_syscall(t->arch())) {\r\n  \/\/ do something specifically for mmap2\r\n} else {\r\n  \/\/ do something with mmap\r\n}\r\n<\/pre>\n
    Finally, various bits of rr’s main binary and its testsuite are written in assembly, so of course those needed to be carefully ported over.<\/p>\n
    That’s all the major source-code related work that needed to be done. I’ll leave the target-specific runtime work required for a future post.<\/p>\n
    <\/a>x86-64 support for rr hasn’t been formally released, but the x86-64 support in the github repository<\/a> is functional: x86-64 rr passes all the tests in rr’s test suite and is able to record and replay Firefox mochitests.\u00a0 I will note that it’s not nearly as fast as the x86 version; progress is being made in improving performance, but we’re not quite there yet.<\/p>\n
    If you’re interested in trying 64-bit rr out, you’ll find the build and installation instructions<\/a> helpful, with one small modification: you need to add the command-line option -Dforce64bit=ON<\/tt> to any cmake<\/tt> invocations.\u00a0 Therefore, to build with Makefiles, one needs to do:<\/p>\n
    git clone https:\/\/github.com\/mozilla\/rr.git\r\nmkdir obj64\r\ncd obj64\r\ncmake -Dforce64bit=ON ..\/rr\r\nmake -j4\r\nmake check\r\n<\/pre>\n
    Once you’ve done that, the usage instructions<\/a> will likely be helpful.\u00a0 Please try it out and report bugs if you find any!<\/p>\n","protected":false},"excerpt":{"rendered":"
    (TL;DR: rr from git can record and replay 64-bit programs.\u00a0 Try it for yourself!) Over the last several months, I’ve been devoting an ever-increasing amount of my time to making rr able to trace x86-64 programs.\u00a0 I’ve learned a lot along the way and thought I’d lay out all the major pieces of work that […]<\/p>\n","protected":false},"author":320,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[72422,72423],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/posts\/340"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/users\/320"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/comments?post=340"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/posts\/340\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/media?parent=340"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/categories?post=340"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/nfroyd\/wp-json\/wp\/v2\/tags?post=340"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}