{"id":3496,"date":"2013-06-13T11:13:42","date_gmt":"2013-06-13T19:13:42","guid":{"rendered":"http:\/\/blog.mozilla.org\/webdev\/?p=3496"},"modified":"2013-09-30T14:29:08","modified_gmt":"2013-09-30T22:29:08","slug":"dxr-digests-the-firefox-codebase","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/","title":{"rendered":"DXR Digests the Firefox Codebase"},"content":{"rendered":"<p>How do you reason about a codebase the size of Firefox\u2019s? How do you figure out who calls the function you\u2019re messing with, without losing your context in an endless stream of false-positive-infested greps?<\/p>\n<p>You bust out DXR, Mozilla\u2019s code search and static analysis tool.<\/p>\n<p>DXR begins with the technology that powered the much-missed Google Code Search: <a href=\"https:\/\/www.youtube.com\/watch?v=lWKEphKIG8U&amp;feature=player_detailpage#t=499s\">inverted trigram indices<\/a> which turbo-charge a <a href=\"http:\/\/code.google.com\/p\/re2\/\">fast regex implementation<\/a>. It then adds a static analysis engine that rides sidecar with the C++ compiler, noting the location and relationship of code constructs. As a result, you can not only do regex searches that return as fast as you can type, but you can also ask \u201cWho subclasses this?\u201d \u201cWhere is this macro defined?\u201d and \u201cWhere are all the references to this type?\u201d<\/p>\n<p>Last week saw the first production push of DXR since its major redesign early this year. It now rides on a modern foundation of Flask and WSGI; blocks regressions with a looser-coupled, higher-coverage test harness; and delivers a glut of new features and fixes. Here are some highlights.<\/p>\n<h3>Indexing and Search<\/h3>\n<ul>\n<li>Faster, index-accelerated performance on many kinds of queries<\/li>\n<li>Querying for callers and callees<\/li>\n<li>Ability to find callers of constructors<\/li>\n<li>Recognition of typedefs<\/li>\n<li>Links to macro references<\/li>\n<li>Links from class declarations to class definitions<\/li>\n<li>Better detection of JS symbols<\/li>\n<li>Differentiation between a const and a non-const function of the same name<\/li>\n<li>Handy-dandy path breadcrumbs so you can tell what file you\u2019re reading and move around in the source tree<\/li>\n<li>Sorted folder listings<\/li>\n<li>First-class support for mobile and WebKit<\/li>\n<\/ul>\n<h3>Infrastructure<\/h3>\n<ul>\n<li>Several thousand lines less code<\/li>\n<li>Slimmer builds which consist of only data, not Python code, making development easier and deployment cleaner<\/li>\n<li>Replacement of our ad hoc process pooling implementation with Python 3\u2019s concurrent.futures (knocking off about 200 lines)<\/li>\n<li>Flask-powered WSGI instead of old-school CGI<\/li>\n<li>Redesigned URLs for MXR parity and future extensibility<\/li>\n<li>A Vagrant box to ease the spin-up of new contributors<\/li>\n<li><a href=\"https:\/\/wiki.mozilla.org\/Better_DXR_Testing\">A brand new test harness<\/a>, exercised by a Vagrant-based Jenkins job<\/li>\n<\/ul>\n<h3>Bug Fixes<\/h3>\n<ul>\n<li>Trouble-free entry of queries. (Bad JS! Go sit in the corner.)<\/li>\n<li>A fix for a crash when doing negated phrase matching<\/li>\n<li>Fixes for URL- and HTML-escaping bloopers, including shameful XSSes<\/li>\n<li>No more duplicate result lines in some types of queries<\/li>\n<\/ul>\n<h3>What\u2019s Next?<\/h3>\n<p>I am pleased to announce that the next release is scheduled for <i>one minute from now<\/i>. Yes, DXR is now under continuous deployment: if it passes tests, it goes up on the servers, eliminating human bottlenecks and errors and letting us deliver goodies\u2014and get feedback\u2014promptly. A great many UI and capability enhancements are on deck next, along with speedy new server hardware to make queries even snappier.<\/p>\n<p>Want to look under the hood of DXR? Check out <a href=\"https:\/\/wiki.mozilla.org\/DXR\">https:\/\/wiki.mozilla.org\/DXR<\/a>, and join us in #static on irc.mozilla.org. Happy hacking!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>How do you reason about a codebase the size of Firefox\u2019s? How do you figure out who calls the function you\u2019re messing with, without losing your context in an endless stream of false-positive-infested greps? You bust out DXR, Mozilla\u2019s code search and static analysis tool. <a class=\"go\" href=\"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/\">Continue reading<\/a><\/p>\n","protected":false},"author":213,"featured_media":3502,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[311,4712,288],"tags":[20275,20274,91,610,4719],"coauthors":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v22.5 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>DXR Digests the Firefox Codebase - Mozilla Web Development<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Erik Rose\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/\",\"url\":\"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/\",\"name\":\"DXR Digests the Firefox Codebase - Mozilla Web Development\",\"isPartOf\":{\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/blog.mozilla.org\/webdev\/files\/2013\/06\/Screen-Shot-2013-06-13-at-12.12.17-.png\",\"datePublished\":\"2013-06-13T19:13:42+00:00\",\"dateModified\":\"2013-09-30T22:29:08+00:00\",\"author\":{\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/#\/schema\/person\/e99c85edf86c46b46e5284384d5a7c12\"},\"breadcrumb\":{\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/#primaryimage\",\"url\":\"https:\/\/blog.mozilla.org\/webdev\/files\/2013\/06\/Screen-Shot-2013-06-13-at-12.12.17-.png\",\"contentUrl\":\"https:\/\/blog.mozilla.org\/webdev\/files\/2013\/06\/Screen-Shot-2013-06-13-at-12.12.17-.png\",\"width\":273,\"height\":209},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/blog.mozilla.org\/webdev\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"DXR Digests the Firefox Codebase\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/#website\",\"url\":\"https:\/\/blog.mozilla.org\/webdev\/\",\"name\":\"Mozilla Web Development\",\"description\":\"For make benefit of glorious tubes\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.mozilla.org\/webdev\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/#\/schema\/person\/e99c85edf86c46b46e5284384d5a7c12\",\"name\":\"Erik Rose\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.mozilla.org\/webdev\/#\/schema\/person\/image\/1c7953cea7e690a9e31cf08a4d68d829\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/73bfa51d6f44afed026160b59299faf2?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/73bfa51d6f44afed026160b59299faf2?s=96&d=mm&r=g\",\"caption\":\"Erik Rose\"},\"description\":\"Erik chips away at the barrier between human cognition and machine execution, through projects like DXR (search &amp; static analysis on Mozilla codebases), Fathom (semantic extraction from web pages), parsers, new languages, and a whole mess of Python libraries.\",\"sameAs\":[\"https:\/\/www.grinchcentral.com\/\",\"https:\/\/x.com\/ErikRose\"],\"url\":\"https:\/\/blog.mozilla.org\/webdev\/author\/erosemozilla-com\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"DXR Digests the Firefox Codebase - Mozilla Web Development","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/","twitter_misc":{"Written by":"Erik Rose","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/","url":"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/","name":"DXR Digests the Firefox Codebase - Mozilla Web Development","isPartOf":{"@id":"https:\/\/blog.mozilla.org\/webdev\/#website"},"primaryImageOfPage":{"@id":"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/#primaryimage"},"image":{"@id":"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/#primaryimage"},"thumbnailUrl":"https:\/\/blog.mozilla.org\/webdev\/files\/2013\/06\/Screen-Shot-2013-06-13-at-12.12.17-.png","datePublished":"2013-06-13T19:13:42+00:00","dateModified":"2013-09-30T22:29:08+00:00","author":{"@id":"https:\/\/blog.mozilla.org\/webdev\/#\/schema\/person\/e99c85edf86c46b46e5284384d5a7c12"},"breadcrumb":{"@id":"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/#primaryimage","url":"https:\/\/blog.mozilla.org\/webdev\/files\/2013\/06\/Screen-Shot-2013-06-13-at-12.12.17-.png","contentUrl":"https:\/\/blog.mozilla.org\/webdev\/files\/2013\/06\/Screen-Shot-2013-06-13-at-12.12.17-.png","width":273,"height":209},{"@type":"BreadcrumbList","@id":"https:\/\/blog.mozilla.org\/webdev\/2013\/06\/13\/dxr-digests-the-firefox-codebase\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/blog.mozilla.org\/webdev\/"},{"@type":"ListItem","position":2,"name":"DXR Digests the Firefox Codebase"}]},{"@type":"WebSite","@id":"https:\/\/blog.mozilla.org\/webdev\/#website","url":"https:\/\/blog.mozilla.org\/webdev\/","name":"Mozilla Web Development","description":"For make benefit of glorious tubes","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.mozilla.org\/webdev\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/blog.mozilla.org\/webdev\/#\/schema\/person\/e99c85edf86c46b46e5284384d5a7c12","name":"Erik Rose","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.mozilla.org\/webdev\/#\/schema\/person\/image\/1c7953cea7e690a9e31cf08a4d68d829","url":"https:\/\/secure.gravatar.com\/avatar\/73bfa51d6f44afed026160b59299faf2?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/73bfa51d6f44afed026160b59299faf2?s=96&d=mm&r=g","caption":"Erik Rose"},"description":"Erik chips away at the barrier between human cognition and machine execution, through projects like DXR (search &amp; static analysis on Mozilla codebases), Fathom (semantic extraction from web pages), parsers, new languages, and a whole mess of Python libraries.","sameAs":["https:\/\/www.grinchcentral.com\/","https:\/\/x.com\/ErikRose"],"url":"https:\/\/blog.mozilla.org\/webdev\/author\/erosemozilla-com\/"}]}},"jetpack_featured_media_url":"https:\/\/blog.mozilla.org\/webdev\/files\/2013\/06\/Screen-Shot-2013-06-13-at-12.12.17-.png","_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/posts\/3496"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/users\/213"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/comments?post=3496"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/posts\/3496\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/media\/3502"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/media?parent=3496"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/categories?post=3496"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/tags?post=3496"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/blog.mozilla.org\/webdev\/wp-json\/wp\/v2\/coauthors?post=3496"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}