The JS parser API has landed

Today I pushed a patch to our development repository to expose the JavaScript Parser API to code with chrome privileges. I’ve fixed all known major bugs, and it’s been holding up under sustained attack from Jesse Ruderman‘s fuzz tester without crashing for the better part of a year. So hopefully this means the landing will stick. If so, then this should land in Nightly within the week and Aurora by next week. If all goes well, the library will ship with Firefox 7.

Once it ships, addons will be able to write:

> Components.utils.import("resource://gre/modules/reflect.jsm");
> var expr = Reflect.parse("obj.foo + 42").body[0].expression
> expr.left.property.name
"foo"
> expr.right.value
42

I hope people find this useful for introspecting on code, building development tools, or writing static analyses. I also hope people come up with cool new applications I haven’t even thought of.

12 Responses to The JS parser API has landed

  1. Colby Russell

    Hell yes.

  2. This looks quite interesting for Komodo IDE (as we are based on the Mozilla code base), I’d like to use something like this for generating JavaScript code information, to be then used for Komodo’s code intelligence (aka completions and calltips).

    One thing that always seems missing from these types of tools is the JavaScript comments themselves:

    // This variable holds the users custom configuration details
    var config = {};

    Do (or can) these types of comments show up when running Reflect.parse?

  3. √ėyvind Sean Kinsey

    Is the AST documented somewhere? There are now a lot of tools based on parsers (eg. JSHint, minifiers etc), and if these could hook into such an API, then that could potentially lead to a lot of cool stuff.

  4. Yes, I linked to the API docs in the post. They’re at

    https://developer.mozilla.org/en/SpiderMonkey/Parser_API

    Dave

  5. Ridiculously cool.

  6. This is awesome. Any reasons it’s only exposed to code with chrome privilege?

  7. @Todd: I’m afraid since this just exposes the parse tree produced by SpiderMonkey, the comments have already been dropped.

    Dave

  8. Just being careful. Once we get a little more user testing through the chrome version, I intend to make it available to web content.

    Dave

  9. Mike Samuel

    With this can that something like hygienic macros ( http://wiki.ecmascript.org/doku.php?id=strawman:quasis-alt ) be built in library code using an ES5 module loader with an eval hook?

  10. Hi Mike,

    First, a nit: the quasis proposal isn’t really macros. Macros expand at compile-time. Also, macros are meant to enable programmers to extend a language’s syntax with new constructs that look like ordinary constructs in the base language. When your syntax always involves explicit quoting, it’s not quite the same thing. That may sound trivial, but there are big practical differences. (For example, you couldn’t use quasis to implement for-loops in terms of while loops, because they aren’t syntactically powerful enough to extend the parser with the for-loop syntax.)

    Module loaders aren’t in ES5 but they’re in Harmony for ES.next; they do indeed give you the ability to extend the language. However, this parser API is not extensible. It just gives you access to the fixed SpiderMonkey parser. So there’s no way — currently, anyway — that you could use it to parse JS + new syntax.

    I’m interested in experimenting with macros for JavaScript, but we’d still have to extend the core parser. At that point, the JS parser API could potentially be useful, but it might not really be necessary. Once we’re building support for macros into the language, we’d probably go all the way to supporting macros as part of the language, rather than requiring the programmer to roll their own.

    Dave

  11. Mike Samuel

    Dave, while I agree that macros typically depend on “compile-time” modification, I think it’s a silly definition. Defined strictly, since JavaScript does not mandate a compile-step there is no compile-time, so macros are impossible for the language. You could replace “compile-time” with “module-load-time”, but that seems an arbitrary choice of replacement.

    The key feature that people want when they ask for a macro systems seems to be the ability to have the program modify its own parse tree to efficiently specify new constructs. Whether the macros are expanded at compile-time or just-in-time seems irrelevant to those key use cases. And the proposal above describes just such a just-in-time macro expansion scheme.

    I think you’re right that experimentation is the goal, but, with module-eval hooks, why are core-parser extensions necessary?

  12. Mike, macro systems have decades of history, and I’m not being pedantic when I say that the phase distinction is a fundamental part of that history. It’s not silly when you’re implementing an in-language parser generator to generate the parser tables before running the program. It’s not silly when you’re implementing a statically typed sub-language or DSL and the types get checked at compile-time. Or a static analysis, or a partial evaluation, or any number of other things where you want to do a computation before the rest of the program starts running.

    There’s also research showing all sorts of pitfalls that happen when you do runtime versions of macros: Lisp’s “fexprs” were a dynamic analog of macros, and they are problematic for efficient compilation (somewhat similarly to ‘with’ in JS). See for example, Mitch Wand’s “The Theory of Fexprs is Trivial” or Kent Pitman’s “Special Forms in Lisp.” There’s current research going on to try to figure out how to do runtime versions of macros that don’t violate static scoping, such as John Shutt’s work at WPI. But they’re very new, the research is ongoing, and there’s pretty much no work yet on how to implement them effectively in language runtimes or how to program with them effectively. And you give up on all the practical applications of the phase separation such as the ones I described above.

    But moreover, as I said in my last comment, there are real differences between macros that can be made to look “native” in their host language, and syntax extensions that are always isolated in an explicitly quoted context. The former makes it possible to truly extend the language, to create syntactic abstractions in the host language itself. That is traditionally what macros are. The latter is something else, something akin to printf, where you embed a DSL in a string literal and write an interpreter to evaluate it at runtime. As I said before, they’re not unrelated to macros, but they’re not the same thing.

    Dave