Dec 07

Recent Progress

Looks like pork is slowly going to get merged back into oink. This makes me happy as it will result in decreased merging headaches and gives more visibility to my work outside of Mozilla. My elkhound changes are already in!

Recently I added support for retaining gnu attributes to elsa and corresponding features dehydra and garburator. Now dehydra can verify things based on attributes and  garburator gained a way to rewrite special cases like classes that are always allocated on the stack. Elsa still drops most attributes, but at least classes, methods and variable declarations are covered.

I also spent a couple of days investigating gcc plugins. Turns out modifying gcc to support plugins is dead easy, but getting anything useful done in GCC requires a steep learning curve. I tried to find how to enumerate all of the toplevel declarations in the source, but I couldn’t find the correct global variable that corresponds to the toplevel scope(aka the Translation Unit?). I have a few more ideas of what to try next. Once I do that, it shouldn’t take much work to make a basic gcc-hosted version of dehydra. There is also a gcc plugin branch hosted in the gcc svn, but I can’t find any example code for it. It isn’t a big deal since none of the plugins I’ve seen mentioned venture outside of intra-function analyses.

I am still pondering on how to tackle rewriting Mozilla to use exceptions. It is the key to improving overall readability/perf of Moz C++, but the logistics of writing the corresponding analyses+rewrites followed by a parallel manual correction step are still making my head spin. All I’m sure about is that the first step to exceptions would be to enable the OOM exceptions and do the corresponding exception safe analysis+rewrite.

Dec 07


Often there are two ways to write code. One way is to design an API and have code patterns adhere to how the API is supposed to be used. Another way is to rely on language features to accomplish the same thing. Typically API-pattern approaches are chosen because compilers are too immature or just don’t provide the necessary features. Sometimes compilers do catch up and the possibility of utilizing newer language features appears.

In the case of the exception rewrite the task is to rewrite code from a pattern-based (compiler in your head) approach to a more strict C++ construct-based exception paradigm. Unfortunately APIs don’t enforce their usage as much as a compiler (in part because we don’t have app-spefic compiler plugins) so transforming that into a strict compiler-friendly form automatically isn’t always realistic (example). See my previous post for more examples.

Having done more work on the exception conversion I believe that it is possible to switch Mozilla to exceptions to the point of getting it to compile. Unfortunately, I don’t think that it’s possible to do this in the Mozilla2 timeframe due to the large amount of manual labour required.

Due to various use cases that don’t fit the exception model there is a need for an nsresult-lint tool to detect funny (see above) nsresult patterns so code can be manually fixed to enable thrower to transform code correctly.

I expect conversion to exceptions to consist of the following large steps:

XPCOMGC -> nsresult-lint -> thrower automatic conversion -> nsexception-lint -> outparamdel

  1. XPCOMGC needs to land first to simplify memory management. Otherwise there will be a lot more nsCOMPtr<>s already_AddRefed<>s and friends.
  2. nsresult-lint would flag code for clean up to assist with the multitude of special cases in the code preventing it from transformation
  3. thrower needs to do some reasonably sophisticated static analysis (sensitive to control flow) to ensure that code is rewritten correctly. The analysis step isn’t ridiculously hard, but it is considerably more complex than what is done in existing tools.
  4. nsexception-lint tool will flag exception-unsafe code. I expect this to highlight a fair amount of code that needs to be converted to RAII. It will take more manual labour to fix flagged code here than in than step2.
  5. Once exceptions are used the return value is freed up for outparamdel to utilize. This will be a nice optimization and code clean up.

I think the best bet with exceptions would be to start working on them during the moz2 development cycle to have them land early in post-moz2.

Or as an alternative we could try to do just the OOM exceptions which are less frequent which would look like:

XPCOMGC -> thrower automatic conversion(OOM cases are easier) -> nsexception-lint

In this case the only significant piece of work is nsexception-lint which would be needed later for a full-blown exception rewrite. It wouldn’t be so bad to convert code to RAII even before that is required for the full exception rewrite.

For now I’m going let thrower rest in the pork hg repository while I try to make a static checker plugin for gcc.
Continue reading →

Nov 07

Garburator works!

A few weeks ago I convinced myself that is possible to rewrite Mozilla to avoid COMPtrs on the stack. Since then I’ve changed my mind a few times and felt like I may not be able to get this rewrite working. However, after three or four false starts, I finally managed to work out a metal model of the stack nsCOMPtr usage. With a combination of automatic blacklisting of tricky code, manual demacroing and lots of help from Benjamin I got the generated 3.2MB patch to compile.

I am sure that there are lots of bugs to be found still, but at least we’ve discovered the pattern that the code follows. I am also sure that there are lots of unpleasant surprises to be discovered and dealt with in the near future.

The bright side is that the result of these rewrites we should get a less buggy codebase that is easier to work on, more efficient and compiles to smaller binaries. My other big wish is to significantly reduce the amount of C++ magic in the codebase.

I am happy that garburator works as it means I can go back to playing outparamdel. Hopefully, once garburator+outparamdel are applied on all possible methods we’ll end up with relatively nice looking C++ code and a healthy performance boost.

Oct 07

Rewriting Tools for Mozilla 2: Moving Forward as Planned

In the Beginning There Was a Void

Approximately a year ago, Brendan discussed with me the crazy possibility of rewriting most of the Mozilla code automatically to modernize the codebase. The benefits were huge. Gecko would use the C++ standard library to improve code readability and reducing size, XPCOM would be ripped out of the core to improve performance and decrease footprint, etc.

It seemed like a good idea, but in reality no other giant C++ project has attempted this before so we were not sure of how realistic it was. I spent a year in a lonely corner of Mozilla trying to materialize the idea.

Brendan & Graydon pointed me to elsa, the C++ parser that supposedly could parse Mozilla. However, it turned out that it was only able to parse an old version of Mozilla and rejected the new source. One of the elsa maintainers even tried to convince us to it was not designed for source-to-source transformations and wouldn’t work that way.

After I patched up elsa and started devising ways to use it for source rewriting I ran into more pain. After a few false starts, I realized that C++ in Mozilla is actually a mix of CPP and C++ and one can not rewrite C++ without dealing with the mess that is macro expansion. MCPP was pointed out to me as a good starting point for hacking on a preprocessor. So I designed an inline log for macro expansion. To my surprise the maintainer of MCPP, Kiyoshi MATSUI, volunteered to implement the spec and thus saved me from a world of pain. (For which I am eternally grateful as I can’t imagine a more depressing pastime than working on the root of all evil: the C preprocessor).

In parallel with Kiyoshi’s work I modified elkhound & elsa to make the C++ parser a lot more suitable for source transformations. I learned about LR & GLR parsing and confirmed my suspicion that I don’t want to write parser generators for a living.

Happy Conclusion

All this work finally got us what we discussed last September: a framework for doing lots of boring code rewrites.

The first big Moz2 task is switching from reference counting to garbage collection. Today, garburator produced a gigantic patch for subset of the content/ module and all of the affected files compiled. Hopefully next week I’ll have a multi-megabyte patch for the whole of Mozilla that compiles and possibly runs.