Mozilla Add-ons Community Blog
Categories: developers jetpack

Add-on SDK Repacks proposal

8 responses

One thing we have planned to make part of the Add-on SDK story is to have “automatic repacks” for developers using the SDK. What this means is that if you host your add-on, built with the SDK, on AMO (addons.mozilla.org), we would automatically rebuild it everytime we release a new version of the Add-on SDK to maintain compatibility with a new version of Firefox. This would make sure that your add-on was always up-to-date with the latest version of Firefox.

This is still the plan, but for our first attempt we noticed some things that were wrong with the plan as we currently have it and would like to propose a way of getting past these problems and getting to a new repack process.

What Went Wrong

The first problem we faced is that the repacker is the first service to use AMO’s brand-new APIs for updating add-ons, and we hadn’t tested the interaction between the two sites sufficiently beforehand, so we had to implement fixes for issues we encountered during the repack testing, which slowed down the testing.

Second, our testing was short and not deep enough for what we were trying to accomplish. The repack team didn’t get enough testing in themselves, nor did they make it possible for other project participants to test repacks and suggests issues and fixes before the production run of repacking.

Finally, what we are fundamentally trying to do is repackage a derived format without the source code it came from as reference, like decompiling a program, making changes, and then recompiling it. This is something that is hard to do and as it turned out, didn’t work very well. We could fix that by including the source code in the XPI. However, including the source code in the XPI would defeat our goal of making the XPI size as small as possible (this is extremely important for mobile versions of Firefox).

A Proposal

So what can we do about these issues? First of all, the repack team will answer the testing problems by starting testing earlier and sharing the results with everyone. In addition, the use of the AMO APIs is now solved as the calls we are using are now verified and tested.
However, the issue about the required sources for a clean rebuild is a bit more difficult and needs a new proposal.

Short-term

In the short-term I propose that we only repack add-ons which are built using the Add-on Builder. This is the quickest and safest solution as we have access to all the sources with Builder-based add-ons. The repack process is much simpler too as we don’t have to unpack a XPI first. For those who use a local installation of the Add-on SDK to build an add-on, we will document how to repack your add-on and the process for pushing the new one to AMO. In addition, you can move your add-on to the Add-on Builder and take advantage of the automatic repacks if you’d like, but do keep in mind that the Add-on Builder is still in beta at this point.

Long-term

The long-term solutions will need to be studied and debated a bit more. One option we have is to create a “source package” that can be pushed to AMO for use in repacking. However, this would mean that AMO would have to store these source packages and generate XPIs from them in addition to taking on responsibility for repacking when a new SDK is released. Obviously this would involve adding a great deal of functionality to AMO, as well as tooling up the services side to store these new, larger packages.

Another option is to land parts of the Add-on SDK into Firefox itself, including API implementations and the module loader. If an API already in Firefox needs updating to maintain compatibility with changes to Firefox, we won’t need to repack add-ons that use that API, as the updates can be made to that new version of the browser rather than each individual add-on. Doing this has other benefits as well. For example, it may help reduce XPI sizes tremendously. However, this too would take a bit of time to implement and is thus a long-term solution.

At this point we think that the short-term solution of just repacking Add-on Builder-based add-ons would be the best approach. We would also provide documentation describing the simplest approach for repacking add-ons along with best practices and troubleshooting. In addition, we would look into how best to notify authors when their add-ons need to be repacked manually.

Still, we also want to know what you think about this as well as the long-term ideas. Please feel free to leave your comments here or on our jetpack discussion forum

8 comments on “Add-on SDK Repacks proposal”

  1. Robert O’Callahan wrote on

    Surely bundling the Addons SDK APIs into Firefox is the right answer here?

    There really doesn’t seem to be a good reason not to other than “it’s less convenient for the Addons SDK team”.
    http://mykzilla.blogspot.com/2011/08/why-add-on-sdk-doesnt-land-in-mozilla.html

  2. Mook wrote on

    Alternative long-term option:

    Bring back extension dependencies; add in support for downloading the missing dependency (probably with a McCoy-style hash); ship the Add-on SDK addons as only the addon-specific part and have Firefox pull in the main Jetpack piece. The first download will still be large (since it includes the Jetpack parts), but subsequent installs will be small. The Jetpack part can be updated independently.

    This will require phasing in, of course; but so will the option to land the Jetpack parts in Firefox. Done right, this also would support things like language packs for addons better.

    Just an off-the-cuff thought.

  3. zalun wrote on

    We could also build XPIs with the source inside, so inside uploaded package there will be a compiled XPI next to the source.
    Standard download would only use XPI from inside of the package, some special link would provide full package, or just the source…

  4. Wil Clouser wrote on

    Your short term proposal goes against the promise of ease of development for the SDK. The salespitch has always been that add-ons built with the SDK will continue to be compatible with new versions of Firefox (that statement is all over the FAQ still: https://wiki.mozilla.org/Jetpack/FAQ). Now it sounds like you’re amending that statement so it’s only true if people use the Builder tool.

    What is your suggestion for all the developers who live in remote regions and don’t have fast connections to the data center in San Jose or who want to develop code in their own editors and environments?

  5. Myk Melez wrote on

    roc: There are two discrete questions, and it’s important to be clear about the distinction between them.

    The first is whether to land the SDK’s codebase in mozilla-central, and continue development of the SDK in mozilla-central, so it is slightly less inconvenient for Firefox/Gecko developers to run its tests when they occasionally break them.

    That was the question I was addressing in my blog post , and I continue to believe the answer is no: the significant and frequent advantages for SDK engineers of a separate Git repository for this distinct-albeit-dependent product far outweigh the occasional slight inconvenience to Firefox/Gecko developers.

    (But I do think it’s worth making that inconvenience even slighter in ways like I outlined in the blog post.)

    The question Dave asks, on the other hand, is whether to integrate parts of the SDK’s functionality into Firefox so it is no longer necessary to bundle them into each addon package and then update those packages for each new incompatible version of Firefox.

    I mentioned this idea at the end of my blog post, and I continue to think it’s a good one, for multiple reasons, especially now that Firefox has moved to rapid releases (the old strategy was partly designed to enable us to ship new addon-facing APIs more rapidly than Firefox). I think that’s the avenue we should pursue to address this issue (although source packages are worth pursuing for other reasons).

    mook: indeed, addon dependencies are indeed an option. That code doesn’t seem well-exercised, but that could be fixed. I’m not sure what the experience for users is like, and I wouldn’t want users to have to manage dependencies (or, ideally, even to see them), so I’d want to dig further into that before going down that road. And then there’s the question of dependency hell (addon A depends on SDK addon v10; addon B depends on SDK addon v20, which contains a breaking change since v10; user installs addons A and B).

    zalun: yup, that’s an option too and is basically the same thing Dave is talking about when he describes “source packages.” This’d require changes to AMO, which feels hard these days, given all the important work on the AMO team’s plate.

    Wil: the proposal doesn’t change our goal of keeping SDK-based addons compatible with new versions of Firefox, it just stops trying to achieve this via automatic repacks for locally-developed addons.

    If we implement this proposal, we’ll continue to pursue our other strategies for compatibility, which include compatibility-minded API design, a very high bar for breaking changes to supported APIs, and regular SDK updates, all of which are intended to make manual repacking a simple, safe procedure. And we’ll also pursue the best longer term solution that will enable us to eliminate the burden of repacks entirely.

    The way I look at it is this: with a lot of work, we could continue to provide automatic repacks for all SDK-based addons, but they would end up being at best a mediocre experience for developers who work locally, given the limitations inherent in repacking an addon without access to its source.

    And given our limited resources, that effort would be better spent 1. providing a great automatic repack experience for Builder-based developers, 2. providing the best manual repack experience we can create for local developers, and 3. eliminating the need to do repacks entirely.

  6. Mook wrote on

    @Myk: Right, dependency hell was ignored by assuming that the SDK would forever remain completely backwards compatible (so always using the newest version of the SDK is acceptable). This constraint is also true for shipping the equivalent with the app/platform. The hash bit was meant to mean signing key + url, so that Firefox can pull in the SDK automatically and never need to tell the user about it (other than showing up in about:addons – that will require quite a bit of UI work to support this). Anyway, in the worst case, this just tells the user about conflicts rather than letting the user find out their browser is broken at next start (which is what happens currently).

    Either way, the work required (on both the platform and amo) would probably make this never happen… too bad.

  7. Mergo wrote on

    I didn’t know where to ask this, but, when will we have XUL compatibility with Jetpack? Jetpack seems seriously limited when it comes to integration with the browser toolbar and other built-ins. Also, wouldn’t the experienced XUL developers be less reluctant to make the switch to Jetpack if they could still use XUL and integrate slowly?

  8. Tony Mechelynck wrote on

    I suppose, and hope, that what you mean by saying “include the add-on SDK in Firefox” is actually “include the add-on SDK in Toolkit” (not in some Firefox-only frontend code) so that Thunderbird and SeaMonkey will be able to install Jetpacks just as easily as Firefox.