New feature pages, now with forms & queries!

I’m not going to go into a whole lot of detail since I’ve already blogged fairly extensively about this here, here, and here), so here’s the tl;dr version:

  1. The old feature pages worked well enough to get us through our first release cycle in the new Train Model world, but they fundamentally kinda sucked.
  2. We learned a lot in the process, and we’ve now launched the second version of the system, which includes a more logical page structure and uses much smarter tools. It’s still the wiki, but it’s the wiki with a couple of completely fantastic add-ons (this one, and this one).
  3. The most important benefits are:
    • New feature pages can be created and edited using a simple web form
    • The resulting feature pages support database-like queries for automatically generating lists, alternate views, and etc.

In other words: creating and editing feature pages is way easier than it used to be, and we no longer have to manually update things like the release trackers & feature lists because we can just build custom queries to do all of that for us.

Forms? In a wiki?

It’s true! Here’s a demo!

  1. Go here: Feature page demo
  2. Click the “Edit with form” link at the top
  3. Gasp with surprise and delight

Edit with form

Any feature page that is part of the new system will have the “Edit with form” link at the top. I strongly suggest you use the form because editing the raw text is significantly crappier and more error prone.

Migrating the old to the new

Unfortunately, the old pages don’t work with the new system at all — and we can’t use the form to edit/update them — so all still-relevant feature pages need to be migrated to the new system.

Happily, most of the existing feature pages have been migrated. If there are pages that haven’t been migrated that should be, you are welcome to migrate them yourself or ping me (deb-at-mozilla-com or “dria” on IRC) for help or if you have any questions.

Creating new feature pages

Do you want to create a new feature page? It’s super easy!

Create a new feature page

  1. Go here: Create a new feature page
  2. Follow the directions (4 steps, very simple)
  3. Ta dah!

Now the really cool part: Queries

The new system allows us query the feature pages and return results based on a variety of queryable properties in the pages themselves.

The query system is both straightforward and powerful. Here’s an example query that returns a list of all the prioritized feature pages for Mobile, and formats them as a simple table:

{{#ask: [[Category:Feature Page]] [[Feature list::Mobile]] [[Feature priority::!Unprioritized]]
| ?
| ?Feature name#
| ?Feature priority#
| ?Feature engineering team#
| ?Feature stage#
| ?Feature status#
| ?Feature feature manager#
| sort=Feature priority
| format=table
}}

You can see the result of that query here: Basic table.

It’s possible to make things a lot prettier, however, through a combination of templates, tables, and CSS:

<table class="querytable sortable">
<tr>
<td class="header" style="width: 5%;">Pr</td>
<td class="header" style="width: 30%;">Feature</td>
<td class="header" style="width: 15%;">Team</td>
<td class="header" style="width: 15%;">Stage</td>
<td class="header" style="width: 15%;">Status</td>
<td class="header" style="width: 20%;">Feature manager</td>
</tr>
{{#ask: [[Category:Feature Page]] [[Feature list::Mobile]] [[Feature priority::!Unprioritized]]
| ?#
| ?Feature name#
| ?Feature priority#
| ?Feature engineering team#
| ?Feature stage#
| ?Feature status#
| ?Feature feature manager#
| mainlabel=-
| sort=Feature priority
| format=template
| limit=500
| template=FeatureListTable
}}
</table>

This query creates a result table that looks like this: Mobile Feature list. Shiny!

Other query examples

Here are some other examples of what the queries can do. Feel free to “edit” any of these pages to see what the queries look like — they’re simple and powerful.

NOTE: please don’t actually change the code on any of these, as they’re mostly live and in-use — if you copy the code to another page on wiki.mo, you can mess around with it there. Thanks!

For more information about queries and how to write your own, see the add-on’s query documentation, with a super handy page about Selecting pages.

Pro tip!

If you make a change to a feature page and the query
doesn’t seem to be updating properly, please append

?action=purge

to the end of the URL and reload. This forces a
refresh of the cache, which is a bit aggressive sometimes.

Questions? Feedback? Need help?

If you would like to chat about this new system, or if you need help or have questions, please feel free to ping me any time, through any of the usual channels.

Posted in Features, Product Management | Leave a comment

goals for multi-process firefox

Since the release of Firefox 4 we’ve been working again to bring multi-process content support to Firefox. I thought that it would be good to write a post to try to lay out some of the reasons why we’re doing this. Although it might be obvious to many people, it’s good to actually lay it out on paper so that we have a clear understanding of why we’re doing something. It helps us determine what to prioritize as well as helps us measure when we’re ready to ship.

There are several areas listed below – performance, multi-core, memory behaviour, etc. For each of these areas there’s still a lot of work to do outside of the scope of the multi-process work. What this means is that every release of Firefox will get faster, more stable and has better multi-web page interactive performance even if it doesn’t include support for multi-process. But we know that in order to get across some hurdles we’re going to need to invest in multi-process model. That’s what this post is about. Multi-process is not a panacea for any of these topics, but it does give us a leg up on some of the more systemic problems.

Performance

In the case of Electrolysis we’re not talking about the kind of performance that’s usually referenced in the press or is the subject of benchmarks. What we’re really talking about with multi-process performance is responsiveness:

  • How long does it take for a mouse click to be recognized?
  • When you resize the window does it feel smooth?
  • Does the browser mysteriously pause from time to time?
  • Are animations smooth, without pauses?

These are all examples of measurements when building a responsive browser. At a basic level we’re talking about making sure that the main UI of the browser isn’t away from the mainloop for more than fifty milliseconds. We’ve made great strides here, and Firefox 5 is a great browser from a responsiveness standpoint. But we know that if we want to separate chrome and content concerns that we’re going to have to go to multi-process.

This is due to two reasons:

The cost of garbage collection goes up as the heap size of your process goes up.

In a single-process model, all web pages share the same heap. They also share memory with the chrome that control them. This means that whenever you have to garbage collect (look for unused objects in JavaScript) or cycle collect (which looks for cycles between C++ and JavaScript) you need to scan the entire JS heap looking for objects. As the amount of heap you have allocated increases, the amount of time you need to spend collecting increases with that size. For this reason, it’s better for pause times to have many small processes instead of one big process.

Now, this isn’t entirely true. We can garbage collect individual compartments where memory and objects are collected. We actually did this in Firefox 4, and it really helped our interactive performance, especially when running complex animations. But there are still cycles that exist across C++ and JavaScript, and we still need to cycle collect across the entire process. We do a lot of our GCing on other threads, including cycle collection, but it still stops the main thread. And because each GC affects the main thread, it causes pauses that can be felt in the UI. This means we have lots of little garbage collection events instead of one big one, but they still all block the main UI.

Garbage collection in content causes pauses in the main UI

Sometimes content gets large. Big web applications like gmail, facebook and twitter (yes, twitter is actually a pretty big web app) cause memory and garbage collection events to happen often. When they do, for reasons stated above, they still block the chrome. Compartments mitigate much of the pain here, but even if it’s for short periods of time, little pauses add up, and the user can feel them. We’d like to make sure that garbage collection for pages doesn’t really affect the main UI.

You can start to see a preview of tools for measuring responsiveness in one of Ted’s posts. Our investment in tools is happening along side of the multi-process work so we’re able to measure if we’re making progress in overall browser responsiveness. Those tools still need to be “productized” but since responsiveness is our primary metric and purpose for multi-process Firefox, we need to measure so we know we’re actually making forward progress.

Support for multi-core machines

There’s a basic problem with the web. The DOM is single-threaded. JavaScript, how CSS is resolved, and how objects are added and removed from the content model all assume that there’s a single view for a document.

This doesn’t mean that we don’t use threads throughout the browser. The networking stack, image decoding, much of our I/O, video and audio decoding and all kinds of other things are threaded and off the main loop of the browser. But the content itself is required to be single threaded.

Computing is quickly moving to a multi-core model. The speeds of processors aren’t increasing as much as they have been in the past, largely due to the constraints imposed by power and heat as well as the move to mobile. Basically everyone at this point has a multi-core processor on their desktop or laptop. And multi-core processors are starting to show up in mobile devices as well.

So one of the easiest ways to take advantage of multiple processors is to have each DOM assigned to its own processor, and the easiest way to do that is to have a few processes that can each be assigned to their own CPU.

We’re also investing in longer-horizon projects to examine what a multi-threaded DOM and layout engine might look like, but those are far enough away and risky enough where we know that there will be lots of value in building a multi-process browser for quite a while.

Predictable memory behaviour

Although we’ve made vast improvements to memory handling since the release of Firefox 4, we’re still faced with the fundamental problem of memory fragmentation. Because we’re based on C and C++, objects in our graph are often not relocatable. Over the long term, heap allocation will grow and cause memory to “leak.” This isn’t a problem that’s specific to Firefox. Just about every long-running process with even mildly complex allocation patterns suffers from this problem.

You can see this in the difference between the system memory reporting tools and what the internal allocator reports as memory that’s allocated. That “missing memory” is sometimes memory held in reserve, but they are often enough holes brought about by memory fragmentation. We also do some larger allocations in anonymous memory maps, but most small allocations still happen in pools that are allocated on the heap.

Physical pages of memory are allocated at the operating system layer and handed to user processes, at the process level, as virtual pages. The best way to return those to the operating system is to exit the process. It’s a pretty high-level granularity for recycling memory, for for very long-running browser sessions it’s the only way to get predictable memory behaviour. This is why content processes offer a better model for returning memory to the operating system over time.

Crash protection

We introduced protection from crashes in plugins with the release of Firefox 3.6.4. We implemented it because of the reliability problems that plugins – in particular Flash – were suffering from. Crashes in Flash were causing overall browser stability problems, and reflecting poorly on Firefox’s perceived reliability.

Although the number of crashes caused by content is relatively small – on the order of 1-2 crashes per 100 users per day – crashes that can be contained to the content processes are easier to identify, easier to diagnose and don’t take down the entire browser.

There’s also another nice benefit to having content processes. When there’s a crash, it’s much easier to tell what site caused the crash. In a single-process model, you can guess based on all of the sites that a person has open, but it could be any of them, and you have to look at a large sample of data and correlate sites to crash signatures to see patterns in the data. With a single tab (or small group of tabs) the number of sites is reduced so the crash can be more easily identified.

Sandboxing for security

The last goal that we have for adding support for multiple content processes to Firefox is for security. Some operating systems now have the ability to put a process into a “low rights mode” where they can’t access a lot of system resources. This means that even if there is a security problem in a content process, that the amount of damage that content process can do is limited to what the sandbox allows.

This system is imperfect, of course. Having the ability to talk to the “more privileged” chrome process can still result in exploits that have raised permissions. And it doesn’t protect one web site from another malicious web site. But it is a positive step forward, and is well-worth the investment.

Posted in Roadmaps | 33 Comments

Firefox 5 minor update for Mac coming soon

We will be making an update available to Firefox 5 before the scheduled release of Firefox 6.  This update will affect Macintosh users only.  This update will fix two specific problems:

  • There is a bug present in the upcoming release of OSX 10.7 that causes Firefox to crash when using downloadable fonts.  Downloadable fonts are used widely on the web, so users would have suffered severe crash problems.  We alerted Apple to the problem before the release of 10.7 but they did not fix the problem before 10.7 went to final release.  We’ve changed the font APIs that we’re using to newer versions which appear to fix the problem.  Users on 10.6 and before will continue to use the old APIs to reduce risk.  Later versions of Firefox will move all 10.6 and later users to the new APIs.
  • Apple recently updated Java on OSX 10.5 and it broke the Java plugin in Firefox.  Although the number of people using Firefox on 10.5 is quite small, this update fixes Firefox so that it can load the Java plugin on 10.5.

We’ve moved to a train model for Firefox, which means that we make releases of Firefox every six weeks for both security and new features.  Our model includes the capability to do minor updates for security or crash problems.  This will be our first out-of-band update since the minor updates we did in between the release of Firefox 4 and Firefox 5.

People still running the Firefox 3.6 release of Firefox will also see an update.  For 3.6 we will be disabling downloadable fonts for people running on OSX 10.7.

Firefox 3.6 users who upgrade to OSX 10.7 may also see scrollbar rendering issues.  For people who see scrollbar issues, we encourage you to upgrade to Firefox 5.

This update will only affect users of OSX.  Windows and Linux users should not see any updates.

Note: If you would like to help test these changes please try Firefox from our Aurora channel, which already contains these changes.

Posted in Releases | 20 Comments

New Feature Page system coming this week!

A couple of months ago I blogged about how we’re changing Firefox product planning and tracking. We’ve used that system for the release of Firefox 5 and most of Firefox 6, and we’ve learned a lot about what works and what doesn’t.

More recently, I blogged about a new feature page proposal, representing the first major revision to the feature pages system. That proposal has been finalized, and we will be putting it into action over the next couple of weeks.

New feature page structure

The feature page structure is being modified to more closely reflect how a feature moves from initial inception through release. You can see what the new structure looks like on this demo page.

Forms & queries

We’re also now using a couple of new Mediawiki extensions to simplify the process of creating, editing, searching for, and organizing feature pages.

Creating and editing feature pages will be as simple as using a form, both to create and edit the Feature pages. The form simplifies the creation process, and helps minimize the amount of wiki markup you have to do. On the other hand, all of the fields in the form allow for wiki markup, so you can use it anywhere you need to create tables, add formatting, insert pictures, etc.

The new form lives here: https://wiki.mozilla.org/Features/Create_new.

One of the most important changes the new system introduces is that we will no longer have to manually update Feature Lists, the Feature Inbox, or the Release Tracking pages. These pages will instead be autogenerated through a powerful new query system that has been added to the wiki. Additional custom queries for various teams and groups can also be created to help with planning and tracking as Firefox development progresses.

If you need help with either the form or the queries, do not hesitate to contact me through any of the usual channels.

What now?

Now that we have the new system finalized and ready to go, I’ll be working with the Product team to convert all existing feature pages to the new format. I’m hoping to get this finished up within a week or two (by July 15th). At that point we’ll be able to switch over to the new system entirely.

If you are not part of the Product team and you have feature pages that need to be converted, please feel free to use the new form to do so, or to contact me if you would like some help.

Posted in Features, Product Management | 2 Comments

Questions for you (about the new feature page proposal)…

I’ve put together some specific questions for you about the new feature page proposal. It would be great if you could take the time to review the proposal before June 19th.

You can send your feedback either by commenting on this post or by sending email to deb-at-mozilla-dot-com. I’m away until June 20th, but will dig through all the feedback as soon as I return. Thanks!

The questions…

1) Do the five stages — Definition, Design, Planning, Implementation, & Release — make sense? Are there features would not go through all five stages? Are there projects that would require more or different stages?

2) Are the sections in Stage I (Definition) — Overview, Use cases, Dependencies, & Requirements — sufficient to clearly define the scope, impact, and requirements of a feature? Is there anything missing?

3) Is there anything needed for Stage II (Design) other than a Functional specification and the User Experience design?

4) Should there be additional review sections explicitly included in Stage III (Planning) in addition to Security and Privacy?

5) When completed, will Stages I, II, and III provide enough information and detail to successfully build the feature? Is there anything else engineers, QA, localization or other teams need in order to start implementation?

6) Do Stages IV (Implementation) and V (Release) need any other detail, steps, or information? As it stands, these parts of the feature page will simply be a list of bugs and a checklist of landing criteria. Is that sufficient?

7) Overall, what do you think of the new feature page format proposal? Is there anything else you think we could do to improve this system as a whole?

Posted in Features, Product Management | 2 Comments

New feature page proposal

tl;dr

We’re working on a new Feature page proposal, making the whole system easier to use. It’s structured around the five stages each feature passes through during its development. It also makes use of a new Mediawiki add-on that lets us create queries and autogenerate lists on other wiki pages. We’re going to start using this new format at the end of June, so please give us your feedback ASAP.

The proposed new format can be found here:

https://wiki.mozilla.org/User:Dria/NewFeaturePageStructure2

The longer version

The new format aims to be more logically — and chronologically — ordered. Rather than being a mishmash of specs and lists and use cases and so forth, the new feature page includes five “stages” that the feature passes through: Definition, Design, Planning, Development, and Release.

We still have whatever flexibility we need to move back and forth between stages as needed as the feature is developed. For the most part, however, a feature will move from one stage to the next, with a straightforward review and sign-off by the relevant teams at the end of each.

We know that not every feature will need as much detail as this. After working through a few dozen of these we’ll have a stronger idea of which parts are necessary or not given the relative complexity of each and the scope of the work being done.

The stages

Stage I: Definition

This is where a feature is born and defined. If you would like to propose a new Feature, start by writing up a clear overview and a by defining the feature’s intended users and use cases. Where possible, you are encouraged to add to the list of the dependencies and requirements, but the Product team will help flesh out these parts if you don’t include them.

Stage II: Design

In this stage, Products, Engineering, UX, and/or User Research work together to finalize the feature’s functional specification and, if needed, the user experience design.

Stage III: Planning

With the requirements and design clarified and approved, the feature moves into the planning stage, where the Engineering team sorts out the overall technical approach that will be taken, and other stakeholder teams (Security, Privacy, QA, Accessibility, Localization, etc.) have the opportunity to review the feature in detail. This stage is complete when all stakeholder teams finish their reviews and are comfortable with the requirements, design, and plan for implementation.

Stage IV: Development

This is where coding begins and activity for feature development moves into Bugzilla. Detailed progress tracking will not be done on the wiki — that should all happen in Bugzilla, and will not be replicated on the feature pages. Product or project managers are responsible for updating the status on the feature pages if and when needed.

Stage V: Release

This stage documents the final checklist of everything the team feels should happen before a feature can land. Over time, we’ll develop a standardized checklist for this, but we’ll sort that out as we go. For now, the team will create their own checklist and ensure that everything on it is complete before the feature lands.

When a feature is released as part of a shipped product, it is considered “Complete” and the feature page is then archived for future reference.

Queryable data

The new feature page format includes some other information in addition to the stage sections, all of which can be used to query the pages and magically generate lists on other pages. This is all done by way of templates, and should be seamless to anyone creating or editing the pages themselves.

The new “Team status notes” table has been included because several of the stakeholder teams (Security, Privacy, Localization, etc) have asked for a way to include their own unique status notes on each feature. All of the values in this table are queryable, and can be used to generate customized lists for each team to simplify feature page & status tracking.

Feedback, comments, and questions

We will not start broadly using this new feature page format until the latter half of June, and we would like as much feedback on it as possible between now and then.

Please leave comments on this post, email them to me directly (deb at mozilla dot com), ping me in IRC (“dria”), or speak directly with me or any other member of the Products team.

Please note: I will be away from Wed Jun 8 until Mon Jun 20. Do not be shy about sending your feedback along in the meantime — I’ll be diving back into this as soon as I return.

Posted in Features, Product Management | 4 Comments

Feature List changes: May 27 – June 1

Desktop

Completed

Changed

Mobile

  • No changes

Platform

  • No changes

Services

  • No changes
Posted in Features, Product Management | Leave a comment

Feature Pages & Lists & Wikis, oh my

While it’s going to be a few weeks before I’ll be able to begin implementation, I thought I would post a quick update about the upcoming changes to the wiki Feature Pages, Lists, and tracking system.

Long story short: I’m working on an iterative improvement to the existing system that should make the whole thing a lot easier to use. The main improvements are:

1) A refined format for the Feature Pages that includes a more logical (chronological) ordering to the contents, and

2) The use of an extremely handy Mediawiki add-on that makes it possible to query the pages and automatically generate lists on other wiki pages. This will eliminate the need to manually edit the Feature Lists, Inbox, and Release Tracker, while also making it possible for different teams to generate customized lists to meet their own needs (as has been requested by basically everyone ever).

There’s a second add-on I’m hoping we can use that will hugely simplify the Feature Page creation and editing process, but that add-on is still in Security Review and there’s no guarantee it will pass. Fingers crossed.

I’ll be doing a final simplification pass on the latest Feature Page format proposal today and will circulate it for review and feedback before the end of the week. I’ll blog about it on either Friday or Monday, and there will be an opportunity for everyone to give feedback on it for two weeks. Unfortunately, during that two weeks I’ll be on vacation and offline, but will immediately dive back into things when I return, finalizing the changes and starting implementation by the end of June.

If you have any questions or feedback about this stuff, please don’t hesitate to ping me by email (deb-at-mozilla-dot-com) or in IRC (“dria”). Thanks!

Posted in Features, Product Management | 3 Comments

Feature List Changes: May 12-26

This is the second in a weekly series of blog posts outlining the changes in the Feature Lists. It should be the third post, of course, but I missed last week (sorry!). If you have any questions or notice any problems or errors, please leave a note here or contact Deb.

Desktop

Added

Completed

Started (“In progress”)

New Feature Pages

Removed

  • 20 high priority polish bugs (P2)
  • Expose prefs for tabs-on-demand, DNT, always restore tabs (P2)
  • Opt-in activation for plugins that are unknown/slow (P2)
  • Find-in-page behavior and UI improvements (P2)
  • Full set of HTML 5 form controls (P3)
  • Download file type cursors (P3)
  • Continued menu cleanup (OS X, context, tab menu) (P3)
  • Caps lock indicator in password field (P3)
  • File upload indicators on HTTP Post (P3)
  • Better new tab and tab sibling/parent behavior (P3)
  • Reusable Panorama grouping API for e.g VerticalTabs (P3)

Moved

Mobile

Started

Platform

Added

Started (“In Progress”)

Services

No changes.

Posted in Features | Leave a comment

Features Inbox Triage notes: May 25th

We held our inaugural open Features Inbox triage meeting yesterday afternoon, and filled the entire hour with discussions about everything from Panorama through Plugins. Unfortunately we didn’t quite clear out the Inbox completely, but we got through all but one of the newly added items.

Meeting notes — including action items and how we ended up prioritizing the proposed features — are available here: Inbox Triage Meeting: 2011-05-25.

Everything that was prioritized has been moved into the appropriate Features List — if you notice any issues or errors, please contact Deb.

Next Inbox Triage will be Wednesday, June 1st.

Posted in Features, Meetings, Product Management | Leave a comment