Mozilla has been working with security researcher and analyst Rich Mogull for a few months now on a project to develop a metrics model to measure the relative security of Firefox over time. We are trying to develop a model that goes beyond simple bug counts and more accurately reflects both the effectiveness of secure development efforts, and the relative risk to users over time. Our goal in this first phase of the project is to build a baseline model we can evolve over time as we learn what works, and what does not. We do not think any model can define an absolute level of security, so we decided to take the approach of tracking metrics over time so we can track relative improvements (or declines), and identify any problem spots. This information will support the development of Mozilla projects including future versions of Firefox.
Below is a summary of the project goals, and the xls of the model is posted at http://securosis.com/publications/MozillaProject2.xls. The same content as a set of .csvs is available here: http://securosis.com/publications/MozillaProject.zip [Update] There also a copy for OpenOffice: http://securosis.com/publications/MozillaProject2.ods
This is a preliminary version and we are currently looking for feedback. The final version will be a far more descriptive document, but for now we are using a spreadsheet to refine the approach. Feel free to download it, rip it apart, and post your comments. This is an open project and process. Eventually we will release this to the community at large with the hope that other organizations can adapt it to their own needs.
We would love to get your opinions on this, and if you are not comfortable commenting here you can mail Rich directly at firstname.lastname@example.org. When we have reviewed the feedback, we will post here with findings and continue the effort with your help.
To develop a metrics based model to track the relative security of Firefox, evaluate the effectiveness of security efforts within the development and testing process, and measure the window of exposure of Firefox users to security vulnerabilities.
To develop an open base model that can be standardized and expanded upon for other software development efforts to achieve the same goals.
1. Track security trends in the development of Firefox.
2. Measure the effectiveness of various tools, stages and techniques of secure development.
3. Measure the exposure window when new vulnerabilities are discovered- the time to get x% of the user base protected. Will include sub-metrics to measure the efficiency of the process, from initial response, through patch generation, through user base updated. Correlate by severity of vulnerability.