Introducing Minion

yboily

4

Minion is a platform developed by the Security Automation team at Mozilla to enable integration and adoption of automated security testing that has been under development for the past year.

The platform allows any team to set up the basic requirements to perform automated scanning and testing of websites and services by providing sensible defaults for plugins that enable scanning of many types of web applications and services.

With the 0.3 release of Minion there are several milestones that have been achieved that have allowed us to start using Minion internally across our development community, quality assurance, and security teams.

Architecture

Minion is intended to be a platform that is simple to use, easy to deploy, simple to extend, and flexible enough to be integrated into any development or operations workflows. At a high level there are three major components in Minion: Plugins, Task Engine, and Front End.

Minon Plugins are light-weight wrappers that perform tasks such as configuring, starting, stopping a plan, and accept a set of callbacks to notify the caller that information is available. In order to be used, Plugins require a plugin runner that handles the invocation of the plugins as well as the results; in addition to supporting Minion’s task engine, the Minion backend repository includes command-line scripts to execute plugins. This provides support for testing during development of new plugins and allow a high degree of flexibility in how plugins are used outside of Minion.

The Task Engine is the core platform; it provides an API for managing and configuring Plans (collections of plugins and configurations), collections of users, sites and services, and the results of executions of Plans against those targets.

The Front End is a web application that provides both administration and usage of Minion; users can perform most of the configuration tasks needed to set up Minion plans, targets and users, as well as review the results of Minion scans. Being a Mozilla project, the front-end uses Persona for authentication, but all access control based decisions are built into Minion itself.

Minion Plugins

At their heart, Minion plugins are automation scripts designed to abstract away the platform, operating system, and features that an individual security tool implements, and provide a single mechanism for configuring the tool, initiating a scan, and collecting the results.

It may be helpful to look at the code for an existing plugin to better understand how they work; the AlivePlugin is a clear, simple example.

The Alive plugin is an extremely basic plugin that confirms that a host is reachable, but it implements all of the required features, and extends a BlockingPlugin. The plugin exposes some member variables that provide user interface cues (the name, links for additional information), and in this case, some built in report objects. In the do_run method the actual logic of the scan is performed, and since there is no detailed setup or stopping functionality is required, the BlockingPlugin starting and stopping functionality is sufficient.

Two base classes for plugins are provided in the Minion backend to get developers started:

  • BlockingPlugin this plugin provide the basic functionality to support a plugin that performs a task, and reports it’s completion state at the end. This is suitable for creating straightforward plugins directly within Python
  • ExternalProcessPlugin this plugin provides the functionality required to kick-off an external tool, and provides the basis for several other extensions, especially those that wrap existing security tools.

In addition to several basic “proof of technology” plugins that collect details about targets and provide best practice information, the Minion development team is currently maintaining three other extensions:

  • OWASP Zed Attack Proxy This plugin wraps the OWASP ZAP platform and enables detailed application scanning
  • Skipfish a simple, but powerful web fuzzer from Google
  • nmap a port scanning tool that is generally accepted as the best in it’s class

Minion Task Engine

The Task Engine provides the core functionality for managing users, groups, sites, scans, and results within the Minion platform. Acting as a central hub, the Task Engine maintains a register of available plugins, provides facilities for creating and modifying plans, and managing user access to Minion, including which sites they can scan.

Plugins

Plugin deployment is one of the only features of Minion that cannot currently be managed from within the Front-End; this is a result of the configuration needed to deploy them, but the Minion Front-End provides the ability to review the available plugins, and get the class details, which is the information required to add a plugin to a Plan.

Plans

A Minion Plan is JSON document that provides some information about what the plan does, and a sequence of tools to invoke. An example can be found below:

 

{
     "name": "Fuzz and Scan",
     "description": "Run Skipfish to fuzz the application, and perform a ZAP scan.",
     "workflow": [
          {
               "plugin_name": "minion.plugins.skipfish.SkipfishPlugin",
               "description": "",
               "configuration": {}
          },
          {
               "plugin_name": "minion.plugins.zap_plugin.ZAPPlugin",
               "description": "Run the ZAP Spider and Scanner",
               "configuration": {
                    "scan": true
               }
          }
     ]
}

In this example, the name and description are intended to be human readable descriptions of what the plan will do, while the workflow array contains a set of plugin names, a description that can will be included in the plan details, and a set of configuration details that may be plugin specific.

Users and Invites

Minion is intended to be a team oriented tool; as a result, the the platform allows user and group management. User accounts are created through an invitation mechanism, or via the administrative interface. The invitation system allows administrators to pre-create groups, sites and plans within Minion, and then add a user to that group before the user has enrolled. Once the invite is issued, an email will be sent to the user and the user can then access a configured profile.

Groups

Groups are the mechanism by which administrators can control how users have visibility into sites and results within in Minion. In order for a user to be able to interact with a site via Minion, that user needs to be added to the group, and the site needs to be associated with that group. This provides extremely fine grained control over visibility into scan results. Currently group membership allows both viewing of scans and the ability to re-execute a scan, but as the project progresses, constraints can be added to allow users to review results, but not initiate scans.

Minion Front-End

Designed to be easy to use and provide instant feedback, the front-end provides access to the Minion platform. Each of the pieces of the functionality described above is accessible via the front-end, and is explicitly enabled by calling the web services exposed by the Task Engine. One of the advantages of the architecture is that the front-end can be easily re-engineered with no impact to the back-end or plugins.

Technologies

Minion is built with Python, Angular.js, and several packages that assist in ensuring a reliable end to end service. These technologies were selected by our development team, but the architecture, and each of the service boundaries are intended to use JSON calls to permit easy integration with other services. Because of the design principles applied, it is entirely possible to implement plugins that run on any operating system or platform, and do not need to reside on the same service. With the appropriate network configurations it is possible to deploy the front-end, task engine, and plugins on different networks, which allows users to isolate the amount of attack surface that needs to be deployed in sensitive networks.

Road Map

There are several features that are under active development, and should be implemented over the next several releases.

Authentication & Access Management

Site Ownership Verification

This is a critical feature that enables users to demonstrate ownership of a site before initiating scans.

Granular Access Control

The ability to govern users ability to scan by group and site ownership as well as role.

Plugin Improvements

Improved Results Reporting

Minion is only as good as it’s plugins. Now that we have a working and reliable core platform, refinement of plugin results, and improving reporting is a core objective.

Deferred Execution Plugins

Sample implementations of invoking third party services so that we can demonstrate integrating with other Security as a Service platform

Reporting Plugins

Currently we have assigned risk ratings to findings based on our best practices, but that is not necessarily reflective of the priority of issues to other teams. We intend to implement a pluggable reporting interface, including the ability to add plugins to modify the risk ratings based on the security posture and priorities of the teams using Minion.

Front End

Landing Pages

Currently Minion is designed for technical users who have a need to see deep technical details. In the future, it may be desirable to generate metrics and dashboards, and to facilitate that Landing page support will be implemented to allow customization for user views.

Task Engine Improvements

Cohort

Minion is designed to support dynamic analysis via web application scanning. This is only one part of the story regarding how to perform automated security testing. Cohort is a branch of Minion that will enable analysis of source code repositories and perform static analysis.

Historical Issues

In order to facilitate ongoing tracking of a security program, support and integration for third party issue trackers (initial targets are Bugzilla and Github), and the ability to compare multiple scans over time will be implemented.

Why Minion?

The Mozilla Security team supports hundreds of websites of services, and products used by hundreds of millions of users. In addition our team supports hundreds of employees and thousands of community members that contribute to Mozilla products and services. Scaling to that level is not feasible without improving automation capabilities. While it would be much easier to solve this problem for ourselves, Mozilla’s mission is to support the open web, and protect our users. By building Minion as a foundation for a security as a service platform, integrating open source and free tools, then releasing it as open source, we aim to contribute a platform that can be used by any team to dramatically improve their coverage, and integrate security testing automation in all parts of their IT operations and software development processes.

Minion is an open source project, and we welcome contributors, users, and feedback!

Finally, I would like to extend a huge thanks to Stefan Arentz, Simon Bennetts, Yeuk Hon Wong, Matthew Fuller, and all of the other developers who have moved Minion from a sheet of paper and a set of shell scripts to a production service!

4 responses

  1. Michael Liu wrote on :

    Execellent! this would be wonderful tool for trying out some ad-hoc web app testing strategies

  2. oxdef wrote on ::

    What about scanning of auth protected web applications?

  3. oxdef wrote on ::

    Cool! What about scanning of auth protected web applications?

  4. Mathew Porter wrote on ::

    Will have to get the team to have a look at Minion and see how we can utilise it in our workflow.