The following blog post is contributed by Yeuk Hon, an intern who has been with the Security Automation team at Mozilla over the summer. Today is his last day with Mozilla, and this post serves as a tutorial on how to write Minion Plugins. As an aside, I would also like to thank Yeuk Hon for his awesome work over the summer! – Yvan Boily
Hello, the Web! I am Yeuk Hon, a summer intern working with the Security Assurance team. In this blog post, I will go over how a Minion plugin works and how to write a Minion plugin that works for your tool. Before you dive into my blog post, I encourage you to read Yvan’s Introducing Minion if you are not familiar with Minion already.
To recap briefly, Minion was created to make a web platform where developers can kick off active vulnerability scans against their own sites. The ability to execute a Python script to invoke other tools and applications makes Minion powerful, easy to use and extend.
Minion Recap
Minion’s workflow is quite simple. A developer would come on Minion, select a plan for a site, click the scan button and then wait for the scan report to come back. A Minion plan is a JSON document containing a list of workflows. A workflow is a JSON hashtable (or dictionary in Python) that specifies which Minion plugin and what configuration parameters to use. In essence, you can run a scan using a single Minion plugin or multiple Minion plugins with multiple configurations.
Here is an example of a Minion plan:
[
{
"configuration": {},
"description": "Check to see if Set-Cookie has HttpOnly and secure flag enabled",
"plugin_name": "minion.plugins.setcookie.SetCookiePlugin"
},
{
"configuration": {
"auth": {
"type": "basic",
"username": "foo",
"password": "bar"
},
"scan": true
},
"description": "Run an active scan using X scanner",
"plugin_name": "minion.plugins.example.ExampleScanner"
}
]
This example plan will use two plugins. The first plugin does not expect any additional configurations while the second plugin, “ExampleScanner” is told to do a scan and the scanner is given the basic auth login. Configuration parameters can vary from one plugin to another, but we will try to document common plugin configuration patterns. For example, minion-zap-plugin and minion-skipfish-plugin both use the same authentication configuration pattern as shown above.
Plugin execution
There are two types of plugins in general: blocking plugins and external process plugins. Blocking plugins are Python scripts that do not invoke external processes like nmap. External process plugins use external processes like nmap to do the actual scan work, but the Python script helps spawning, collecting and returning results back to the Minion backend.
Minion’s backend uses the Twisted framework to drive events. To keep track of states, Minion uses RabbitMQ as broker and celery workers for queuing, state bookkeeping. and tasks execution. Below is a simplified diagram showing the plugin configuration, activation, and completion in the backend.
The main takeaways from the diagram are:
- Minion’s backend spawns a process using Python’s subprocess called minion-plugin-runner.
- The runner will invoke the plugin’s “do_start” method to run the plugin. A Blocking plugin is handled by calling Twisted’s “deferThread” and an external process plugins invokes an external process via Twisted’s “spawnProcess”.
If you have used Twisted before, you probably realized that method names like “do_start” are Twisted conventions. As a plugin author, knowing Twisted can be helpful and so we recommend you to check out this Twisted guide.
Example 1: SetCookiePlugin
That’s a lot of information to digest so let’s look at an actual plugin. We will continue with the Set-Cookie plugin we listed in our example Minion plan. This snippet shows part of the plugin, but you can find the full source code here.
import requests
from minion.plugins.base import BlockingPlugin
class SetCookiePlugin(BlockingPlugin):
PLUGIN_NAME = "SetCookie"
PLUGIN_VERSION = "0.1"
FURTHER_INFO = [ {"URL": "http://msdn.microsoft.com/en-us/library/windows/desktop/aa384321%28v=vs.85%29.aspx", "Title": "MSDN - HTTP Cookies"} ]
def do_run(self):
r = requests.get(self.configuration['target'])
if 'set-cookie' not in r.headers:
return self.report_issues([
{'Summary': "Site has no Set-Cookie header",
'Description': "The Set-Cookie header is sent by the server in response to an HTTP request, which is used to create a cookie on the user's system.",
'Severity': "Info",
"URLs": [ {"URL": None, "Extra": None} ],
"FurtherInfo": self.FURTHER_INFO}])
else:
# take care of cases where
# (1) HttpOnly flag is not set,
# (2) secure flag is not set
# (3) both flags ARE set
We first set the name of the plugin and the version of the plugin. Since the list of references (we called them further info in plugin) are static, we can make them class static member variable at the class level. We will talk about what BlockingPlugin class does later, but for the meantime, all we have to know do for this plugin is to override the do_run method. We just check to see if “set-cookie” is in the response header and then report our observation back to Minion in Minion’s report format. If set-cookie is not present, there is no risk so the level of severity is hardcoded to Info.
The report scheme looks like this:
[
{
"Summary": "One sentence description of the issue (required)",
"Description": "In-depth description of the issue and why the issue matters (required)",
"Solution": "Mitigations (optional)",
"Severity": "High/Medium/Low/Info/Error/Fatal (required)",
"URLs": [
{
"URL": "http://target_site.com/path1",
"Extra": "Extra information on why this particular URL is affected"
}],
"FurtherInfo": [
{
"URL": "http://reference1.com/",
"Title": "Reference read title 1"
}]
}
]
The URLs are also optional and used to indicate what parts of the site are affected by the same issue. The values of severity level are High, Medium, Low, Info, Error and Fatal. Solution is optional if the issue doesn’t need to offer a solution. For example, when severity level is Info you don’t need any solution.
To implement a plugin, there are a few things to do.
- A plugin must be a class and the top of the class chain should be “minion.plugins.base.AbstractPlugin“
- “do_configure“, “do_start“, “do_stop“ methods are implemented
- use self.report_issue to return a list of issues in JSON format
- the “self.configuration“ contains all the configuration parameters passed from Minion plan in addition to the site url (which is defined as self.configuration[‘target’]).
Plugin classes
We spoke earlier we generalized plugins into blocking and external process, Minion is shipped with “BlockingPlugin“ and “ExternaProcessPlugin“ out of the box for plugin authors to use:
-
“minion.plugins.base.BlockingPlugin“ is used as a parent class for plugins written in pure Python that don’t require running a subprocess.
- * “minion.plugins.base.ExternalProcessPlugin“ is used as a parent class for plugins that require launching an external process.
You can create your own plugin type by subclassing “AbstractPlugin“ (or further subclass from “BlockingPlugin“ and “ExteranlProessPlugin“) if you have to.
Example 2: SetCookieScannerPlugin
Here is the full source code running a Go program which mirrors the SetCookie plugin we spoke earlier.
from minion.plugins.base import ExternalProcessPlugin
# Set-Cookie checker by running setcookie scanner written in Go
class SetCookieScannerPlugin(ExternalProcessPlugin):
PLUGIN_NAME = "SetCookieScanner"
PLUGIN_VERSION = "0.1"
def do_start(self):
scanner_path = self.locate_program("setcookie_scanner")
if not scanner_path:
raise Exception("Cannot find setcookie_scanner program.")
self.stdout = ""
self.stderr = ""
# spawn by calling the executable and a list of args
self.spawn(scanner_path, [self.configuration['target']])
def do_procss_stdout(self, data):
self.stdout += data
def do_process_stderr(self, data):
self.stderr += data
def do_process_ended(self, process_status):
if self.stopping and process_statsu == 9:
self.report_finish("STOPPED")
elif process_status == 0:
# try to convert the JSON outputs in stdout
stdouts = self.stdout.split('\n')
minion_issues = []
for stdout in stdouts:
try:
minion_issues.append(json.loads(stdout))
except ValueError:
logging.info(stdout)
pass
self.report_issues(minion_issues)
self.report_finish()
else:
self.report_finish("FAILED")
We subclass “ExternalProcessPlugin“ and use “self.spawn“ to call the Go command-line program. If your program speaks JSON, it is easy to parse the output (and this program is written to mirror the Python code, so the Go program actually outputs the standard issue format that Minion is expecting to use). It would be awesome if other scan tools and Minion can agree on a common report scheme because then writing plugins and exporting to other tools become trivial and possible.
Grow the Ecosystem!
Here is a list of plugins we have developed so far:
- basic plugins
- minion-zap-plugin
- minion-skipfish-plugin
- minion-setcookie-plugin
- minion-nmap-plugin
- minion-ssl-plugin
- minion-breach-plugin
Also, a community member has been writing a minion-arachni-plugin for Arachni.
But that’s not enough. We want to make security reviews agile by allowing developers to create and to use different plugins to keep their application secured from common vulnerabilities on their own. To achieve this goal, we must build an ecosystem with a great number of useful plugins. If you haven’t heard, Minion IS open source. Fork it on https://github.com/mozilla/minion and help us grow. Minion can become big. Let us know your ideas and talk to us to get you started. Personally, I feel Minion has the potential to grow into something big like OpenStack and Docker.
Anyway, you can reach us via
Although my internship is coming to an end this week, I will continue to contribute on Github like the rest of our Minion developers always do. I want to thank all of you awesome Mozillians and Minion users for the support and guidance over the past 12 weeks. In particular, I want to thank my Web Security Automation team:
- Stefan Arentz for being Hacker’s Best Mentor and leading Minion development
- Yvan Boily for providing resources for Minion development
- Simon Bennetts for mentoring me on improving minion-zap-plugin
- Mark Goodwin for his initial security review and support
and a big thanks to Stephen Donner (Mozilla Web QA manager) for his commitment to use Minion.
