{"id":743,"date":"2014-07-30T14:50:58","date_gmt":"2014-07-30T14:50:58","guid":{"rendered":"http:\/\/mozscienceblog.wpengine.com\/?p=743"},"modified":"2019-02-28T12:40:46","modified_gmt":"2019-02-28T20:40:46","slug":"discovery-of-scientific-software","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/foundation-archive\/mozilla-science\/discovery-of-scientific-software\/","title":{"rendered":"Discovery of scientific software"},"content":{"rendered":"

This is a guest post by Jure Triglav, an open science hacker. You can check out http:\/\/juretriglav.si<\/a>\u00a0for more of his projects or follow him on Twitter.<\/a><\/em><\/p>\n

TL;DR: An open API for science helps researchers discover great software. Install the Scholar Ninja extension<\/a>\u00a0and you\u2019ll get recommendations (based on software citations) on-the-fly while browsing GitHub.<\/strong><\/p>\n

\"scientificsoftware-blogpost\"<\/p>\n

A while back I wrote about an open distributed search engine for science, Scholar Ninja<\/a>, and about how great it will be to have an open API which you can query and get to all of science, no matter if you\u2019re human or machine<\/strong>. Having the world\u2019s knowledge openly accessible like that will result in a paradigm shift. I dare you to say it ain\u2019t so! (Also, do check out ContentMine<\/a>!)<\/p>\n

While that project is still in early stages of development (most recently, we even had to turn off our core feature, WebRTC<\/a>), lots of people have asked what the use cases for such a search engine\/API would be, anyway. A great many don\u2019t appreciate how important this will be, and while that might seem silly at first glance, the vast majority of researchers is quite satisfied with the closed, machine-unfriendly Google Scholar and other closed behemoths. How can we show them the light?<\/p>\n

After a lot of conversations at the #OKFest14<\/a> and more recently at the Mozilla Science Lab Code Sprint<\/a>, one use case that surfaced from virtually every discussion, was helping scientists (or anyone) discover great software in their field (or any field, or anywhere!). As a tsunami of code is reaching the shores of Science, discovering scientific software will only get more important:<\/p>\n

\"Bj1axSBCMAAjBsd\"<\/a><\/p>\n

Luckily, it\u2019s quite possible to use Scholar Ninja for scientific software discovery. In fact, it\u2019s more than possible \u2014 it\u2019s already done.<\/p>\n

Before we get ahead of ourselves, let me provide just a bit of necessary backstory: Scholar Ninja<\/a> indexes (or rather, did and will again<\/em> index, see issues<\/a>) every paper you read online and adds paper\u2019s metadata, keywords and URLs to a globally distributed search index<\/strong>, which is based on browsers, WebRTC and magic. Everyone who has the extension installed is a node in a Chord DHT network<\/a> and is both an indexer and a server of content. Scholar Ninja\u2019s mission in life is to become a complete and completely open search engine for science.<\/strong><\/p>\n

To get right back to the main story here \u2014 if we can get URLs from each paper, that means we can get a large portion of software citations, which usually look something like this:<\/p>\n

… reads longer than 300 bases were separated by barcode and trimmed using sickle (https:\/\/github.com\/najoshi\/sickle<\/a>); 72.3% of reads were retained …<\/p><\/blockquote>\n

That is, URL inlined directly into the text. Or referenced classically in the References section:<\/p>\n

24. Nikhil J. Sickle – a windowed adaptive trimming tool for FASTQ files using quality. https:\/\/github.com\/najoshi\/sickle<\/a>.<\/p><\/blockquote>\n

In both cases the URL is right there, and that appears to be case for most software citations. The URL itself usually points to places like GitHub, BitBucket, SourceForge, Google Code, R Project, etc.<\/p>\n

Alright, enough mumbling, imagine that you\u2019re new to the field of bioinformatics and your first task on the job is to process a vast number of FASTQ files representing partial genetic sequences and assemble them into a coherent sequence. \u201cNo problem\u201d, you say, \u201cfirst I need to analyse the sequences and trim them when their quality<\/a> gets too low.\u201d You start to do this by hand and a week goes by, then two, then three weeks, woosh<\/em>. Luckily a wise co-worker walks by your tiny windowless office, sees what you\u2019re doing, chuckles kindly, then proceeds to tell you about this great piece of software called sickle<\/strong><\/a> and gives you a link to the GitHub repository. Now, naturally, you find sickle<\/strong><\/a> fantastic and wonder how you ever could have processed your FASTQ files without its windowed adaptive trimming functionality. \u201cSickle is the best!\u201d<\/strong> You\u2019d like to know if there are more tools like it out there that could save you hours and days and weeks of work in the future. Unfortunately your co-worker just left the department to go sailing around the world for a year, so you have no one to ask. What can you do?<\/p>\n

Here is where the power of an open API for science can really start to shine: How about you take a weekend and create a scientific software recommender system based on software citations that you get from the API?<\/strong><\/p>\n

Let\u2019s do it!<\/p>\n

Step 0. Create open API for science<\/h2>\n

This is what Scholar Ninja hopes to provide in the future. It\u2019s a sizeable project, fraught with many perils, not least of which are the many licensing restrictions publishers place on scientific content. But I digress. We\u2019ve had a working version of the network running with around 70 nodes at peak, but due to bugs in Chrome<\/a> and our own implementation, it turned out to not be ready for prime time just yet. However, once complete, if you\u2019ll run a Scholar Ninja super-node in Node.js, you\u2019ll be able to join the network and query it just like you would query a good old regular HTTP API.<\/p>\n

Step 1. Use open API for science<\/h2>\n

OK, step 0 just gave you a great API to work with. For our purpose, you could for example ask it to give you all citations of sickle<\/strong>:<\/p>\n

GET \/citations\/software?url=github.com\/najoshi\/sickle<\/code><\/p>\n

And you’ll get a nice JSON back, with the papers that cite sickle<\/strong>. OK, so now query the API to give you other software which these papers cite:<\/p>\n

GET \/software?cited_in=[\"10.1155\/2014\/404578\", \"10.1371\/journal.pone.0101021\", ...]
\n<\/code><\/p>\n

And again get a nice JSON back:<\/p>\n

[{
\n\"id\": \"github.com\/jstjohn\/SeqPrep\",
\n\"source\": \"doi:10.1155\/2014\/404578\",
\n},
\n{
\n\"id\": \"github.com\/vsbuffalo\/scythe\",
\n\"source\": \"doi:10.1186\/gb-2013-14-6-r66\",
\n},
\n...]
\n<\/code><\/p>\n

Well that was easy. We have our data, all we need to do now is package it up and present it to the user.<\/p>\n

Step 2. Win.<\/h2>\n

For the purposes of this demonstration, I\u2019ve indexed 848,418 open access papers from Europe PMC<\/a> and analysed 18,765,516 citations, plus 10,837 software citations found within (source code here<\/a>). When Scholar Ninja\u2019s network will be fully operational, the indexing will be real-time and organic, as users navigate the web and read scientific papers \u2014 but have patience, we\u2019re not there yet. However, to help pass the time, if you install the extension<\/a> right now, every GitHub page where scientific software is detected and recommendations are available, will be enhanced by a neat unobtrusive panel full of great scientific software:<\/p>\n

\"interface2\"<\/p>\n

There are eleven<\/a> recommendations for https:\/\/github.com\/najoshi\/sickle<\/a>. Eleven pieces of software were cited in papers where sickle<\/strong> was cited and in papers which the paper where sickle<\/strong> was cited cited. Wooh.<\/em> The external link icon above signifies citations, for example, we found 6 citations for scythe<\/strong>. The rest should be self explanatory. This view is then simply embedded into the standard GitHub interface:<\/p>\n

\"interface3\"<\/p>\n

Awesome? I sure think so. Be sure to install the extension<\/a> and try it out! And this is just a very very tiny glimpse of what the future holds if we build an open API for science.<\/strong> Come on, join us!<\/a><\/p>\n

\"XKpHM\"<\/p>\n","protected":false},"excerpt":{"rendered":"

This is a guest post by Jure Triglav, an open science hacker. You can check out http:\/\/juretriglav.si\u00a0for more of his projects or follow him on Twitter. TL;DR: An open API for science helps researchers discover great software. Install the Scholar … Continue reading<\/a><\/p>\n","protected":false},"author":144,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[290376],"tags":[],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/posts\/743"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/users\/144"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/comments?post=743"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/posts\/743\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/media?parent=743"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/categories?post=743"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/foundation-archive\/wp-json\/wp\/v2\/tags?post=743"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}