New Workshop on Negative Results in e-Science

This guest post is by Ketan Maheshwari, Daniel S. Katz,  Justin Wozniak, Silvia Delgado Olabarriaga, and Douglas Thain on the ERROR Conference, 3 September 2015.

Introduction

Edison performed 10,000 failed experiments before successfully creating the long-lasting electrical light bulb. While Edison meticulously kept a list of failed experiments, a wider dissemination of earlier failures might have led to a quicker invention of the bulb and related technologies. Scientists learn a great deal from their own mistakes, as well as from the mistakes of others.  The pervasive use of computing in science, or “e-science,” is fraught with complexity and is extremely sensitive to technical difficulties, leading to many missteps and mistakes. Our new workshop intends to treat this as a first-class problem, by focusing on the hard cases where computing broke down. We believe that these computational processes or experiment that yielded negative results can be a source of information for others to learn from.

Why it’s time for this workshop

  1. Publicizing negative results leads to quicker and more critical evaluation of new techniques, tools, technologies, and ideas by the community.
  2. Negative results and related issues are real and happen frequently. A publication bias towards positive results hurts progress since not enough people learn from these experiences.
  3. We want to get something valuable out of failed experiments and processes. This redeems costs, time and agony. Analysis of these failures helps narrow down possible causes and hastens progress.
  4. We want to promote a culture of accepting, analyzing, communicating and learning from negative results.

The ERROR Workshop

The 1st E-science ReseaRch leading tO negative Results (ERROR) workshop (https://press3.mcs.anl.gov/errorworkshop), to be held in conjunction with the 11th IEEE International Conference on eScience (http://escience2015.mnm-team.org) on 3 September 2015 in Munich, Germany will provide the community a dedicated and active forum for exchanging cross-discipline experiences in research leading to negative results.

The ERROR workshop aims at providing a forum for researchers who have invested significant efforts in a piece of work that failed to bear the expected fruit. The workshop will provide the community a dedicated and active forum for exchanging cross-discipline experiences in research leading to negative results. The focus is on various aspects of negative results, such as premises and assumptions made, divergence between expected and actual outcomes, possible causes and remedial actions to avoid or prevent such situations, and possible course corrections. Both applications and systems areas are covered, including topics in research methodology, reproducibility, the applications/systems interface, resilience, fault tolerance and social problems in computational science, and other relevant areas. We invite original work in an accessible format of 8 pages.