{"id":75322,"date":"2025-01-21T08:55:36","date_gmt":"2025-01-21T16:55:36","guid":{"rendered":"https:\/\/blog.mozilla.org\/?p=75322"},"modified":"2025-01-21T10:54:58","modified_gmt":"2025-01-21T18:54:58","slug":"dataset-convening","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/en\/mozilla\/dataset-convening\/","title":{"rendered":"Mozilla, EleutherAI publish research on open datasets for LLM training"},"content":{"rendered":"\n
\"A
Participants of the Dataset Convening in Amsterdam.<\/figcaption><\/figure>\n\n\n\n

Update: Following the 2024 Mozilla AI Dataset Convening, AI builders and researchers publish best practices for creating open datasets for LLM training. <\/em><\/h3>\n\n\n\n
\n\n\n\n

Training datasets behind large language models (LLMs) often lack transparency, a research paper published by Mozilla and EleutherAI explores how openly licensed datasets that are responsibly curated and governed can make the AI ecosystem more equitable. The study is co-authored with thirty leading scholars and practitioners from prominent open source AI startups, nonprofit AI labs, and civil society organizations who attended the Dataset Convening on open AI datasets in June 2024.

Many AI companies rely on data crawled from the web, frequently without the explicit permission of copyright holders. While some jurisdictions like the EU and Japan permit this under specific conditions, the legal landscape in the United States remains murky. This lack of clarity has led to lawsuits and a trend toward secrecy in dataset practices\u2014stifling transparency, accountability, and limiting innovation to those who can afford it.<\/p>\n\n\n\n

For AI to truly benefit society, it must be built on foundations of transparency, fairness, and accountability\u2014starting with the most foundational building block that powers it: data. <\/p>\n\n\n\n

The research, \u201cTowards Best Practices for Open Datasets for LLM Training,<\/strong>\u201d outlines possible tiers of openness, normative principles, and technical best practices for sourcing, processing, governing, and releasing open datasets for LLM training, as well as opportunities for policy and technical investments to help the emerging community overcome its challenges. <\/p>\n\n\n\n

READ THE RESEARCH HERE<\/strong><\/a><\/p>\n\n\n\n

Building toward a responsible AI future requires collaboration across legal, technical, and policy domains, along with investments in metadata standards, digitization, and fostering a culture of openness. <\/p>\n\n\n\n

To help advance the field, the paper compiles best practices for LLM builders, including guidance on Encoding preferences in metadata<\/strong>, Data sourcing, Data Processing, Data Governance\/Release, and Terms of Use.<\/strong><\/p>\n\n\n\n

To explore the recommendations check the full paper<\/a> (also available on arXiv<\/a>). <\/p>\n\n\n\n

We are grateful to our collaborators \u2013 273 Ventures, Ada Lovelace Institute, Alan Turing Institute, Cohere For AI, Common Voice, Creative Commons, Data Nutrition Project, Data Provenance Initiative, First Languages AI Reality (Mila), Gretel, HuggingFace, LLM360, Library Innovation Lab (Harvard), Open Future, Pleias, Spawning, The Distributed AI Research Institute, Together AI, and Ushahidi\u2013 for their leadership in this work, as well as Computer Says Maybe for their facilitation support. <\/p>\n\n\n\n

We look forward to the conversations it will spark.<\/p>\n\n\n\n

\n\n\n\n

Previous post published on July 2, 2024:<\/em><\/p>\n\n\n\n

Mozilla and EleutherAI brought together experts to discuss a critical question: How do we create openly licensed and open-access LLM training datasets and how do we tackle the challenges faced by their builders?<\/em><\/h3>\n\n\n\n

On June 11, on the eve of MozFest House in Amsterdam<\/a>, Mozilla<\/a> and EleutherAI<\/a> convened an exclusive group of 30 leading scholars and practitioners from prominent open-source AI startups, nonprofit AI labs and civil society organizations to discuss emerging practices for a new focus within the open LLM community: creating open-access and openly licensed LLM training datasets. <\/p>\n\n\n\n

This work is timely. Although sharing training datasets was once common practice among many AI actors, increased competitive pressures and legal risks have made it almost unheard of nowadays for pre-training datasets to be shared or even described by their developers. However, just as open-source software has made the internet safer and more robust, we at Mozilla and EleutherAI believe open-access data is a public good that can empower developers worldwide to build upon each other\u2019s work. It fosters competition, innovation and transparency, providing clarity around legal standing and an ability to stand up to scrutiny.<\/p>\n\n\n\n

Leading AI companies want us to believe<\/a> that training performant LLMs without copyrighted material is impossible. We refuse to believe this. An emerging ecosystem of open LLM developers have created LLM training datasets \u2014such as Common Corpus<\/a>, YouTube-Commons<\/a>, Fine Web<\/a>, Dolma<\/a>, Aya<\/a>, Red Pajama<\/a> and many more\u2014that could provide blueprints for more transparent and responsible AI progress. We were excited to invite many of them to join us in Amsterdam for a series of discussions about the challenges and opportunities of building an alternative to the current status quo that is open, legally compliant and just. 
During the event, we drew on the learnings from assembling \u201cCommon Pile\u201d (the soon-to-be-released dataset by EleutherAI composed only of openly licensed and public domain data) which incorporates many learnings from its hugely successful predecessor, \u201cThe Pile<\/a>.\u201d At the event, EleutherAI released a technical briefing and an invitation to public consultation on Common Pile.<\/p>\n\n\n\n

\"A
Participants engaged in a discussion at \u201cThe Dataset Convening,\u201d hosted by Mozilla and EleutherAI on June 11, 2024 to explore creating open-access and openly licensed LLM training datasets.<\/figcaption><\/figure>\n\n\n\n

Our goal with the convening was to bring in the experiences of open dataset builders to develop normative and technical recommendations and best practices around openly licensed and open-access datasets. Below are some highlights of our discussion:<\/p>\n\n\n\n