Five institutions including the Allen Institute for Artificial Intelligence recently announced the most comprehensive open source model “OLMo” in history, disclosing the model's model weights, complete training code, data sets and training process, setting a new benchmark for future work in the open source community. .
For many years, language models have been at the core of natural language processing (NLP) technology. Considering the huge commercial value behind the models, the technical details of the largest and most advanced models are not public.
Now, a truly completely open source large model is here!
Researchers from the Allen Institute for Artificial Intelligence, the University of Washington, Yale University, New York University, and Carnegie Mellon University jointly published a work that will go down in the history of the AI open source community——
They will open source almost all the data and information in the process of training a large model from scratch!

Specifically, the Open Language Model (OLMo) experiment and training platform launched by the Allen Institute for Artificial Intelligence provides a completely open source large model, as well as all data and data related to the training and development of this model. technical details–
Training and modeling:It includes complete model weights, training code, training logs, ablation studies, training metrics, and inference code.
Pre-training corpus:A pre-trained open source corpus containing up to 3T tokens, and the code to generate these training data.

Model parameters:The OLMo framework provides four 7B size models under different architectures, optimizers and training hardware systems, as well as a 1B size model, all models are trained on at least 2T tokens.
At the same time, the code used for model inference, various indicators of the training process, and training logs are also provided.

Assessment Tools:The evaluation tool suite during the development process is disclosed, including more than 500 checkpoints and evaluation code included in each 1000 steps of each model training process.
All data is licensed under Apache 2.0 (free for commercial use).

Such thorough open source seems to set a pattern for the open source community – in the future, if you are not open source like me, don't say that you are an open source model.

Performance evaluation
Judging from the core evaluation results, OLMo-7B is slightly better than similar open source models. Out of the first 9 reviews, OLMo-7B ranked in the top three in 8 of them, surpassing all other models in 2 of them.
OLMo-7B surpasses Llama 2 on many generation tasks or reading comprehension tasks (such as truthfulQA), but performs worse on some popular question and answer tasks (such as MMLU or Big-bench Hard).

The figure below shows the changing trend of the accuracy of 9 core tasks. Except for OBQA, as OLMo-7B receives more data for training, the accuracy of almost all tasks shows an upward trend.

At the same time, core evaluation results of OLMo 1B and its siblings show that OLMo is on the same level as them.

Using the Allen Institute for AI's Paloma, a benchmark, and available checkpoints, the researchers analyzed the relationship between a model's ability to predict language and model size factors such as the number of tokens trained.
As can be seen, OLMo-7B is on par with mainstream models in performance. Among them, the lower the number of bits per byte (Bits per Byte), the better.

Through these analyses, the researchers found that the efficiency of models in processing different data sources varies greatly, which mainly depends on the similarity between the model training data and the evaluation data.
In particular, OLMo-7B performs well on data sources mainly based on Common Crawl (such as C4).
However, OLMo-7B is less efficient than other models on data sources that are not closely related to web scraping text, such as WikiText-103, M2D2 S2ORC, and M2D2 Wikipedia.
RedPajama's evaluation shows a similar trend, possibly because only 2 of its 7 fields come from Common Crawl, and Paloma gives equal weight to each field in each data source.
Given that curated data sources like Wikipedia and arXiv papers provide far less heterogeneous data than web scraped text, maintaining high efficiency over these language distributions will be more difficult as pre-training datasets continue to grow.

OLMo architecture
In terms of model architecture, the team is based on the decoder-only Transformer architecture, adopts the SwiGLU activation function used by PaLM and Llama, introduces rotation position embedding technology (RoPE), and improves GPT-NeoX-20B's word-based A tokenizer for section pair encoding (BPE) to reduce personally identifiable information in model output.
In addition, in order to ensure the stability of the model, the researchers did not use bias terms (this is the same as PaLM).
As shown in the table below, the researchers have released two versions, 1B and 7B, and also plan to launch a 65B version soon.

The table below details the performance of the 7B architecture with these other models at similar scales.

Pre-training dataset: Dolma
Although researchers have made some progress in obtaining model parameters, the current level of openness of pre-training datasets in the open source community is far from enough.
Previous pre-training data is often not made public as the model is open source (let alone closed source models).
Documentation of these data often lacks sufficient detail that is critical to replicating the study or fully understanding the work involved.
This situation makes language model research more difficult—for example, understanding how training data affects model capabilities and its limitations.
In order to promote open research in the field of language model pre-training, researchers constructed and made public the pre-training data set Dolma.
This is a diverse, multi-source corpus containing 3 trillion tokens obtained from 7 different data sources.
On the one hand, these data sources are common in large-scale language model pre-training, and on the other hand, they are also accessible to the general public.
The table below gives an overview of the data volume from each data source.

Dolma's construction process includes six steps: language filtering, quality filtering, content filtering, deduplication, multi-source mixing and tokenization.
During the organization and final release of Dolma, researchers ensured that documentation from each data source remained independent.
They also open sourced a set of efficient data sorting tools, which can help further study Dolma, replicate results, and simplify the sorting of pre-training corpus.
In addition, researchers have also open sourced the WIMBD tool to facilitate data set analysis.


Training OLMo
Distributed training framework
The researchers used PyTorch's FSDP framework and ZeRO optimizer strategy to train the model. This approach effectively reduces memory usage by splitting the model's weights and their corresponding optimizer states across multiple GPUs.
When processing models up to 7B in size, this technology enables researchers to process micro-batch sizes of 4096 tokens per GPU for more efficient training.
For the OLMo-1B and 7B models, the researchers fixed a global batch size of approximately 4M tokens (2048 data instances, each instance containing a sequence of 2048 tokens).
For the OLMo-65B model currently being trained, the researchers adopted a batch size warm-up strategy, starting at about 2M tokens (1024 data instances), and then doubling the batch size for every additional 100B tokens until it finally reaches The size is about 16M tokens (8192 data instances).

In order to speed up model training, the researchers used mixed precision training technology, which is implemented through the internal configuration of FSDP and the amp module of PyTorch.
This method is specially designed to ensure that some key calculation steps (such as the softmax function) are always performed with the highest accuracy to ensure the stability of the training process.
Meanwhile, most other calculations use a half-precision format called bfloat16 to reduce memory usage and increase computational efficiency.
In certain configurations, model weights and optimizer state are saved with maximum accuracy on each GPU.
The weights within each Transformer module are temporarily converted to bfloat16 format only when performing forward and backpropagation of the model, that is, calculating the model's output and updating the weights.
In addition, when gradient updates are synchronized between GPUs, they will also be performed with the highest accuracy to ensure training quality.
optimizer
The researchers used the AdamW optimizer to adjust model parameters.
Regardless of the size of the model, the researchers will gradually increase the learning rate within the first 5,000 steps of training (approximately processing 21B tokens), a process called learning rate warm-up.
After the warm-up is completed, the learning rate will gradually decrease linearly until it drops to one-tenth of the maximum learning rate.
In addition, the researchers will also clip the gradients of the model parameters to ensure that their total L1 norm does not exceed 1.0.
In the table below, the researchers compare their optimizer configuration at the 7B model scale to other recent large language models using the AdamW optimizer.

data set
The researchers used a 2T token sample in the open data set Dolma to construct their training data set.
The researchers connected the tokens of each document, added a special EOS token at the end of each document, and then divided these tokens into groups of 2048 to form training samples.
These training samples are randomly shuffled in the same way for each training session. The researchers also provide tools that allow anyone to recover the specific data order and composition of each training batch.
All models that the researchers have released have been trained for at least one round (2T tokens). Some of these models were additionally trained by running a second round of training on the data, but with a different random shuffling order.
According to previous research, the impact of such reuse of small amounts of data is minimal.
Nvidia and AMD both want YES!
In order to ensure that the code base can run efficiently on both NVIDIA and AMD GPUs, the researchers selected two different clusters for model training and testing:
Using the LUMI supercomputer, researchers deployed up to 256 nodes, each equipped with 4 AMD MI250X GPUs, each with 128GB of memory and a data transfer rate of 800Gbps.
Powered by MosaicML (Databricks), the researchers used 27 nodes, each equipped with 8 NVIDIA A100 GPUs, each with 40GB of memory and a data transfer rate of 800Gbps.
Although the researchers fine-tuned the batch size to improve training efficiency, after completing the evaluation on 2T tokens, there was almost no difference in the performance of the two clusters.

Summarize
Unlike most previous models that only provide model weights and inference code, the researchers open sourced the entire content of OLMo, including training data, training and evaluation code, as well as training logs, experimental results, important findings, and records of Weights & Biases, etc. .
Additionally, the team is investigating how to improve OLMo through instruction optimization and different types of reinforcement learning (RLHF). These fine-tuned codes, data and fine-tuned models will also be open source.
Researchers are committed to continuing to support and develop OLMo and its framework, promote the development of open language models (LM), and help the development of the open research community. To this end, the researchers plan to introduce more models of different scales, multiple modalities, data sets, security measures and evaluation methods to enrich the OLMo family.
They hope that through continued thorough open source work in the future, they will strengthen the power of the open source research community and trigger a new wave of innovation.
team introduction
Yizhong Wang

Yizhong Wang is a doctoral student in the Paul G. Allen School of Computer Science and Engineering at the University of Washington under the supervision of Hannaneh Hajishirzi and Noah Smith. At the same time, he is also a part-time research intern at the Allen Institute for Artificial Intelligence.
Previously, he interned at Meta AI, Microsoft Research and Baidu NLP. Previously, he received a master's degree from Peking University and a bachelor's degree from Shanghai Jiao Tong University.
His research directions are Natural Language Processing, Machine Learning, and Large Language Models (LLM).
– Adaptability of LLMs: How to more efficiently build and evaluate models that can follow instructions? What factors should we consider when fine-tuning these models, and how do they affect the generalizability of the model? Which types of supervision are both effective and scalable?
– Continuous learning for LLM: Where is the line between pre-training and fine-tuning? What architectures and learning strategies can allow LLM to continue evolving after pre-training? How does existing knowledge within the model interact with newly learned knowledge?
– Application of large-scale synthetic data: Today, when generative models rapidly generate data, what impact does this data have on our model development and even the entire Internet and society? How do we ensure we can generate diverse and high-quality data at scale? Can we distinguish this data from human-generated data?
Yuling Gu

Yuling Gu is a researcher on the Aristo team at the Allen Institute for Artificial Intelligence (AI2).
In 2020, she received her bachelor's degree from New York University (NYU). In addition to her computer science major, she also minored in an interdisciplinary major, Language and Mind, which combines linguistics, psychology, and philosophy. She subsequently earned a master's degree from the University of Washington (UW).
She is passionate about the integration and application of machine learning techniques and cognitive science theories.
References:
olmo/olmo-paper.pdf