rnj-1.5

style="vertical-align: middle;" src="https://img.shields.io/badge/%F0%9F%8C%90%20Website-essential.ai-4b9fe1?color=4b9fe1&logoColor=white"/>

  style="vertical-align: middle;"
  src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-rnj--1-ffc107?color=ffc107&logoColor=white"/>

  style="vertical-align: middle;"
  src="https://img.shields.io/badge/Discord-Essential%20AI-7289da?logo=discord&logoColor=white&color=7289da"/>

  style="vertical-align: middle;"
  src="https://img.shields.io/badge/Twitter-essential__ai-white?logo=x&logoColor=white"/>

We introduce rnj-1.5-instruct, a long-context follow-up to rnj-1-instruct, that extends the context window from 32k to 160k. For more context and details about the Rnj-1 family, and rnj-1-instruct in particular, please see this page and our blog.

rnj-1.5-instruct extends rnj-1's long context abilities beyond 32k, scoring 77% on RULER on a 128k context window. This release also offers stronger coding abilities on a wider range of harnesses. We improve our SWE-Bench Verified performance on mini-swe-agent by 5% and we outperform, by a significant margin, the best known 8B model results on the SWE-Agent harness, achieving a 40% resolve rate.

The improvements in rnj-1.5 emerge from our work in a few key areas:

1. Architecture: To ease inference compute and storage costs that grow with sequence length for global self-attention, we explore block-local attention layers [5, 6] which carry fixed compute and storage cost per position. Following [4], we interleave block-local self-attention with global attention, keeping layers predominantly local and a few global layers to enhance associative interactions over long distances. After careful ablations, we discovered a local-global layer pattern LLLGLLLGLLLGLGGGGGLGLLLGLLLGLLLL, where L and G stand for block-local and global self-attention layers respectively, that lumps global layers in the middle and retains the model's global attention capabilities while gaining the latency benefits of local self-attention. Our results support the findings from GLM-5 [1].

2. Evals: We discovered that RULER's needle-in-a-haystack (NIAH) task has two fundamental issues. (a) The predominant focus on text-based evaluation overlooks measuring long context abilities necessary for coding tasks. (b) The needle is a foreign random string in a coherent English essay, which shifts the task from identifying an indistinguishable needle to picking the "odd one out." We thus created a granular NIAH eval called "lookback evals" from GitHub repositories. In lookback evals, the needles are semantically camouflaged within the haystack. We add a canary marker to the needles to distinguish between the model's recall from its weights versus retrieval capabilities. For granular insights on long-context abilities, we bucket performance for needle positions over (0–8k, 8k–16k, 16k–32k) distances from the end. Rnj-1.5's performance remains consistent across buckets, showcasing stronger retrieval capabilities throughout the long context.

3. Long context mid-training data:

   1. STEM: We convert a large collection of science-focused PDF documents into text using OlmOCR 2 (which gave strongest results on OCR benchmarks) unlocking ~75B high quality tokens [3].
   2. Code: Prior work [2] has shown that training on repository-level documents improves long-context code understanding. Following this approach, we concatenate files from code repositories in lexicographic order to expose the model to longer, repository-scale code contexts during mid-training. We also include repository-level fill-in-the-middle examples, using surrounding files and broader project structure to support infilling.

4. Long context tasks:

   1. QA needles — Similar to data created in [8], we take a document, split into paragraphs, and select a range of paragraphs adding up to 2k tokens. We then ask a strong model to create a QA pair that tests the reading comprehension. Task is to generate answer given question and document. We pack the pairs.
   2. Practice Problems — we create synthetic QA pairs with reasoning from practice problems in our PDFs. The context contains complete paragraphs, always ending with the page preceding the practice problem page.
   3. Paragraph Swaps — Similar to permutation data created in [9], we take (two|three) paragraphs in the document and (swap them|permute them so that each paragraph is in one of the others' place). If they were put back in their proper locations, the document would read more coherently and consistently. The task is to determine which paragraphs were permuted, and (without explanation), output (the paragraph | first five words of the paragraph) that should appear (earlier|later|first|second|third) in the document.
   4. Common Word Extraction (Frequency Thresholds) — Given a document (list of words), the task is to output a list of all words (longer than four characters) that occur with frequency above some threshold, sorted in alphabetical order. The threshold is chosen randomly subject to a constraint that the resulting list of words is not too long (≤ 10). Our task has some conceptual similarities to the CWE synthetic aggregation task used in [7]; however, our task does not require model prompting to generate QA pairs and is therefore simpler yet effective.

5. Expanded SWE trajectories: We created a synthetic task generation pipeline that allowed us to mine ~200k SWE tasks from 3500 GitHub repositories. Each task consists of the issue description, dockerized repository at failing commit, pass-to-pass, and fail-to-pass tests. Three teacher models were used to generate ~600k synthetic trajectories in the mini-swe-agent environment, including both resolved and unresolved trajectories.

6. **