DeepSeek R-1 Model Overview and how it Ranks against OpenAI’s O1

DeepSeek is a Chinese AI business “devoted to making AGI a reality” and open-sourcing all its designs. They started in 2023, but have been making waves over the past month approximately, and specifically this previous week with the release of their two latest thinking models: DeepSeek-R1-Zero and the advanced DeepSeek-R1, also referred to as DeepSeek Reasoner.

They’ve released not only the models but also the code and examination prompts for public usage, in addition to a comprehensive paper outlining their method.

Aside from producing 2 extremely performant designs that are on par with OpenAI’s o1 design, the paper has a great deal of important info around reinforcement learning, chain of thought reasoning, prompt engineering with thinking designs, and more.

We’ll start by concentrating on the training process of DeepSeek-R1-Zero, which distinctively relied exclusively on support knowing, rather of conventional monitored learning. We’ll then move on to DeepSeek-R1, how it’s thinking works, and some prompt engineering best practices for reasoning models.

Hey everyone, Dan here, co-founder of PromptHub. Today, we’re diving into DeepSeek’s most current model release and comparing it with OpenAI’s reasoning models, particularly the A1 and A1 Mini models. We’ll explore their training process, thinking capabilities, and some key insights into prompt engineering for reasoning designs.

DeepSeek is a Chinese-based AI business committed to open-source advancement. Their current release, the R1 thinking design, is groundbreaking due to its open-source nature and innovative training methods. This includes open access to the models, prompts, and research documents.

Released on January 20th, DeepSeek’s R1 accomplished excellent efficiency on different standards, matching OpenAI’s A1 models. Notably, they also introduced a precursor design, R10, which acts as the foundation for R1.

Training Process: R10 to R1

R10: This design was trained exclusively using support knowing without supervised fine-tuning, making it the first open-source design to accomplish high efficiency through this method. Training involved:

– Rewarding right responses in deterministic tasks (e.g., mathematics issues).
– Encouraging structured reasoning outputs using design templates with “” and “” tags

Through thousands of versions, R10 developed longer reasoning chains, self-verification, and even reflective habits. For example, throughout training, the design showed “aha” moments and self-correction behaviors, which are uncommon in traditional LLMs.

R1: Building on R10, R1 added numerous improvements:

– Curated datasets with long Chain of Thought examples.
– Incorporation of R10-generated reasoning chains.
– Human preference alignment for sleek reactions.
– Distillation into smaller designs (LLaMA 3.1 and 3.3 at different sizes).

Performance Benchmarks

DeepSeek’s R1 model carries out on par with OpenAI’s A1 designs throughout many reasoning benchmarks:

Reasoning and Math Tasks: R1 competitors or surpasses A1 designs in precision and depth of thinking.
Coding Tasks: A1 designs generally carry out better in LiveCode Bench and CodeForces jobs.
Simple QA: R1 frequently outpaces A1 in structured QA tasks (e.g., 47% precision vs. 30%).

One notable finding is that longer thinking chains normally enhance efficiency. This aligns with insights from Microsoft’s Med-Prompt framework and OpenAI’s observations on test-time compute and reasoning depth.

Challenges and Observations

Despite its strengths, R1 has some limitations:

– Mixing English and Chinese actions due to a lack of supervised fine-tuning.
– Less sleek actions compared to talk designs like OpenAI’s GPT.

These concerns were addressed throughout R1’s improvement process, consisting of supervised fine-tuning and human feedback.

Prompt Engineering Insights

A fascinating takeaway from DeepSeek’s research study is how few-shot prompting degraded R1’s efficiency compared to zero-shot or succinct customized triggers. This aligns with findings from the Med-Prompt paper and OpenAI’s recommendations to limit context in reasoning designs. Overcomplicating the input can overwhelm the model and decrease accuracy.

DeepSeek’s R1 is a considerable action forward for open-source thinking models, showing abilities that match OpenAI’s A1. It’s an amazing time to explore these designs and their chat user interface, which is totally free to use.

If you have questions or want to find out more, examine out the resources connected listed below. See you next time!

Training DeepSeek-R1-Zero: A reinforcement learning-only technique

DeepSeek-R1-Zero sticks out from many other advanced models because it was trained utilizing just reinforcement learning (RL), no supervised fine-tuning (SFT). This challenges the existing conventional approach and opens brand-new opportunities to train thinking designs with less human intervention and effort.

DeepSeek-R1-Zero is the first open-source design to confirm that innovative reasoning abilities can be established purely through RL.

Without pre-labeled datasets, the design finds out through trial and error, fine-tuning its habits, parameters, and weights based exclusively on feedback from the options it produces.

DeepSeek-R1-Zero is the base model for DeepSeek-R1.

The RL procedure for DeepSeek-R1-Zero

The training procedure for DeepSeek-R1-Zero involved presenting the model with numerous thinking tasks, ranging from mathematics issues to abstract logic obstacles. The design generated outputs and was evaluated based upon its performance.

DeepSeek-R1-Zero got feedback through a benefit system that assisted direct its learning process:

Accuracy benefits: Evaluates whether the output is right. Used for when there are deterministic results (mathematics issues).

Format rewards: Encouraged the design to structure its thinking within and tags.

Training timely template

To train DeepSeek-R1-Zero to create structured chain of idea series, the researchers utilized the following timely training template, changing timely with the thinking question. You can access it in PromptHub here.

This design template prompted the design to clearly detail its idea procedure within tags before delivering the final response in tags.

The power of RL in thinking

With this training process DeepSeek-R1-Zero began to produce advanced thinking chains.

Through thousands of training actions, DeepSeek-R1-Zero progressed to solve progressively complicated issues. It discovered to:

– Generate long reasoning chains that enabled much deeper and more structured problem-solving

– Perform self-verification to cross-check its own answers (more on this later).

– Correct its own mistakes, showcasing emergent self-reflective habits.

DeepSeek R1-Zero performance

While DeepSeek-R1-Zero is mainly a precursor to DeepSeek-R1, it still achieved high performance on numerous criteria. Let’s dive into a few of the experiments ran.

Accuracy improvements throughout training

– Pass@1 accuracy started at 15.6% and by the end of the training it improved to 71.0%, equivalent to OpenAI’s o1-0912 model.

– The red strong line represents performance with majority ballot (comparable to ensembling and self-consistency techniques), which increased precision further to 86.7%, surpassing o1-0912.

Next we’ll look at a table comparing DeepSeek-R1-Zero’s performance across multiple reasoning datasets against OpenAI’s reasoning designs.

AIME 2024: 71.0% Pass@1, a little listed below o1-0912 but above o1-mini. 86.7% cons@64, beating both o1 and o1-mini.

MATH-500: Achieved 95.9%, beating both o1-0912 and o1-mini.

GPQA Diamond: Outperformed o1-mini with a score of 73.3%.

– Performed much worse on coding tasks (CodeForces and LiveCode Bench).

Next we’ll take a look at how the response length increased throughout the RL training process.

This graph reveals the length of reactions from the model as the training process progresses. Each “action” represents one cycle of the model’s learning procedure, where feedback is offered based upon the output’s performance, examined using the timely template gone over earlier.

For each concern (representing one step), 16 actions were tested, and the typical precision was determined to make sure stable evaluation.

As training progresses, the design generates longer thinking chains, enabling it to fix progressively complex thinking jobs by leveraging more test-time calculate.

While longer chains don’t always ensure better results, they typically associate with improved performance-a pattern likewise observed in the MEDPROMPT paper (learn more about it here) and in the original o1 paper from OpenAI.

Aha moment and self-verification

Among the coolest elements of DeepSeek-R1-Zero’s advancement (which likewise applies to the flagship R-1 model) is just how excellent the model became at reasoning. There were advanced reasoning habits that were not clearly configured but arose through its support discovering process.

Over countless training actions, the design began to self-correct, reevaluate problematic logic, and verify its own solutions-all within its chain of thought

An example of this kept in mind in the paper, referred to as a the “Aha moment” is below in red text.

In this instance, the design actually stated, “That’s an aha minute.” Through DeepSeek’s chat feature (their version of ChatGPT) this kind of thinking normally emerges with phrases like “Wait a minute” or “Wait, but … ,”

Limitations and obstacles in DeepSeek-R1-Zero

While DeepSeek-R1-Zero had the ability to perform at a high level, there were some disadvantages with the design.

Language blending and coherence issues: The model periodically produced actions that combined languages (Chinese and English).

Reinforcement learning trade-offs: The lack of supervised fine-tuning (SFT) meant that the design did not have the refinement required for completely polished, human-aligned outputs.

DeepSeek-R1 was established to resolve these issues!

What is DeepSeek R1

DeepSeek-R1 is an open-source reasoning model from the Chinese AI lab DeepSeek. It develops on DeepSeek-R1-Zero, which was trained entirely with support knowing. Unlike its predecessor, DeepSeek-R1 integrates supervised fine-tuning, making it more refined. Notably, it outperforms OpenAI’s o1 design on a number of benchmarks-more on that later on.

What are the main differences in between DeepSeek-R1 and DeepSeek-R1-Zero?

DeepSeek-R1 builds on the foundation of DeepSeek-R1-Zero, which functions as the base design. The 2 differ in their training approaches and general performance.

1. Training technique

DeepSeek-R1-Zero: Trained completely with support knowing (RL) and no supervised fine-tuning (SFT).

DeepSeek-R1: Uses a multi-stage training pipeline that consists of monitored fine-tuning (SFT) first, followed by the exact same reinforcement finding out process that DeepSeek-R1-Zero wet through. SFT assists enhance coherence and readability.

2. Readability & Coherence

DeepSeek-R1-Zero: Struggled with language mixing (English and Chinese) and readability concerns. Its thinking was strong, however its outputs were less polished.

DeepSeek-R1: Addressed these problems with cold-start fine-tuning, making reactions clearer and more structured.

3. Performance

DeepSeek-R1-Zero: Still a very strong reasoning design, sometimes beating OpenAI’s o1, but fell the language mixing concerns decreased use significantly.

DeepSeek-R1: Outperforms R1-Zero and OpenAI’s o1 on most reasoning standards, and the reactions are a lot more polished.

In short, DeepSeek-R1-Zero was a proof of idea, while DeepSeek-R1 is the fully version.

How DeepSeek-R1 was trained

To tackle the readability and coherence concerns of R1-Zero, the researchers integrated a cold-start fine-tuning stage and a multi-stage training pipeline when constructing DeepSeek-R1:

Cold-Start Fine-Tuning:

– Researchers prepared a high-quality dataset of long chains of thought examples for preliminary monitored fine-tuning (SFT). This data was gathered utilizing:- Few-shot triggering with in-depth CoT examples.

– Post-processed outputs from DeepSeek-R1-Zero, refined by human annotators.

Reinforcement Learning:

DeepSeek-R1 underwent the very same RL procedure as DeepSeek-R1-Zero to improve its thinking capabilities further.

Human Preference Alignment:

– A secondary RL stage improved the design’s helpfulness and harmlessness, making sure better positioning with user needs.

Distillation to Smaller Models:

– DeepSeek-R1’s thinking capabilities were distilled into smaller, efficient models like Qwen and Llama-3.1 -8 B, and Llama-3.3 -70 B-Instruct.

DeepSeek R-1 benchmark efficiency

The scientists evaluated DeepSeek R-1 throughout a variety of standards and versus top models: o1, GPT-4o, and Claude 3.5 Sonnet, o1-mini.

The standards were broken down into numerous categories, revealed listed below in the table: English, Code, Math, and Chinese.

Setup

The following criteria were used throughout all models:

Maximum generation length: 32,768 tokens.

Sampling setup:- Temperature: 0.6.

– Top-p value: 0.95.

– DeepSeek R1 exceeded o1, Claude 3.5 Sonnet and other models in the majority of reasoning standards.

o1 was the best-performing design in four out of the five coding-related benchmarks.

– DeepSeek performed well on innovative and long-context task job, like AlpacaEval 2.0 and ArenaHard, exceeding all other designs.

Prompt Engineering with reasoning designs

My favorite part of the article was the scientists’ observation about DeepSeek-R1’s sensitivity to prompts:

This is another datapoint that lines up with insights from our Prompt Engineering with Reasoning Models Guide, which referrals Microsoft’s research study on their MedPrompt structure. In their research study with OpenAI’s o1-preview model, they discovered that frustrating thinking models with few-shot context broken down performance-a sharp contrast to non-reasoning designs.

The crucial takeaway? Zero-shot prompting with clear and succinct directions appear to be best when using thinking designs.