Overview

  • Founded Date October 9, 2017
  • Posted Jobs 0
  • Viewed 7

Company Description

Open-R1: a Totally Open Reproduction Of DeepSeek-R1

Hey there! This article is an introduction to the task, not a claim that we’ve reproduced R1 yet. We’re integrating in the open, so as quickly as we have numbers, we’ll share them. You can follow our progress on Hugging Face and GitHub.

True, however it appears like there’s nothing to be examined as of right now. I assume the ultimate goal is to train a brand-new reasoning model and then utilize the same examination metrics as o1 and the DeepSeek-R1.

Well, there should be at least some sanity check and recognition to guarantee the design was trained correctly.

Oh yes, if you are talking about the assessment variety of deepseek’s design it’s coming extremely soon!

As pointed out in the blog post there is no design called Open-R1 to test at all … not yet anyhow. This is a blog describing that Hugging face will take the R1 Deepseek model, work out how it was constructed as outlined in the paper and from what they released, and then reproduce that process.

in fact this is pretty much how science works … A develops a strategy, discovery or innovation and it is checked by B, C and D to see if it is reproduceable. Thats been the cornerstone of research study now for a couple of centuries.

This blog is not saying they have currently done so … Its a blog laying out an intent to start training a design like R1 and calling it Open-R1.

Also DeepSeek-R1 was just launched last week, and even in their paper they described the calculate hours required. While those are low calculate hours for a SOTA model this does not suggest you can train said model in a week. I ‘d personally like to be able to train a transformer design in a week, but we might need to wait a while for that level of compute technology.

So there are no benchmarks for a design that has not been developed yet right? As outlined in the blog, and again in reply to your question.

However fear not, there is a GitHub Repo already and factors (hell I might join myself), some prelim work done, and a master plan. A good beginning position.

n
@edbeeching
has actually examined the launched models currently

( src: https://x.com/edwardbeeching/status/1884273209136275742)

R1 just trained on o1 outputs, so collectively …/ s. This is what the new AI czars are stating

Hi! This post is an introduction to the task, not a claim that we have actually recreated R1 yet. We will absolutely share the missing piece when we have them, you can anticipate the designs and datasets to be upload in this Hugging Face org and the code to be in this GitHub repo

That’s good and crucial to understand this tremendous buzz that lacks technical understanding and description. Science has to do with reproduction, and if they declare to be open, let them fullfill the open part.

Please do release the training cost.

We will!

Excalidraw Hi n
@bojan2501
thanks, we will undoubtedly be working hard to make certain this training recipe can work for small language designs on customer hardware because not everybody has a cluster of H100s at home:-RRB- The tool we used for the images was Excalidraw! https://excalidraw.com

anticipating it! WTF are your speaking about?

need to be a joke

It’s truly cool to see how the whole open source neighborhood comes together!

Ops …

5.5 M is number reporter in the deepseekv3 tech report (simply the training, not the experiment afaik), for R1 tough to estimate tbh but much less than 5.5 M imo

Historically, they have actually never ever released code or datasets of their LLM training, so I would not anticipate this time to be different. If they would launch it that would be fantastic naturally!

Yes obviously!

So generally you’re asking to change existing censorship with another flavour of censorship?

The code for the designs are inside the design repositories, e.g. for V3: https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py

Hello Team, I’m Ray Bernard, the author and creator of EQUATOR. My research group will be dealing with a paper focused on replicating specific elements of DeepSeek R1. Our goal is to replicate the cold start and provide your group with a dataset that includes COT and other strategies to support these efforts. We like to contribute our work to help. Please let me understand if you find this useful. Best, Ray Bernard https://www.facebook.com/groups/1186310571520299/

Where is the assessment numbers? without it you can’t call it recreation.

8 replies

True, however it looks like there’s nothing to be evaluated since today. I presume the supreme objective is to train a brand-new reasoning design and then use the exact same examination metrics as o1 and the DeepSeek-R1.

That’s quite interesting, I was asking myself why the concerns the author exposed here are not being asked by others? I believe the work they have actually done is remarkable but at the very same time I wonder why they wouldn’t put these missing out on pieces on if they are expected to be totally open.
Why even without reproduction and comprehension of the development they could affect so much the marketplace in this way?

4 replies

Hi! This blog post is an introduction to the project, not a claim that we have actually replicated R1 yet. We will absolutely share the missing out on piece when we have them, you can anticipate the designs and datasets to be upload in this Hugging Face org and the code to be in this GitHub repo

Interesting read, and it is great that we see more effort into this direction: more optimization and less brute force.
Also question what tool did the author use for developing step diagram.

2 replies

Excalidraw I’m so grateful that initiative like this currently exist, I’m gon na attempt to contribute:-RRB- 1 reply

anticipating it! So racist articel

2 replies

WTF are your discussing?

Awesome to have this open recreation started!

For Step # 1 check out https://github.com/open-thoughts/open-thoughts!

https://x.com/ryanmart3n/status/1884284101265612856

Let’s do this thing!

1 reply

It’s truly cool to see how the entire open source neighborhood comes together!

Does anybody know the real training cost of r1? I can’t discover it in the paper or the announcement post. Is the 6M expense reported by media simply the number drawn from v3’s training cost?

2 replies

Ops …

Has anyone asked the DeepSeek team to publish their training data and code, or at least share them independently with an independent replication job like this? Have they rejected such a demand?

A loyal duplication depends upon using the same dataset and hyperparameters. Otherwise, any significant inconsistencies with the published benchmarks would be tough to pin down-whether due to training data distinctions or the duplication method itself.

1 reply

Historically, they have never released code or datasets of their LLM training, so I wouldn’t expect this time to be various. If they would release it that would be remarkable obviously!

In the meantime we need to make best guess quotes and see if we can arrive ourselves.

You offer great replication process of Deepseek thinking training. I will attempt something comparable to it.

This is really excellent information, can we tweak with specific usage case when code is launched?

1 reply

Yes obviously!

Please consider eliminating prejudiced, tainted or unaligned training information and make an effort to get rid of copyrighted works from the crawl from consumption. This will make the model more functional. If you reused anthropic curation checks, this might likewise help, remove obviouslybiased information will likely include a lot of worth. We don’t want another polluted, unaligned open source design, right? And no business would ever use deepseek or a design that reuses it, right?
We appreciate your work for the advantage of humankind, we hope.
Miike C from NJ

1 reply

So basically you’re asking to replace existing censorship with another flavour of censorship?

Can’t wait! Hopefully the design will be uncensored however whatever you can do is alright! Love seeing open source structure itself up. I’m not wise adequate to in fact assist however I can contribute ethical assistance lol

Hello guys, I am even simply attempting to find code for DeepSeek-V2, in order to totally comprehend multi-head hidden attention. You do not appear to have code in Hugging Face even for that. Or am I missing something? Don’t see anything in src/transformers/models. MLA is not properly described in their paper, so it would be essential to have code for this.