Hi, everyone. My name is Mark, and I lead research at OpenAI. Yesterday, we took O1 out of preview, and we launched it in Chat GPT. We're soon going to launch it in the API. If you haven't been following O1, it's our latest series of model improvements that allow the models to think for a while before they come back with a response. Today, we're really excited to preview our latest advancement in our model customization program.
It'll let users fine-tune O1 on their own datasets. And again, this isn't standard fine-tuning. This is reinforcement fine-tuning, which really leverages the reinforcement learning algorithms that took us from advanced high school level to expert PhD level for your own use cases. I want to stress again that this is a preview of something that we're going to launch publicly next year.
But if you are a university or you're a researcher or you're an enterprise, we'll give you some information on how you can access our output program later. So why would you want this thing? Well, it allows you to take your golden datasets and turn them into unique offerings that will give you the same magic that we have for your own users and your own customers.
So I'll let John, Julie, and Justin say a little bit more. Yeah, hello everyone. Yeah, my name is John Allard and I'm an engineer here at OpenAI. Hi everyone, I'm Julie Wong. I'm a researcher here at OpenAI. I'm Justin Reese. I'm a computational biologist at Berkeley Lab.
Today, we're so excited to be introducing this new way of model customization for our O1 series of models, reinforcement fine-tuning, or RFT for short. For the first time, developers, researchers, and machine learning engineers will be able to use reinforcement learning to create expert models capable of excelling at their specific tasks within their domain.
We believe that any field which requires deep expertise in their AI models stands to benefit. So if you work in, say, legal, finance, engineering, insurance, this one's for you. For example, we recently partnered with Thomson Reuters to use reinforcement fine-tuning to fine-tune O1 Mini to be a legal assistant in their co-counsel AI. This tool assists their legal professionals in accomplishing some of their most analytical workflows.
Yeah, so some of you will be familiar with the supervised fine-tuning API that we launched early last year. And supervised fine-tuning is really powerful. What you're trying to do is get the model to replicate features that it finds in input text or images. And this is great if you want to change the tone or the style or the response format of the model.
Now, with reinforcement fine-tuning, or reindeer enforcement fine-tuning, I should say. With reinforcement fine-tuning, it's actually different. So you're not just teaching the model to mimic its inputs, what you're teaching it to do is to learn to reason in entirely new ways over custom domains.
And the way this works is that when the model sees a problem, we give it space to think through the problem, and then we grade the final answer from the model. And then using the power of reinforcement learning, we reinforce lines of thinking that led to correct answers, and we disincentivize lines of thinking that led to incorrect answers. And what you'll see is that with as little as a few dozen examples, the model will learn to reason in new and effective ways over custom domains.
That's crazy that you can do that with just 12 examples. That's not something you can do with regular fine tuning. Yeah, exactly. Yeah. I mean, in the space of large language models and large machine learning, a few dozen examples is basically nothing. Yeah. So for the first time, our model customization platform will support reinforcement learning. And notably, this is the same technique that we use internally at OpenAI to train our frontier models, like GPT-4.0 and the O1 series.
One area with many exciting applications is scientific research, but don't just take our word for it. That's why we're joined today by Justin Reese. Justin is a researcher at Berkeley Lab, and one of his areas of study is using computational methods to understand the genetic causes underlying rare diseases. Justin, thank you so much for being here. Do you mind telling us a little bit more about your research and how reinforcement fine-tuning might help?
Sure, thanks. It's great to be here. So one of the areas of my research is rare genetic disease. So contrary to the name, rare genetic disease is actually not rare. So any one rare disease is rare, but if you put them all together, they're actually quite common. And so we're talking about 300 million people globally who suffer from a rare disease.
And what's more, these people often have a long diagnostic odyssey of months to years before they find out about their condition. It's like the whole population of the US. Yes, it's not a small number of people. And so what we're working on is better computational tools and methods to really research what's important and to help us understand and treat these diseases. So we do our work in kind of an academic setting and learning more about the rare disease and their causes. And the hope is we'll be able to advance the health care for these folks going down the line.
And now, assessing neurodisease is kind of hard because you kind of have to have two things. You have to have sort of expert domain knowledge about the medical side of things, and you also have to have sort of systematic reasoning over the biomedical data. And this is an area where we think that the O1 model can really help us out with its reasoning capabilities. That makes a lot of sense. You know, our large language models have domain knowledge, and our O1 models are really systemic reasoners. So it seems like now there's a pretty good computational method for addressing some of these. That's right.
Can you tell us a little bit more about the data sets that you're using? Sure. So this was sort of a collaborative effort between our group and Charite Hospital in Germany and Peter Robinson's lab and the Monarch Initiative. And what we did really was to extract disease information from hundreds of scientific publications that were case reports about rare disease.
And so we sort of curated the information, and that's lists of signs and symptoms that were present in the patient and that were excluded in the patient, and then of course the disease that they had, and importantly for this conversation, the causative gene that was mutated that was causing the problems in these folks. I see. So you and maybe some doctors are trying to figure out, given a patient's symptoms, what gene might have mutated to cause those symptoms?
Yeah, that's right. And so something we've been working on together with the OpenAI team is training the O1 models to reason more effectively about the causes of disease.
Incredible. Thank you, Justin. We're now going to give you a preview of reinforcement fine-tuning at work. And not to steal any thunder, but we're going to take O1 Mini and make it exceed the performance of O1 on this task. That's the O1 that we just launched yesterday. And this matters so much because O1 Mini is a smaller, faster, and cheaper model than O1.
Yeah, so using Justin's dataset, we're going to show you can just drastically improve the performance of O1 Mini on this task where, given a list of symptoms, you're trying to predict which gene might be responsible for the genetic disease. And so to give an overview of this process, we're going to start by looking at datasets that are used to train the model and graders that are used to evaluate the model.
And then we're going to launch a training job on OpenAI's training infrastructure. And finally, we'll evaluate the resulting fine-tuned model so we can see how it's improved over the base model that we started with.
So to start us off, we're going to jump over to the OpenAI development platform. And we're going to go ahead and we're going to create a new model. So we've had supervised fine-tuning for a bit over a year now. What we're going to do is we're going to select reinforcement fine-tuning. Now we're going to be training 01, so we'll select that as the base model. And now we need to upload a training data set. And now training data sets, they're just JSONL files, which is just a file where each line in the file is an example that you want the model to be trained on.
For this case, Justin and his colleagues assembled a data set of about 1,100 examples. And so I'll go ahead and upload that one.
And just so that we get a really good feel for how this data set works and what this task is, we'll zoom in on an individual data point really quickly. And so this is what an individual data point looks like. And there's really three important things here. So the first is the case report. And this is a description of the patient and the patient's symptoms. So we see that the patient was a 51-year-old woman. The disease onset was not specified. We have a list of symptoms like hypertellerism, hyperparathyroidism, and others.
As Justin said earlier, we have the absent symptoms. These are the symptoms that are not present. And this is important because it helps the model to rule out genes that it might think would otherwise be responsible for the symptoms that are present.
Next, we have the instructions. And I'm sure if you're watching this live stream, you're familiar with prompting. And so all we're doing here is just prompting the model for what we want it to do for this task. And so what we're saying is, you know, given the list of symptoms and the case report, can you list all the genes that you think might be responsible for the genetic disease that you think is present? And then we also ask it to provide an explanation for why it thinks those genes might be responsible.
Finally, we also have the correct answer. And so this is the gene that we happen to know is responsible, but importantly, we're not showing this to the model during the training process. That would be cheating. But we're using it internally during the training process to grade the model's outputs or to check if the model's correct. This is a pretty hard task. I definitely have no hope of answering this question. Yeah. I mean, you can tell that we've come up far away from just trying to count the number of R's in the word strawberry. Yeah.
So now, when we give the model this prompt, this case report, and these instructions, the model's going to output something like this, which is a list of genes that it thinks might be responsible. And importantly, the genes are in sorted order, where the first gene in the list is the one that it thinks is most likely to be responsible, the second one in the list is the one that it thinks is second most likely, and so on and so forth.
Cool. So we'll hop back over. And so next we need to upload some validation data. And validation data, it's going to be in the exact same format as the training data. But importantly, there's no overlap.
in the correct genes between the validation data set and the training data set. And what that means is that the model can't cheat. It has to, or it can't learn to just memorize a list of symptoms and associate those with the gene. It has to actually generalize from the training data set to the validation data set.
Gotcha. So I mean, where does the reinforcement part come in? We talked about grading. Is that part of the process here? Yeah, that's a really good question. So grading is done by this concept of graders that we're introducing here. And so graders are really simple. What a grader does is it takes the output from the model, and it takes the correct answer, and it compares them, and it returns a score between 0 and 1. And so 0 means that the model did not get the answer correct at all, and 1 means the model got the answer correct. And you can also give partial credit. So it can be anywhere in that range.
So for this specific task, we have a grader that looks like this. So it takes the correct answer that we happen to know, and it takes the output from the model, which is the list of genes, and it produces a score. So in this case, FoxE3 is the correct answer. It was second in the list of genes, and so it gets a score of like 0.7.
I see. So if it had instead said Foxy 3 was first in the list, I would have gotten a grade of 1? Yeah, exactly. And then as it gets further and further along the list, the score kind of gradually decays to 0. Ah, nice. Makes sense. But what if I have a task that isn't grading a ranked list? Do we have other graders that are more general?
Yeah, yeah. So we're supplying kind of a collection of graders that we think pretty effectively cover the space of possible intents that you might have while doing reinforcement fine-tuning, and we're always adding more. Yeah, and eventually, we're going to hopefully let you define your own graders. Yeah, yeah. Maybe like upload a Python file or something and do some custom grading. Yeah. Cool. So we've defined our training data set. We've defined our validation data set. Let me go ahead and copy in the grader really quick.
And now OpenAI allows you to set, you know, we allow you to customize these fine-tuning runs by setting hyperparameters, but we set some pretty good defaults. So I'm just going to go ahead and click Create here. Now, what this is doing is, you know, we've just kicked off a training job.
And so the really cool thing is that you bring the data set and you bring the grader. And these are the places where you really have domain expertise and where you can really contribute to this problem. And then you get to leverage the full power of OpenAI's reinforcement learning algorithms and our full distributed model training stack to customize a frontier model for your use case. So as a user, I just have to bring my data set and grader and OpenAI takes care of everything else. Yeah, exactly. Yeah.
So reinforcement fine-tuning jobs can take anywhere from a few hours to a few days to run. So we're going to jump over to a job that I ran earlier this week on the same data set, just so we can kind of see the results. So I'll jump over here.
So I had this job that I ran earlier this week. It completed successfully. It produced a fine-tuned model for us. And there's one thing that I want to look at, which is the validation reward score. And so what this is, it's the average score from the grader on the validation data set and how it changed over the course of the fine-tuning run.
And so what we can see is that the score is going up. And as we said earlier, since there's no overlap in genes between the training data set and the validation data set, it means that the model really learned to generalize on our task. It wasn't simply memorizing a list of symptoms and mapping those to genes. So while this is cool, the chart goes up and to the right, which is what we like to see, it'd be nice if we could get a better feel for how the model has actually changed during the fine-tuning process. And so we'll take a closer look at that now.
All right, so we're going to pop over to the evaluations dashboard, which is a product in our developer platform that we launched earlier this year. There's a lot of numbers, but don't worry, we're going to go through all of them. So I've set up three different runs here. The first one was a run against our O1 model, which we released yesterday. The second was against O1 mini, which was the starting point of our fine-tuning job. And then finally, the reinforcement fine-tuned O1 mini.
Now, we looked at the reward going up and to the right, but what does that actually mean for this task? I've set up three different evaluations to sort of assess that. The first one is top at one, which is how often is the correct answer, the very first item in the list? Top at five, which is how often is the correct answer in the top five elements of the list? And finally, top at max.
Did we at all put the right answer in our list? And so looking at top at one, we can see that our starting point, O1 Mini, got 17% on our data set of about 200. O1 got 25%, so it's doing better. But then our fine-tuned O1 Mini got 31%
I took a screenshot of this and I put it into ChatGPT and asked it to make me a plot, a Christmas-themed plot. And here's a nice visualization of those nine numbers that we saw earlier. So you can see our starting point, 01mini, across top at 1, top at 5, and top at max.
our O1 model, and then finally, our best performing model, which is this O1 mini fine tune here in dotted red line. So looking at these results, what do you think, Justin? Well, I think this is pretty impressive performance, and especially the increase in the validation data, because that implies that the model is learning something generally about how to reason over these kind of data, which is pretty exciting.
And so an obvious question you might ask is, how is this doing compared to existing bioinformatics tools? And I don't really have an apples-to-apples comparison, because typically in this kind of experiment you would provide genomic sequencing data, and we haven't included that here. But the sort of open-ended querying of models here over incomplete symptom lists is new and exciting, I think.
Great. So these are aggregate statistics, but let's look at the actual model responses. So I'm going to pop on over to this data tab. Let's filter by the passes. And so here is the input that we're giving to the model. So the problem, as John described earlier, is to identify genes that may be responsible for a set of observed symptoms.
We ask the model to output a dictionary containing both a string that explains, you know, why did I pick these genes, and of course, the genes themselves in ranked order. And then finally, we have the symptom list as well. So this patient presented with subependymal nodules, seizure, yeah, and a couple other things. We then run our model. So this was our O1 model.
This was our fine-tuned O1 mini model. We gave it that input, and now the output is this dictionary that we described earlier. So, reasoning. The combination of subependymal nodules, seizure cortical tubulars, are indicative of this complex, which is commonly caused by mutations in these genes.
It lists a couple of other potential ones, and then it says TSC2 is the most likely candidate. And if we scroll back on over to our answer, we'll see that TSC2 is, in fact, the correct answer. So that allowed us to get a pass on TopAt1, TopAt5, and TopAtMax.
So looking at this output, Justin, is this a useful output for the model to be giving back? Yeah, absolutely. So it's particularly useful to see the model's reasoning. And that's a big contribution here. And also, obviously, the rank list of answers. So even if the correct answer is not first, you can look at all the possibilities. And it's also great to see the fine tuning improves the performance in the rank list of possible answers. So that right answer is getting closer to one. So that's gratifying.
Justin, kind of zooming out a little bit, like how does reinforcement learning shape your field? Can you talk about some trends in biology? Sure. So I think there's a lot of interest in the research community in using these models for these kind of tasks. And so the feeling for this particular use case is that the best solution in the near term is probably a hybrid solution between existing bioinformatics tools and these
models like O1. And so this, I think, is excellent progress in sort of characterizing the strengths of these models and also how we can use tools like fine-tuning to improve performance. And so, like I said, there's not really a comparable benchmark to compare the two, but it's definitely progress in how we can use these models to kind of understand disease. And then, you know, in a larger sense, just sort of how we can incorporate these models into a workflow that will eventually improve healthcare for these folks.
Amazing. Thank you, Justin. So while we've just shown you an exciting application of reinforcement fine-tuning in scientific research, this is a general purpose technique. We've seen promising results in datasets from biochem, from AI safety, from legal, and from healthcare as well.
We can think of hundreds of more examples or tasks that we can use this model on, but we know that you can probably think of many more. So that's why we're so excited to be expanding our Alpha program today to enable more people to push the boundaries of the capabilities of our O1 models on the tasks that matter the most to them.
Yeah, so we've been working with a small group of trusted partners to really test out reinforcement fine-tuning, and today we're expanding alpha access via what we're calling the reinforcement fine-tuning research program. So this program is ideal for organizations who are currently working on very complex tasks with teams of experts and who think that they might benefit from AI assistance on these tasks.
So, you know, if you're interested in applying for one of these limited spots, you can find a link to the application in the description of this live stream. And as Mark said earlier, you know, we plan on launching this product reinforcement fine tuning publicly early next year. Yeah, we're all really truly excited to see what you do with reinforcement fine tuning. And really speaking as a researcher, there's nothing that makes us happier than seeing our models being adapted and used to advance science knowledge in the real world. Do you have a joke for us today? Well, as so happens, I do.
As it's becoming tradition, I have a Christmas-themed joke. So, you know, we live in San Francisco. Self-driving vehicles are all the rage. And actually, Santa's been trying to get in on this, too. He's trying to make a self-driving sleigh, but he, for some reason, his models just keep not identifying trees, and the sleigh is hitting trees left and right. Do you guys have any guesses why?
No. He didn't pine-tune his models. Oh, geez. Okay. All right. Please join us next week. We'll have a lot more to share. Thank you.