How good is Codex?

OpenAI recently released Codex (paper), a large-scale language generation model trained on public programming source code from GitHub. Its most prominent use so far is GitHub Copilot, which is an extension for VSCode¹ that autocompletes code being edited. However, there are many uses for Codex beyond code completion. OpenAI recently ran the OpenAI Codex Challenge, where competitors were tasked with solving various programming tasks, and could use Codex to help them. I participated in the challenge and got 75th place². Due to some overloaded servers during the challenge, many people were unable to actually do actually compete, so that ranking is mostly due to me figuring out the best times to reload to get the challenge to work. After the challenge, OpenAI sent me an invite to try out the Codex model, which I used to do the testing in the post.

In this post, I find that Codex understands the Linux source code, can write simple scripts, can sometimes write commit messages, can write good explanations of code and provide good explanations of red-black trees, can be prompted to make the code it writes more secure and more.

I’ve used a variety of languages in this post. OpenAI says that Codex has support for “JavaScript, Go, Perl, PHP, Ruby, Swift, TypeScript, SQL, and even Shell”, and in my experience Codex does work quite well for all of those listed. I’ve also seen it write code in other languages, such as C, C++, and Rust fairly well. Codex’s ability to write in a language is mostly based on how much training data it has seen in that language, so more obscure languages don’t work as well. Some languages are also just easier for the model to work with: I find that in general, Codex works better with languages that make more things explict. Codex struggles to write code that passes Rust’s borrow checker, since whether some code passes the borrow checker is a very complex function of the code (especially with non-lexical lifetimes). It is comparatively easy to check if some code is syntactically valid, and so Codex very rarely writes code that is not syntactical.

I will evaluate the strengths and weaknesses of Codex, determine what tasks it is well-suited for, and determine what the limits of what Codex can do. I will also explore many potential use-cases for Codex. While source code language models still have a long way to go, Codex is already very exciting, and I can’t wait to see applications of it, and how the model itself can be improved.

Overall, I think that Codex’s main strength currently is for writing simple pieces of code that don’t require much understanding of any particular niche topic, or having a deep understanding of the codebase. Codex is able to write code that it has seen before in different ways and languages, and weave them together to create to code. But it isn’t yet able to write new algorithms, find clever optimizations, or consistently generate code without logic bugs. Hopefully over time better models will be developed that can exceed the ablities of Codex. There is still a long way to go in the development of source code language models, but Codex is already exicting today.

In this article, the input to the Codex model is bold, and the rest of the code is the output of the model. I also include the model parameters used to generate the output. Note that there is an element of randomness to the model’s generations when the temperature is non-zero, so trying them yourself in those cases will give different results. Also note that in a few cases Codex generates the actual email addresses of other people, which I replaced with [email removed] to avoid spammers. Interestingly, the Copilot extension checks for email addresses and similar information in generated outputs, and discards generations that contain them. Here, the first two lines of this script were provided by me, and the third was generated by Codex.

The model

There are actually two Codex models currently, davinci-codex and cushman-codex. The cushman-codex model is a smaller version of the davinci-codex model, which runs faster but at the cost of lower quality results. Despite this, cushman is actually quite competitive with davinci, and I have a hard time finding inputs where cushman did significantly worse than davinci. Often davinci-codex and cushman-codex do the same thing in different ways. GitHub Copilot seems to use a variant of cushman for cases where faster completions are desired (such as autocomplete), and an undocumented “earhart” model when slower completions are desired (such as when a user clicks a button to explictly request completions). Following the naming conventions of the current models where the first letter represents quality, earhart would provide results that are higher quality than davinci. Alas, I am unable to test this.

Various parameters can be used to control how the model generates it’s output. Tuning these parameters can be quite helpful for getting the desired results. Since Codex is based on GPT-3, the parameters are the same as GPT-3.

Temperature/top-P

The temperature is a value from 0-1 that controls how likely the model is to choose tokens that aren’t the most likely ones. This is useful when you want the outputs to be more random or creative. While higher temperatures are often good for generating natural language text, they work less well for generating source code. This is because source code is more structured and “obvious” than natural language. Higher temperatures will result in outputs that attempt to perform the task in unexpected ways, but it it usually better to do things the obvious way than to devise some clever way to do that. In contrast, with natural language, stating things in uncommon ways is a good thing, since reframing already known things in new ways can help understanding.

Compare the outputs at the minimum temperature of 0, where it provides a short, working snippet:

…and the maximum temperature of 1, where the code is more complex and doesn’t actually work:

This is mentioned in the documentation for Codex as well:

Setting the API temperature to 0, or close to zero (such as 0.1 or 0.2) tends to give better results in most cases. Unlike GPT-3, where a higher temperature can provide useful creative and random results, higher temperatures with Codex may give you really random or erratic responses.

You can also specify a top-P value from 0-1 to achieve similar results to changing the temperature. Changing Top-P values gives similar results to changing temperature values. The API documentation says:

We generally recommend altering this or temperature but not both.

Copilot dynamically chooses the number of times to sample the model (not sure how exactly), and uses a temperature between 0 and 0.8, with a higher temperature resulting when there are more samples.

Frequency/presence penalty

The frequency and presence penalties can be used to force the model to behave in certain ways by discouraging the model from selecting certain tokens. Increasing the presence penalty discourages the model from producing tokens that it has already produced, and the frequency penalty discourages tokens based on how much they have already been in the text. This is useful for useful for stopping the model from getting into a repetition trap: often Codex will get stuck in a loop, repeating the same thing repeatedly. This isn’t unique to Codex: GPT-3 can also get stuck repeating the same thing. For example, here it gets stuck repeating the last two #includes forever:

Increasing the frequency penalty fixes the issue:

Writing prompts

If Codex isn’t giving the results you desire, you can often improve the result quality by designing a better prompt to push Codex towards the desired output. It often seems odd that you have to prompt Codex to write good code to make it do so, but consider that Codex was trained on a lot of low-quality code. As such, the output of the model reflects the fact that most code you can find on GitHub is not designed with security in mind. This happens most in languages that have a lot of insecure written in them, like PHP³. Here, Codex generates PHP code that is vulnerable to SQL injection.

With a bit of prompt engineering, we can fix the issue by asking Codex to do the insert “safely”⁴:

Specifying the language

If all you give to Codex is a single comment, it is unable to guess what language to write the code in. You can usually just specify the language: instead of “Do the task”, say “Do the task in Rust”. If your language frequently has shebangs at the top of the file (like #!/bin/bash or #!/usr/bin/env python3) those also work well for specifying the language.

Long-winded comments

Often Codex will write long-winded comments about what it is doing. While these are sometimes desirable, made-up information about the author and license is typically not desired.

You can get around this by making sure Codex can’t continue the header comment by adding empty lines after comments to stop Codex from being able to continue them.

Codex actually generated a longer script that I cut off that is really bizarre, but at least it doesn’t include a made-up author or license!

Use cases

Writing code

Codex is good at writing code, since that’s what the main intent of it is. But how much complexity can Codex handle in the code it writes? Let’s take a look at a few examples.

GitHub Copilot

I don’t have access to GitHub Copilot, but anyone can download and install the extension. It just doesn’t work without being able to authenticate with the backend. Taking a glance at the Copilot source code, it seems that it is doing a lot more than just sending the current line to the Codex model. Copilot actually tries to parse the code into an AST, and uses that information to help generate results. It seems it is using that to decide when to generate results, and when to stop the generation. This prevents issues with Codex auto-completing the current function, and then deciding to write more functions after that. This is probably why Copilot only supports a couple languages, while Codex can handle more: Copilot hasn’t implemented parsing support for any other ones.

Context

Codex only supports up to 4096 tokens (where 1 token is around 4 characters of English) of combined context and output. As such, it doesn’t work as well with large codebases, since it can’t know about everything that it can do in a codebase.

This is actually somewhat mitigated for code that Codex was trained on (and especially for code that is popular enoguh that Codex has seen it multiple times), since in those cases Codex knows what the codebase looks like. When Codex is prompted to write code for Linux kernel, or kernel modules, it can write code that looks like other code in the kernel, and seems like actual kernel source code:

Note that most of the files it #includes actually have different names in the real kernel source code. But the comment it writes for mmu_init is very impressive: it’s not a direct copy of some other comment in the kernel source code, but is actually better then similar real comments. For comparision, here’s an actual MMU initializer from the kernel source:

/*
 * MMU_init sets up the basic memory mappings for the kernel,
 * including both RAM and possibly some I/O regions,
 * and sets up the page tables and the MMU hardware ready to go.
 */
void __init MMU_init(void)
{

Codex produced what seems to be a better header comment! Codex’s ability also raises some interesting copyright questions: does Codex know enough about the Linux kernel to be a derivative work of it? It certainly knows a lot about the structure and style of the kernel.

Codex is quite good at working with the Linux kernel. Here, I instructed Codex to write a driver. The generated code looks a lot like a real kernel driver. It delcares the right metadata, uses goto-style error handling, and overall looks like the sort of code you would see in the Linux kernel.

Single-file scripts

Codex works best for generating single-file scripts. Here is an example I find impressive: I gave Codex a vague prompt, and it filled in the blanks. It’s not quite right, since it doesn’t authenticate the request to the Gmail API, but adding proper authentication could be done quite easily (HTTP Basic Auth is supported).

Subtle errors

Often Codex will write code that looks right at first glance, but actually has subtle errors. Try to spot the error:

In that snippet, it overwrites the existing contents of ~/count.tsv by using > instead of appending with >>. Changing the prompt even slightly fixes it:

Another one: here, Codex generates the nth prime, not the the nth mersenne prime.

Writing commit messages

We can try feeding Codex a diff and asking it for a commit message. Codex often struggles here due to lacking enough context. I picked a commit with a commit message that is fairly clear from the diff, “Deprecate llvm_asm!” Can Codex write a commit message?

The commit message looks like the sort of one you would see in rustc, but it doesn’t make sense here. rustfmt is unchanged by the provided diff. I’ve tried writing a better prompt to get Codex to write better commit messages, but I haven’t found a prompt that can convince Codex to consistently write good commit messages. Adding example messages for other commits can sometimes give better results, but sometimes Codex will just regurgitate one of the commit messages it saw before. Asking for an explanation of the commit, instead of asking for a commit message directly, sometimes gives interesting results.

This is quite impressive: it managed to extract a link to a relevent GitHub issue from the context lines, add wrote a good explanation. The commit message it wrote after the explanation was quite good as well.

Writing diffs

Let’s try that in reverse: generating a diff, given a commit message.

That is a correctly formatted diff, but for a file that doesn’t exist. I didn’t expect Codex to do well here, since it doesn’t know what the file it needs to modify currently looks like. Let’s see if giving it some context will help. This is making it easy for Codex, since I’ve removed all the other lines in the file. All Codex needs to do is figure out what lines to add:

That doesn’t do it either. Perhaps there is some good way to prompt Codex into writing diffs, but I haven’t found any good way to tell Codex to make a change to some existing code.

Explaining code

Another interesting use-case for Codex is getting it to explain code in natural language. I had to go through many iterations of the prompt in order to get the type of explanations I liked: by changing the prompt can, you change the type of results you get. Replacing “software developer” with “second grader” makes the explanations simpler. The “at a high level” part helps discourage the model from just doing a line-by-line explanation. I also find that increasing the temperature is beneficial here, since explanations are less structured than source code.

Let’s try something harder: a data data structure function from the Linux kernel. This explanation is very impressive: the code doesn’t even explictly mention that it is a red-black tree. The only clue is the rb in the function name, and the contents of the function. Despite this, Codex delivers an explanation of red-black trees. Note that the __rb_insert_color function is totally made up by Codex: it doesn’t actually exist in the kernel source code. But Codex explains it anyways!

Weaknesses

Stopping

Codex will often generate more than the desired code. Codex will often generate a function from a comment, and then start generate whatever functions it thinks should be written next. The easiest way to work around this is to stop generating tokens once you see something like } at the same indentation level as the prompt. You could also be more advanced and parse the Codex outputs to stop when there is a full AST for a function, as Copilot seems to do. In this blog post, I have manually decided where to cut off outputs.

Dependencies

Codex will often reference other code that don’t actually exist. Here, it references a is_prime function without actually writing it.

Most of the time we can fix this by just prompting Codex with the start of the missing function declaration:

Now the code works.

Switching languages

Sometimes Codex will suddenly go: “Suprise! That was actually just part of a string in a different language”. I find this happens most with languages that are often embedded, like LaTeX and HTML.

You can work around this by stopping at the right point: for LaTeX, when you see \end{document} just stop requesting tokens. Some prompt engineering indicating that it’s just a single file might also help.

Inability to backtrack

There’s no easy way to tell Codex to generate in the middle of some text. You either need to get it to generate at the end, or otherwise convey the text after the generation.

That demo was cherry-picked for a good result. Most of the tests I tried didn’t work out well. Most of the time Codex isn’t able to figure that sort of problem out. Here is a fairly typical result, where Codex gives an answer that doesn’t make any sense (a and b aren’t defined here).

Perhaps there is some better way to prompt Codex to generate code in the middle of other code, but I haven’t found any.

Conclusion

(edit: see the introduction at the top for what you are looking for if you just jumped to the conclusion)

That’s it! I had a fun time exploring what Codex can do, and I can’t wait to see what will happen next in the code generation space!.