A collection of policies, heuristics, and tactics I've found helpful for management at Ought

RAFT is a few-shot classification benchmark that tests language models across multiple domains on economically valuable classification tasks with evaluation that mirrors deployment.

Ought is an applied machine learning lab. We’re building Elicit, the AI research assistant. Our mission is to automate and scale open-ended reasoning…

This repository contains the code for "Exploiting Cloze Questions for Few-Shot Text Classification and Natural Language Inference" - timoschick/pet

Large language models generate complex, open-ended outputs: instead of outputting a class label they write summaries, generate dialogue, or produce working code. In order to asses the reliability...

Despite their wide adoption, the underlying training and memorization dynamics of very large language models is not well understood. We empirically study exact memorization in causal and masked...

We show how to derive state-of-the-art unsupervised neural machine translation systems from generatively pre-trained language models. Our method consists of three steps: few-shot amplification,...

The ability to extrapolate from short problem instances to longer ones is an important form of out-of-distribution generalization in reasoning tasks, and is crucial when learning from datasets...

We can think about machine learning systems on a spectrum from process-based to outcome-based: …

Now that Elicit is a standalone company let's review our mission.

How Elicit works and where it doesn't

Pretrained language models (PLMs) have demonstrated remarkable performance in various natural language processing tasks: Unidirectional PLMs (e.g., GPT) are well known for their superior text...

Recently, contrastive learning attracts increasing interests in neural text generation as a new solution to alleviate the exposure bias problem. It introduces a sequence-level training signal...

Eventually, there will be many AI assistants for chat, question-answering, search, web browsing, writing, and general computer use. They'll generally aim to be trustworthy, easy-to-use tools for...

Theorem proving in natural mathematical language - the mixture of symbolic and natural language used by humans - plays a central role in mathematical advances and education, and tests aspects of...

This session starts with an introduction to leveling frameworks and how they can help people manage their careers, as well as give helpful feedback to others...

The two main impacts of Elicit on AI Safety are improving epistemics and pioneering process supervision.

In theorem proving, the task of selecting useful premises from a large library to unlock the proof of a given conjecture is crucially important. This presents a challenge for all theorem provers,...

Fine-tuning pretrained language models (LMs) without making any architectural changes has become a norm for learning various language downstream tasks. However, for non-language downstream tasks,...