Transformers, central to the successes in modern Natural Language Processing, often falter on arithmetic tasks despite their vast capabilities --which paradoxically include remarkable coding...

The new wave of AI systems, ChatGPT and its more powerful successors, exhibit extraordinary capabilities across a broad swath of domains. In light of this, w...

We propose a synthetic reasoning task, LEGO (Learning Equality and Group Operations), that encapsulates the problem of following a chain of reasoning, and we study how the Transformer...

This monograph covers some recent advances in a range of acceleration techniques frequently used in convex optimization. We first use quadratic optimization problems to introduce two key families...

Based on joint work with Yi Zhang, Arturs Backurs, Ronen Eldan, Suriya Gunasekar, Tal Wagner https://arxiv.org/abs/2206.04301.

In this work, we consider the distributed optimization of non-smooth convex functions using a network of computing units. We investigate this problem under two regularity assumptions: (1) the...

Abstract:I will talk about some of the classic problems in the calculus of variations, and describe some of the mathematics which was developed to solve them...

We introduce phi-1, a new large language model for code, with significantly smaller size than competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on 8...

Read our Story, Embracing change and resetting expectations on Microsoft Unlocked

Existing analyses of neural network training often operate under the unrealistic assumption of an extremely small learning rate. This lies in stark contrast to practical wisdom and empirical...

Nathan Labenz sits down with Ronen Eldan and Yuanzhi Li of Microsoft Research to discuss the small natural language dataset they created called TinyStories. ...

Extended motivational speech to study deep learning mathematically.I gave this talk at an NSF Town Hall where the goal was to discuss successes of deep learn...

I give a tentative theoretical justification for why large overparametrization is important in neural networks.Primarily based on "A Universal Law of Robustn...

Crash course on tensors (what they are, what cross norms are, basic generalities about nuclear norm/operator and rank), followed by an application of this to...

With this blog post, it is our pleasure to unveil the NeurIPS paper awards for 2019, and share more information on the selection process…

A novel intelligence has roared into the mainstream, sparking euphoric excitement as well as abject fear. Explore the landscape of possible futures in a brav...

We give a short proof of the Cohen-Rosenfeld-Kolter theorem on the certified robustness of randomized smoothing.Cohen-Rosenfeld-Kolter paper: https://arxiv.o...

New results about finding stationary points of non-convex functions in low dimension. Based on this paper https://arxiv.org/abs/2001.02968 , joint work with ...

Phi-2 is a tiny model that could fit on a phone, but it outperforms huge language models like Llama 2. I explain more about how it was made and what it means...

Computer scientist Yejin Choi is here to demystify the current state of massive artificial intelligence systems like ChatGPT, highlighting three key problems...

Multi-armed bandits a simple but very powerful framework for algorithms that make decisions over time under uncertainty. An enormous body of work has accumulated over the years, covered in several...

We prove an almost constant lower bound of the isoperimetric coefficient in the KLS conjecture. The lower bound has the dimension dependency $d^{-o_d(1)}$. When the dimension is large enough, our...

Deep learning is also known as hierarchical learning, where the learner _learns_ to represent a complicated target function by decomposing it into a sequence of simpler functions to reduce sample...

I describe a mathematical conjecture potentially establishing overparametrization as a law of robustness for neural networks. In particular it would imply th...

Rather than asking simple questions, we try using these models in slightly more realistic scenarios. TL;DR: ChatGPT (3.5) still seems…

We prove that the information-theoretic upper bound on the minimax regret for zeroth-order adversarial bandit convex optimisation is at most $O(d^{2.5} \sqrt{n} \log(n))$, where $d$ is the...

We propose an approach to the science of deep learning that roughly follows what physicists do to understand reality: (1) explore phenomena through controlle...

Simple demonstration of the limitations of kernel methods, even if the training data points can be chosen (the latter makes parity with noise easy to learn f...

Despite the success of deep learning, from a theoretical standpoint, it remains absurdly unclear why deep learning is better than shallow learning. The main ...

We analyse adversarial bandit convex optimisation with an adversary that is restricted to playing functions of the form $f_t(x) = g_t(\langle x, θ\rangle)$ for convex $g_t : \mathbb R \to...

Model-free reinforcement learning (RL) algorithms, such as Q-learning, directly parameterize and update value functions or policies without explicitly modeling the environment. They are typically...

I discuss the power of the "Textbooks Are All You Need" methodology to build much more compact LLMs using higher quality data. I emphasize phi-1 (coding LLM ...