1 code implementation • 10 May 2024 • Jianyu Zhang, Niklas Nolte, Ranajoy Sadhukhan, Beidi Chen, Léon Bottou
Memory Mosaics are networks of associative memories working in concert to achieve a prediction task of interest.
1 code implementation • 1 Mar 2024 • Jianyu Zhang, Léon Bottou
Although training a deep network from scratch using such dropout rates is virtually impossible, fine-tuning a large pre-trained model under such conditions is not only possible but also achieves out-of-distribution performances that exceed those of both ensembles and weight averaging methods such as model soups.
no code implementations • 27 Sep 2023 • Léon Bottou, Bernhard Schölkopf
Many believe that Large Language Models (LLMs) open the era of Artificial Intelligence (AI).
1 code implementation • 20 Dec 2022 • Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, David Lopez-Paz
In this paper, we thus propose model ratatouille, a new strategy to recycle the multiple fine-tunings of the same foundation model on diverse auxiliary tasks.
Ranked #14 on Domain Generalization on PACS
1 code implementation • 14 Dec 2022 • Jianyu Zhang, Léon Bottou
Our thesis is that such scenarios are better served by representations that are richer than those obtained with a single optimization episode.
2 code implementations • 24 Mar 2022 • Jianyu Zhang, David Lopez-Paz, Léon Bottou
On the one hand, a rich representation provides a good initialization for the optimizer.
no code implementations • 17 Jun 2021 • Agnieszka Słowik, Léon Bottou
We show that neither DRO nor curating the training set should be construed as a complete solution for bias mitigation: in the same way that there is no universally robust training set, there is no universal way to setup a DRO problem and ensure a socially acceptable set of results.
no code implementations • 5 Mar 2020 • Alexandre Défossez, Léon Bottou, Francis Bach, Nicolas Usunier
We provide a simple proof of convergence covering both the Adam and Adagrad adaptive optimization algorithms when applied to smooth (possibly non-convex) objective functions with bounded gradients.
1 code implementation • 27 Nov 2019 • Alexandre Défossez, Nicolas Usunier, Léon Bottou, Francis Bach
Source separation for music is the task of isolating contributions, or stems, from different instruments recorded individually and arranged together to form a song.
Ranked #3 on Multi-task Audio Source Seperation on MTASS
1 code implementation • ICLR 2020 • Zhengdao Chen, Jianyu Zhang, Martin Arjovsky, Léon Bottou
We propose Symplectic Recurrent Neural Networks (SRNNs) as learning algorithms that capture the dynamics of physical systems from observed trajectories.
1 code implementation • 3 Sep 2019 • Alexandre Défossez, Nicolas Usunier, Léon Bottou, Francis Bach
We study the problem of source separation for music using deep learning with four known sources: drums, bass, vocals and other accompaniments.
14 code implementations • 5 Jul 2019 • Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, David Lopez-Paz
We introduce Invariant Risk Minimization (IRM), a learning paradigm to estimate invariant correlations across multiple training distributions.
no code implementations • 10 Jun 2019 • Aaron Defazio, Léon Bottou
We propose a system for calculating a "scaling constant" for layers and weights of neural networks.
1 code implementation • NeurIPS 2019 • Chhavi Yadav, Léon Bottou
Although the popular MNIST dataset [LeCun et al., 1994] is derived from the NIST database [Grother and Hanaoka, 1995], the precise processing steps for this derivation have been lost to time.
no code implementations • ICLR 2019 • Carl-Johann Simon-Gabriel, Yann Ollivier, Léon Bottou, Bernhard Schölkopf, David Lopez-Paz
Over the past four years, neural networks have been proven vulnerable to adversarial images: targeted but imperceptible image perturbations lead to drastically different predictions.
1 code implementation • ICLR 2019 • Aaron Defazio, Léon Bottou
The applicability of these techniques to the hard non-convex optimization problems encountered during training of modern deep neural networks is an open problem.
no code implementations • ICLR 2019 • Aaron Defazio, Léon Bottou
We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs.
1 code implementation • NeurIPS 2018 • Alexandre Défossez, Neil Zeghidour, Nicolas Usunier, Léon Bottou, Francis Bach
On the generalization task of synthesizing notes for pairs of pitch and instrument not seen during training, SING produces audio with significantly improved perceptual quality compared to a state-of-the-art autoencoder based on WaveNet as measured by a Mean Opinion Score (MOS), and is about 32 times faster for training and 2, 500 times faster for inference.
no code implementations • 7 Mar 2018 • Xiaoxia Wu, Rachel Ward, Léon Bottou
Adjusting the learning rate schedule in stochastic gradient methods is an important unresolved problem which requires tuning in practice.
1 code implementation • ICLR 2019 • Carl-Johann Simon-Gabriel, Yann Ollivier, Léon Bottou, Bernhard Schölkopf, David Lopez-Paz
Over the past few years, neural networks were proven vulnerable to adversarial images: targeted but imperceptible image perturbations lead to drastically different predictions.
no code implementations • ICML 2017 • Martin Arjovsky, Soumith Chintala, Léon Bottou
We introduce a new algorithm named WGAN, an alternative to traditional GAN training.
1 code implementation • 25 May 2017 • Jean Lafond, Nicolas Vasilache, Léon Bottou
We define a second-order neural network stochastic gradient training algorithm whose block-diagonal structure effectively amounts to normalizing the unit activations.
119 code implementations • 26 Jan 2017 • Martin Arjovsky, Soumith Chintala, Léon Bottou
We introduce a new algorithm named WGAN, an alternative to traditional GAN training.
no code implementations • 17 Jan 2017 • Martin Arjovsky, Léon Bottou
The goal of this paper is not to introduce a single algorithm or method, but to make theoretical steps towards fully understanding the training dynamics of generative adversarial networks.
4 code implementations • 15 Jun 2016 • Léon Bottou, Frank E. Curtis, Jorge Nocedal
This paper provides a review and commentary on the past, present, and future of numerical optimization algorithms in the context of machine learning applications.
2 code implementations • CVPR 2017 • David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Schölkopf, Léon Bottou
Our experiments demonstrate the existence of a relation between the direction of causality and the difference between objects and their contexts, and by the same token, the existence of observable signals that reveal the causal dispositions of objects.
1 code implementation • 11 Nov 2015 • David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, Vladimir Vapnik
Distillation (Hinton et al., 2015) and privileged information (Vapnik & Izmailov, 2015) are two techniques that enable machines to learn from other machines.
no code implementations • 12 Aug 2015 • Robert Nishihara, David Lopez-Paz, Léon Bottou
This work is naturally framed in the extreme bandit setting, which deals with sequentially choosing which distribution from a collection to sample in order to minimize (maximize) the single best cost (reward).
no code implementations • 11 Sep 2012 • Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X. Charles, D. Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, Ed Snelson
This work shows how to leverage causal inference to understand the behavior of complex learning systems interacting with their environment and predict the consequences of changes to the system.
no code implementations • NeurIPS 2007 • Léon Bottou, Olivier Bousquet
This contribution develops a theoretical framework that takes into account the effect of approximate optimization on learning algorithms.