no code implementations • 2 Mar 2024 • Emi Zeger, Yifei Wang, Aaron Mishkin, Tolga Ergen, Emmanuel Candès, Mert Pilanci
We prove that training neural networks on 1-D data is equivalent to solving a convex Lasso problem with a fixed, explicitly defined dictionary matrix of features.
1 code implementation • 19 Dec 2023 • Tolga Ergen, Mert Pilanci
We also show that all the stationary of the nonconvex training objective can be characterized as the global optimum of a subsampled convex program.
no code implementations • 6 Mar 2023 • Tolga Ergen, Halil Ibrahim Gulluk, Jonathan Lacotte, Mert Pilanci
We first show that regularized deep threshold network training problems can be equivalently formulated as a standard convex optimization problem, which parallels the LASSO method, provided that the last hidden layer width exceeds a certain threshold.
no code implementations • 20 Nov 2022 • Tolga Ergen, Behnam Neyshabur, Harsh Mehta
To this end, we study the training problem of attention/transformer networks and introduce a novel convex analytic approach to improve the understanding and optimization of these networks.
1 code implementation • 18 Jul 2022 • Batu Ozturkler, Arda Sahiner, Tolga Ergen, Arjun D Desai, Christopher M Sandino, Shreyas Vasanawala, John M Pauly, Morteza Mardani, Mert Pilanci
However, they require several iterations of a large neural network to handle high-dimensional imaging tasks such as 3D MRI.
no code implementations • 17 May 2022 • Arda Sahiner, Tolga Ergen, Batu Ozturkler, John Pauly, Morteza Mardani, Mert Pilanci
Vision transformers using self-attention or its proposed alternatives have demonstrated promising results in many image related tasks.
no code implementations • NeurIPS Workshop Deep_Invers 2021 • Batu Ozturkler, Arda Sahiner, Tolga Ergen, Arjun D Desai, John M. Pauly, Shreyas Vasanawala, Morteza Mardani, Mert Pilanci
Model-based deep learning approaches have recently shown state-of-the-art performance for accelerated MRI reconstruction.
no code implementations • 13 Oct 2021 • Yifei Wang, Tolga Ergen, Mert Pilanci
Recent work has proven that the strong duality holds (which means zero duality gap) for regularized finite-width two-layer ReLU networks and consequently provided an equivalent convex training problem.
no code implementations • 11 Oct 2021 • Tolga Ergen, Mert Pilanci
We first show that the training of multiple three-layer ReLU sub-networks with weight decay regularization can be equivalently cast as a convex optimization problem in a higher dimensional space, where sparsity is enforced via a group $\ell_1$-norm regularization.
1 code implementation • ICLR 2022 • Arda Sahiner, Tolga Ergen, Batu Ozturkler, Burak Bartan, John Pauly, Morteza Mardani, Mert Pilanci
In this work, we analyze the training of Wasserstein GANs with two-layer neural network discriminators through the lens of convex duality, and for a variety of generators expose the conditions under which Wasserstein GANs can be solved exactly with convex optimization approaches, or can be represented as convex-concave games.
no code implementations • ICLR 2022 • Tolga Ergen, Arda Sahiner, Batu Ozturkler, John Pauly, Morteza Mardani, Mert Pilanci
Batch Normalization (BN) is a commonly used technique to accelerate and stabilize training of deep neural networks.
no code implementations • ICLR 2021 • Arda Sahiner, Tolga Ergen, John Pauly, Mert Pilanci
We describe the convex semi-infinite dual of the two-layer vector-output ReLU neural network training problem.
no code implementations • ICLR 2021 • Tolga Ergen, Mert Pilanci
We study training of Convolutional Neural Networks (CNNs) with ReLU activations and introduce exact convex optimization formulations with a polynomial complexity with respect to the number of data samples, the number of neurons, and data dimension.
no code implementations • 25 Feb 2020 • Tolga Ergen, Mert Pilanci
Our analysis also shows that optimal network parameters can be also characterized as interpretable closed-form formulas in some practically relevant special cases.
no code implementations • ICML 2020 • Mert Pilanci, Tolga Ergen
We develop exact representations of training two-layer neural networks with rectified linear units (ReLUs) in terms of a single convex program with number of variables polynomial in the number of training samples and the number of hidden neurons.
no code implementations • 22 Feb 2020 • Tolga Ergen, Mert Pilanci
We show that a set of optimal hidden layer weights for a norm regularized DNN training problem can be explicitly found as the extreme points of a convex set.
no code implementations • 25 Oct 2017 • Tolga Ergen, Ali Hassan Mirza, Suleyman Serdar Kozat
We investigate anomaly detection in an unsupervised framework and introduce Long Short Term Memory (LSTM) neural network based algorithms.
Semi-supervised Anomaly Detection Supervised Anomaly Detection +1