no code implementations • 23 Apr 2024 • Sachin Garg, Albert S. Berahas, Michał Dereziński
We show that, for finite-sum minimization problems, incorporating partial second-order information of the objective function can dramatically improve the robustness to mini-batch size of variance-reduced stochastic gradient methods, making them more scalable while retaining their benefits over traditional Newton-type approaches.
no code implementations • 15 Nov 2023 • Albert S. Berahas, Lindon Roberts, Fred Roosta
The analysis of gradient descent-type methods typically relies on the Lipschitz continuity of the objective gradient.
no code implementations • 6 Sep 2023 • Suhail M. Shah, Albert S. Berahas, Raghu Bollapragada
We consider network-based decentralized optimization problems, where each node in the network possesses a local function and the objective is to collectively attain a consensus solution that minimizes the sum of all the local functions.
1 code implementation • 25 Jun 2023 • Xubo Yue, Raed Al Kontar, Albert S. Berahas, Yang Liu, Blake N. Johnson
Empirically, through simulated datasets and a real-world collaborative sensor design experiment, we show that our framework can effectively accelerate and improve the optimal design process and benefit all participants.
no code implementations • 1 Jan 2023 • Albert S. Berahas, Miaolan Xie, Baoyu Zhou
A step-search sequential quadratic programming method is proposed for solving nonlinear equality constrained stochastic optimization problems.
1 code implementation • 24 Jun 2021 • Albert S. Berahas, Frank E. Curtis, Michael J. O'Neill, Daniel P. Robinson
A sequential quadratic optimization algorithm is proposed for solving smooth nonlinear equality constrained optimization problems in which the objective function is defined by an expectation of a stochastic function.
no code implementations • 6 Jun 2020 • Majid Jahani, MohammadReza Nazari, Rachael Tappenden, Albert S. Berahas, Martin Takáč
This work presents a new algorithm for empirical risk minimization.
no code implementations • 2 Jun 2020 • Zheng Shi, Nur Sila Gulgec, Albert S. Berahas, Shamim N. Pakzad, Martin Takáč
Discovering the underlying behavior of complex systems is an important topic in many science and engineering disciplines.
no code implementations • 30 May 2019 • Majid Jahani, MohammadReza Nazari, Sergey Rusakov, Albert S. Berahas, Martin Takáč
In this paper, we present a scalable distributed implementation of the Sampled Limited-memory Symmetric Rank-1 (S-LSR1) algorithm.
no code implementations • 29 May 2019 • Albert S. Berahas, Liyuan Cao, Krzysztof Choromanski, Katya Scheinberg
We then demonstrate via rigorous analysis of the variance and by numerical comparisons on reinforcement learning tasks that the Gaussian sampling method used in [Salimans et al. 2016] is significantly inferior to the orthogonal sampling used in [Choromaski et al. 2018] as well as more general interpolation methods.
no code implementations • 3 May 2019 • Albert S. Berahas, Liyuan Cao, Krzysztof Choromanski, Katya Scheinberg
To this end, we use the results in [Berahas et al., 2019] and show how each method can satisfy the sufficient conditions, possibly only with some sufficiently large probability at each iteration, as happens to be the case with Gaussian smoothing and smoothing on a sphere.
Optimization and Control
1 code implementation • 28 Jan 2019 • Albert S. Berahas, Majid Jahani, Peter Richtárik, Martin Takáč
We present two sampled quasi-Newton methods (sampled LBFGS and sampled LSR1) for solving empirical risk minimization problems that arise in machine learning.
no code implementations • 26 Jul 2017 • Albert S. Berahas, Martin Takáč
This paper describes an implementation of the L-BFGS method designed to deal with two adversarial situations.
no code implementations • 17 May 2017 • Albert S. Berahas, Raghu Bollapragada, Jorge Nocedal
Sketching, a dimensionality reduction technique, has received much attention in the statistics community.
no code implementations • NeurIPS 2016 • Albert S. Berahas, Jorge Nocedal, Martin Takáč
The question of how to parallelize the stochastic gradient descent (SGD) method has received much attention in the literature.
no code implementations • 4 Nov 2015 • Nitish Shirish Keskar, Albert S. Berahas
In this paper, we present adaQN, a stochastic quasi-Newton algorithm for training RNNs.