Ali Kavis

Senior Researcher

Optimization & Decision Intelligence Group · ETH Zürich

Scholar GitHub Email ORCID ResearchGate CV

About

I am a Senior Researcher at the Optimization & Decision Intelligence (ODI) Group at ETH Zürich, led by Niao He. In August 2027, I will be joining the Department of Computer Science at Aarhus University as a tenure-track Assistant Professor. I received my PhD from EPFL in August 2023, where I was advised by Volkan Cevher. Prior to joining ETH, I was a Postdoctoral Fellow at UT Austin, hosted by Aryan Mokhtari and Sujay Sanghavi, affiliated with the IFML and the TRIPODS Institute.

My research centers on the design and analysis of universal and parameter-free algorithms for stochastic optimization. I aim to build methods that work well across diverse problem settings without requiring practitioners to hand-tune hyperparameters such as learning rates or noise estimates. My work spans first-order and second-order methods, extensions to games and minimax problems, and applications in deep learning.

Research

I work at the intersection of optimization theory and machine learning, studying how to design algorithms that are both optimal in theory and practical to deploy — without hand-tuning.

Stochastic Optimization

How do gradient-based algorithms behave under noise — and what convergence guarantees can we prove without overly optimistic assumptions? I study both first- and second-order methods in non-convex and constrained settings, including high-probability bounds and almost-sure convergence.

Universal & Parameter-Free Methods

Most algorithms require the practitioner to know or tune problem-specific constants — smoothness, noise variance, diameter. I am interested in methods that adapt to these quantities automatically, achieving optimal rates across problem classes without prior knowledge.

Minimax Optimization & Games

Saddle-point problems and two-player zero-sum games arise throughout machine learning. I study the structural properties that make these problems hard or easy, and design algorithms — including second-order and optimistic methods — that exploit those properties efficiently.

Online Learning

Online learning offers a powerful framework for reasoning about sequential decision-making under uncertainty. I am interested in regret minimization, online-to-batch conversions, and how online learning ideas can be leveraged to build better adaptive algorithms for offline optimization.

Publications

16 peer-reviewed papers · 1 preprint · * denotes equal contribution

Preprint

Online Learning-guided Learning Rate Adaptation via Gradient Alignment

R. Jiang*, A. Kavis*, A. Mokhtari*

Preprint arXiv ↗

2025

Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting

S. Sanyal*, H. Prairie*, R. Das*, A. Kavis*, S. Sanghavi

ICML 2025 Spotlight arXiv ↗

Understanding Self-Supervised Learning via Gaussian Mixture Models

P. Bansal, A. Kavis, S. Sanghavi

NeurIPS 2025 arXiv ↗

2024

Adaptive and Optimal Second-order Optimistic Methods for Minimax Optimization

R. Jiang, A. Kavis, Q. Jin, S. Sanghavi, A. Mokhtari

NeurIPS 2024 arXiv ↗

Advancing the lower bounds: An accelerated, stochastic, second-order method with optimal adaptation to inexactness

A. Agafonov, D. Kamzolov, A. Gasnikov, A. Kavis, K. Antonakopoulos, V. Cevher, M. Takáč

ICLR 2024 arXiv ↗

Universal Gradient Methods for Stochastic Convex Optimization

A. Rodomanov, A. Kavis, Y. Wu, K. Antonakopoulos, V. Cevher

ICML 2024 arXiv ↗

2023

Alternation makes the adversary weaker in two-player games

V. Cevher, A. Cutkosky, A. Kavis, G. Piliouras, S. Skoulakis, L. Viano

NeurIPS 2023 Spotlight Paper ↗

2022

Adaptive Stochastic Variance Reduction for Non-convex Finite-Sum Minimization

A. Kavis*, S. Skoulakis*, K. Antonakopoulos, L. T. Dadi, V. Cevher

NeurIPS 2022 arXiv ↗

Extra-Newton: A First Approach to Noise-Adaptive Accelerated Second-Order Methods

K. Antonakopoulos*, A. Kavis*, V. Cevher

NeurIPS 2022 arXiv ↗

High Probability Bounds for a Class of Non-convex Algorithms with AdaGrad Stepsize

A. Kavis, K. Y. Levy, V. Cevher

ICLR 2022 arXiv ↗

2021

STORM+: Fully Adaptive SGD with Recursive Momentum for Nonconvex Optimization

K. Y. Levy, A. Kavis, V. Cevher

NeurIPS 2021 arXiv ↗

Sifting through the noise: Universal first-order methods for stochastic variational inequalities

K. Antonakopoulos, T. Pethick, A. Kavis, P. Mertikopoulos, V. Cevher

NeurIPS 2021 Paper ↗

2020

On the almost sure convergence of stochastic gradient descent in non-convex problems

P. Mertikopoulos, N. Hallak, A. Kavis, V. Cevher

NeurIPS 2020 arXiv ↗

Double-Loop Unadjusted Langevin Algorithm

P. T. Y. Rolland, A. Eftekhari, A. Kavis, V. Cevher

ICML 2020 arXiv ↗

2019

UniXGrad: A Universal, Adaptive Algorithm with Optimal Guarantees for Constrained Optimization

A. Kavis*, K. Y. Levy*, F. Bach, V. Cevher

NeurIPS 2019 Spotlight arXiv ↗

Efficient learning of smooth probability functions from Bernoulli tests with guarantees

P. T. Y. Rolland, A. Kavis, A. Immer, A. Singla, V. Cevher

ICML 2019 arXiv ↗

2018

Mirrored Langevin Dynamics

Y. Hsieh, A. Kavis, P. Rolland, V. Cevher

NeurIPS 2018 Spotlight arXiv ↗

Contact

Email ali.kavis@inf.ethz.ch Address ETH Zürich, Department of Computer Science
Universitätstrasse 6, 8092 Zürich, Switzerland Scholar scholar.google.com/citations?user=sPrPq6oAAAAJ

I am happy to discuss research collaborations, questions about my work, or opportunities for prospective students and postdocs. The best way to reach me is by email.