Linear Algebra seminars

Yousef Saad examines relaxed gradient descent for large-scale optimization. Relaxing the optimal step length in Cauchy's steepest descent avoids its characteristic zigzag behavior and can bring the search direction close to an eigenvector of the Hessian. Once that alignment is sufficiently accurate, properties of the Lanczos method can accelerate convergence. The talk analyzes several such strategies and illustrates them in global minimization of strictly convex functions, retaining the simplicity and low memory requirements that make gradient methods attractive for machine learning.

Daniel Szyld extends randomized point and block Gauss-Seidel and Gauss-Southwell convergence results from Hermitian positive-definite matrices to certain non-Hermitian classes, including overlapping variables in domain decomposition. The analysis treats a range of sampling probabilities and greedy selection strategies and identifies choices that optimize the bounds. The best expected convergence bounds for randomized methods match those of more expensive deterministic Gauss-Southwell algorithms. These results establish a convergence rate for asynchronous iterations. Joint work with Andreas Frommer.

Erik Boman discusses preconditioning for asynchronous linear solvers. Inner products create synchronization requirements in Krylov methods, while preconditioners can also improve iterations such as Richardson's method. The talk focuses on asynchronous incomplete factorizations and introduces ATS-ILU, an iterative incomplete LU method with synchronous and asynchronous versions that performs competitively with ParILU.

Edmond Chow examines how asynchronous updates improve parallel iterative computation. The first part covers asynchronous versions of classical first- and second-order linear iterations, Chebyshev methods, and multigrid, with attention to efficiency and fault tolerance. The second introduces reinforcement learning and asynchronous state-value estimation for finding optimal policies. When the state space is too large to enumerate, these updates focus computational effort on frequently visited regions.

Davide Palitta introduces the S^T S-SVD, a decomposition of A derived from the SVD of its sketch SA. It is exact with high probability, preserves singular values probabilistically, and makes left singular vectors orthonormal in the sketch-induced seminorm, with lower computational cost. The talk relates this perspective to subspace embeddings and least-squares residuals, assesses sketch quality, and bounds departures from ordinary orthogonality in randomized QR. A further application extends the nearest-orthogonal-matrix problem to S^T S-orthogonality. The work builds on Gilbert, Park, and Wakin and is joint with Valeria Simoncini.

Joel Tropp studies structured dimension reduction through the injectivity of random maps, motivated by fast low-rank approximation and least-squares regression. This viewpoint sharpens guarantees for sparse maps and gives exponential improvements for tensor-product dimension reduction. Experiments assess the resulting structured random matrices on synthetic problems and scientific applications. Joint work with Chris Camaño, Ethan Epperly, and Raphael Meyer, available as arXiv:2508.21189.

Sherry Xiaoye Li surveys randomized construction of hierarchically low-rank matrices, including H/H2, HODLR, HSS, and butterfly formats with different off-diagonal structures. Applications include integral equations, boundary elements, discretized PDEs, and statistical or machine-learning kernel matrices, using either iterative matrix-vector products or direct factorization and solves. Constructing these representations from an implicit dense operator is often the main cost. The talk offers a unified view of sketch distributions, sketch sizes, approximation error bounds, high-performance implementation, applications, and open questions.

Matrix multiplication underpins scientific computing and artificial intelligence, yet practical numerical libraries and hardware accelerators commonly retain the classical cubic-time algorithm despite decades of subcubic theoretical advances. This talk reviews the effort to make faster multiplication algorithms useful in practice. It examines why arithmetic complexity alone does not determine performance: some algorithms require enormous matrices or incur large hidden constants, while communication costs, numerical stability and the match between software and hardware create additional obstacles. The historical perspective connects asymptotic algorithm design to actual performance and power consumption.

This talk examines randomized sampling methods for approximating positive semidefinite matrices. Fast leverage-score approximation combined with the Nystrom method yields provably accurate, linear-time algorithms for kernel ridge regression and kernel principal-component analysis, avoiding the usual quadratic-time cost. Related sampling techniques give relative-error low-rank approximations of positive semidefinite matrices in sublinear time without assumptions about incoherence or condition number. The results illustrate how randomized algorithms can exploit positive semidefinite structure beyond the capabilities of traditional methods. The talk concludes with open questions, particularly about oblivious sketches of kernel matrices.

Building on the preceding linear-algebra tutorials, this lecture presents two further approaches to matrix sketching: leverage-score sampling and the Subsampled Randomized Hadamard Transform. It examines how these methods produce effective compressed matrix representations for a variety of applications.

This tutorial surveys nearly optimal algorithms for regression, low-rank approximation and related numerical problems. The central approach is sketch and solve: compress a large problem into a smaller representation, then apply an algorithm to that reduced problem. These techniques provide fast methods for fundamental machine-learning and numerical-linear-algebra tasks, with running times proportional to the number of nonzero entries in the input.

Randomization offers an alternative approach to large matrix computations arising in scientific data analysis. This talk explains how randomized algorithms approximate matrix multiplication and singular-value decomposition, solve least-squares problems and linear systems, and support data-analysis applications. The accompanying presentation develops matrix sketches through row and column sampling, compares length-squared sampling with leverage-score sampling, and describes their use in low-rank approximation and matrix factorizations. It also explains how leverage scores can be approximated efficiently and how sampling guarantees support least-squares and feature-selection procedures.