Search Results for author:
Found 17 papers, 8 papers with code
Cyclic Sparse Training: Is it Enough?
To achieve this with sparse training instead, we propose SCULPT-ing, i. e., repeated cyclic training of any sparse mask followed by a single pruning step to couple the parameters and the mask, which is able to match the performance of state-of-the-art iterative pruning methods in the high sparsity regime at reduced computational cost.
CAVACHON: a hierarchical variational autoencoder to integrate multi-modal single-cell data
Paired single-cell sequencing technologies enable the simultaneous measurement of complementary modalities of molecular data at single-cell resolution.
Spectral Graph Pruning Against Over-Squashing and Over-Smoothing
Message Passing Graph Neural Networks are known to suffer from two problems that are sometimes believed to be diametrically opposed: over-squashing and over-smoothing.
Not all tickets are equal and we know it: Guiding pruning with domain-specific knowledge
Neural structure learning is of paramount importance for scientific discovery and interpretability.
Masks, Signs, And Learning Rate Rewinding
Learning Rate Rewinding (LRR) has been established as a strong variant of Iterative Magnitude Pruning (IMP) to find lottery tickets in deep overparameterized neural networks.
Preserving local densities in low-dimensional embeddings
We show, however, that these methods fail to reconstruct local properties, such as relative differences in densities (Fig.
Dynamical Isometry for Residual Networks
We propose a random initialization scheme, RISOTTO, that achieves perfect dynamical isometry for residual networks with ReLU activation functions even for finite depth and width.
Why Random Pruning Is All We Need to Start Sparse
Random masks define surprisingly effective sparse neural network models, as has been shown empirically.
Convolutional and Residual Networks Provably Contain Lottery Tickets
The Lottery Ticket Hypothesis continues to have a profound practical impact on the quest for small scale deep neural networks that solve modern deep learning tasks at competitive performance.
Most Activation Functions Can Win the Lottery Without Excessive Depth
For networks with ReLU activation functions, it has been proven that a target network with depth $L$ can be approximated by the subnetwork of a randomly initialized neural network that has double the target's depth $2L$ and is wider by a logarithmic factor.
Plant 'n' Seek: Can You Find the Winning Ticket?
The lottery ticket hypothesis has sparked the rapid development of pruning algorithms that aim to reduce the computational costs associated with deep learning during training and model deployment.
On the Existence of Universal Lottery Tickets
The lottery ticket hypothesis conjectures the existence of sparse subnetworks of large randomly initialized deep neural networks that can be successfully trained in isolation.
Lottery Tickets with Nonzero Biases
The strong lottery ticket hypothesis holds the promise that pruning randomly initialized deep neural networks could offer a computationally efficient alternative to deep learning with stochastic gradient descent.
Scaling up Continuous-Time Markov Chains Helps Resolve Underspecification
Modeling the time evolution of discrete sets of items (e. g., genetic mutations) is a fundamental problem in many biomedical applications.
DRAGON: Determining Regulatory Associations using Graphical models on multi-Omic Networks
The increasing quantity of multi-omics data, such as methylomic and transcriptomic profiles, collected on the same specimen, or even on the same cell, provide a unique opportunity to explore the complex interactions that define cell phenotype and govern cellular responses to perturbations.
Cascade Size Distributions: Why They Matter and How to Compute Them Efficiently
Cascade models are central to understanding, predicting, and controlling epidemic spreading and information propagation.
Initialization of ReLUs for Dynamical Isometry
Deep learning relies on good initialization schemes and hyperparameter choices prior to training a neural network.
