Adaptive Activation Functions

Edit

General • Activation Functions • 14 methods

Adaptive or trainable activation functions are the functions with trainable parameters that are able to adapt (change, optimize) their shape and amplitude to the target dataset.

Methods

Add a Method

Method	Year	Papers
Swish Searching for Activation Functions	2017	325
LEAF Learnable Extended Activation Function (LEAF) for Deep Neural Networks	2023	49
modReLU Unitary Evolution Recurrent Neural Networks	2015	5
Rational Activation function Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks	2019	3
NFN Neo-fuzzy-neuron based new approach to system modeling, with application to actual system	1994	2
PELU Parametric Exponential Linear Unit for Deep Convolutional Neural Networks	2016	2
Rational Activation Function Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks	2019	2
PAU Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks	2019	2
AHAF Adaptive hybrid activation function for deep neural networks	2022	2
DELU Trainable Activations for Image Classification	2023	2
ReLUN Trainable Activations for Image Classification	2023	1
CosLU Trainable Activations for Image Classification	2023	1
NormLinComb Trainable Activations for Image Classification	2023	1