Adaptive Rational Activations to Boost Deep Reinforcement Learning

Latest insights from biology show that intelligence not only emerges from the connections between neurons but that individual neurons shoulder more computational responsibility than previously anticipated. This perspective should be critical in the context of constantly changing distinct reinforcement learning environments, yet current approaches still primarily employ static activation functions. In this work, we motivate why rationals are suitable for adaptable activation functions and why their inclusion into neural networks is crucial. Inspired by recurrence in residual networks, we derive a condition under which rational units are closed under residual connections and formulate a naturally regularised version: the recurrent-rational. We demonstrate that equipping popular algorithms with (recurrent-)rational activations leads to consistent improvements on Atari games, especially turning simple DQN into a solid approach, competitive to DDQN and Rainbow.

PDF Abstract

Results from the Paper


Ranked #3 on Atari Games on Atari 2600 Skiing (using extra training data)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Atari Games Atari 2600 Asterix Rational DQN Average Score 18109 # 30
Atari Games Atari 2600 Asterix Recurrent Rational DQN Average Score 12621 # 37
Atari Games Atari 2600 Battle Zone Rational DQN Average Score 23403 # 35
Atari Games Atari 2600 Battle Zone Recurrent Rational DQN Average Score 25749 # 30
Atari Games Atari 2600 Breakout Recurrent Rational DQN Average Score 336 # 41
Atari Games Atari 2600 Breakout Rational DQN Average Score 316 # 42
Atari Games Atari 2600 Enduro Recurrent Rational DQN Average Score 957 # 31
Atari Games Atari 2600 Enduro Rational DQN Average Score 1043 # 30
Atari Games Atari 2600 James Bond Rational DQN Average Score 1122 # 23
Atari Games Atari 2600 James Bond Recurrent Rational DQN Average Score 1137 # 22
Atari Games Atari 2600 Kangaroo Recurrent Rational DQN Average Score 5266 # 28
Atari Games Atari 2600 Kangaroo Rational DQN Average Score 2941 # 31
Atari Games Atari 2600 Pong Recurrent Rational DQN Average Score 18.13 # 40
Atari Games Atari 2600 Pong Rational DQN Average Score 18.04 # 41
Atari Games Atari 2600 Q*Bert Rational DQN Average Score 14436 # 30
Atari Games Atari 2600 Q*Bert Recurrent Rational DQN Average Score 14080 # 34
Atari Games Atari 2600 Seaquest Rational DQN Average Score 6603 # 29
Atari Games Atari 2600 Seaquest Recurrent Rational DQN Average Score 7460 # 27
Atari Games Atari 2600 Skiing Recurrent Rational DQN Average Score -23582 # 3
Atari Games Atari 2600 Skiing Rational DQN Average Score -23487 # 3
Atari Games Atari 2600 Space Invaders Recurrent Rational DQN Average Score 1395 # 39
Atari Games Atari 2600 Space Invaders Rational DQN Average Score 650 # 48
Atari Games Atari 2600 Tennis Recurrent Rational DQN Average Score 20.6 # 10
Atari Games Atari 2600 Tennis Rational DQN Average Score 20.5 # 11
Atari Games Atari 2600 Time Pilot Recurrent Rational DQN Average Score 13261 # 15
Atari Games Atari 2600 Time Pilot Rational DQN Average Score 17632 # 13
Atari Games Atari 2600 Tutankham Recurrent Rational DQN Average Score 184 # 27
Atari Games Atari 2600 Tutankham Rational DQN Average Score 179 # 29
Atari Games Atari 2600 Video Pinball Rational DQN Average Score 149712 # 30
Atari Games Atari 2600 Video Pinball Recurrent Rational DQN Average Score 86942 # 34

Methods