u/BrianCannard

In 2010, Martens invented deep Hessian-free methods. In 2011, Sutskever collaborated with Martens to apply it successfully to arbitrary RNNs (not special LSTMs, GRUs etc.)

In 2020, NERSC, Lawrence Berkeley National Laboratory, publish applications of combining the method with AdamW. Did everyone went quiet about the method invented by Sutskever and Martens 15 years ago and just forgot about it, or did it become SoTA optimizer secretly used for MoE router nonlinearities?

https://arxiv.org/abs/2006.00719