Prototyping of matrix free Newton methods in Julia

      ___           ___                       ___                       ___       
     /__/\         /  /\          ___        /  /\        ___          /__/|      
    |  |::\       /  /::\        /  /\      /  /::\      /  /\        |  |:|      
    |  |:|:\     /  /:/\:\      /  /:/     /  /:/\:\    /  /:/        |  |:|      
  __|__|:|\:\   /  /:/~/::\    /  /:/     /  /:/~/:/   /__/::\      __|__|:|      
 /__/::::| \:\ /__/:/ /:/\:\  /  /::\    /__/:/ /:/___ \__\/\:\__  /__/::::\____  
 \  \:\~~\__\/ \  \:\/:/__\/ /__/:/\:\   \  \:\/:::::/    \  \:\/\    ~\~~\::::/  
  \  \:\        \  \::/      \__\/  \:\   \  \::/~~~~      \__\::/     |~~|:|~~   
   \  \:\        \  \:\           \  \:\   \  \:\          /__/:/      |  |:|     
    \  \:\        \  \:\           \__\/    \  \:\         \__\/       |  |:|     
     \__\/         \__\/                     \__\/                     |__|/      
               ___         ___           ___           ___                        
              /  /\       /  /\         /  /\         /  /\                       
             /  /:/_     /  /::\       /  /:/_       /  /:/_                      
            /  /:/ /\   /  /:/\:\     /  /:/ /\     /  /:/ /\                     
           /  /:/ /:/  /  /:/~/:/    /  /:/ /:/_   /  /:/ /:/_                    
          /__/:/ /:/  /__/:/ /:/___ /__/:/ /:/ /\ /__/:/ /:/ /\                   
          \  \:\/:/   \  \:\/:::::/ \  \:\/:/ /:/ \  \:\/:/ /:/                   
           \  \::/     \  \::/~~~~   \  \::/ /:/   \  \::/ /:/                    
            \  \:\      \  \:\        \  \:\/:/     \  \:\/:/                     
             \  \:\      \  \:\        \  \::/       \  \::/                      
              \__\/       \__\/         \__\/         \__\/                       
      ___           ___           ___                       ___           ___     
     /__/\         /  /\         /__/\          ___        /  /\         /__/\    
     \  \:\       /  /:/_       _\_ \:\        /  /\      /  /::\        \  \:\   
      \  \:\     /  /:/ /\     /__/\ \:\      /  /:/     /  /:/\:\        \  \:\  
  _____\__\:\   /  /:/ /:/_   _\_ \:\ \:\    /  /:/     /  /:/  \:\   _____\__\:\ 
 /__/::::::::\ /__/:/ /:/ /\ /__/\ \:\ \:\  /  /::\    /__/:/ \__\:\ /__/::::::::\
 \  \:\~~\~~\/ \  \:\/:/ /:/ \  \:\ \:\/:/ /__/:/\:\   \  \:\ /  /:/ \  \:\~~\~~\/
  \  \:\  ~~~   \  \::/ /:/   \  \:\ \::/  \__\/  \:\   \  \:\  /:/   \  \:\  ~~~ 
   \  \:\        \  \:\/:/     \  \:\/:/        \  \:\   \  \:\/:/     \  \:\     
    \  \:\        \  \::/       \  \::/          \__\/    \  \::/       \  \:\    
     \__\/         \__\/         \__\/                     \__\/         \__\/    

The Hessian action is exposed via matrix-vector products:

and matrix-matrix products:

References

These publications motivate and use the hessianlearn library for stochastic nonconvex optimization

• [1] O'Leary-Roseberry, T., Alger, N., Ghattas O., Low Rank Saddle Free Newton. arXiv:2002.02881. (Download)
  BibTeX

  @article{o2020low,
  title={Low Rank Saddle Free Newton: Algorithm and Analysis},
  author={O'Leary-Roseberry, Thomas and Alger, Nick and Ghattas, Omar},
  journal={arXiv preprint arXiv:2002.02881},
  year={2020}
  }
  }