Why is tanh the default activation function instead of ReLU in DeePMD-kit?

[chenggoj]

I'm curious about why DeePMD-kit uses tanh as the default activation function instead of ReLU, which is more commonly preferred in the deep learning field. I noticed that DeePMD-kit currently supports various activation functions, including "gelu_tf", "none", "linear", "gelu", "softplus", "relu6", "tanh", "sigmoid", and "relu".

Here are my questions:

1. Considering both computational efficiency and training accuracy, is "tanh" the most recommended option?
2. Is there a concern about the vanishing gradient problem when using tanh?
3. While I understand I can test different activation functions myself, given the significant computational cost of training different models, I wonder if the development team has conducted such comparative tests and published any results?
4. How significant is the impact of different activation functions on the final model performance in terms of energy and force predictions?

I would greatly appreciate any insights from the development team or the community regarding these questions. This would be particularly helpful for users like me in choosing the most appropriate activation function for specific use cases.

Thank you in advance for any guidance or information!

[njzjz]

ReLU, which is more commonly preferred in the deep learning field

RELU is not differentiable, though considering the definition of the potential energy,

$${\displaystyle W=\int _{C}\mathbf {F} \cdot d\mathbf {x} =U(\mathbf {x} _{\text{A}})-U(\mathbf {x} _{\text{B}})},$$

$U(\mathbf {x})$ is defined by the line integral and thus is obviously differentiable.

[chenggoj]

Thanks for your reply.

"RELU is not differentiable". Based on my understandings, DeePMD-kit is learning or fitting the energy and calculating the force based on the derivative of energies. Is this the reason why "derivable" is the HARD requirement for fitting PES model using NNPs? In other words, if we are using the ReLU as activation function, we will have some issues when calculating/predicting the forces, right?

But how about the "GELU", the smooth version of "ReLU", ? ""GELU" is also totally derivable at any point and keep the ReLU's main shape. I know it is more complex and needs more computational cost. But compared with the possible performance improvement, is it worth? I am just curious if developer team has ever tested the possible effects of different activation fucntions?

Thanks a lot for your time.

[njzjz]

Actually, you need to consider whether the force in the real world is continuous or not. The hard requirement for a PES is physicalness. Otherwise, it does not make sense.

Benchmarking is tracked in #2268.

[chenggoj]

Much appreciated.