Make Binary cross entropy with logit numerically stable for high logit values #2562
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The documentation of binary_cross_entropy_with_logit says that it expects the target to be of type usize which is wrong and yields an error at runtime due to dtype mismatch in the multiplication step.
In the current implementation of binary_cross_entropy_with_logit the
loss will actually be NaN due to taking the log(0) which occurs for high
logits passing through a sigmoid and an affine transformation:
inp.affine(-1., 1.)?.log()?
^ ^ ^
| | |
1.0 | |
0.0 |
NaN
The proposed implementation is actually taken more or less directly from
pytorch
https://github.com/pytorch/pytorch/blob/41977a05314bbf537e1c5d6cf5916a368d1907d9/aten/src/ATen/native/Loss.cpp#L362
|
Just a quick update, I also encountered the case where the logit was so small that the sigmoid returned 0. So I guess we need a better way. Maybe the method tensorflow uses might make sense, they basically calculate the log_sigmoid by using softplus: See log_sigmoid and softplus. This could be implemented like let log_sigmoid_input = (inp.neg()?.exp()? + 1.)?.log()?.neg()?;EDIT: This however will also quickyl overflow for small |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See #2561 for details. I added detailed commentary to the commits.