Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make Binary cross entropy with logit numerically stable for high logit values #2562

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Make Binary cross entropy with logit numerically stable for high logit values #2562

wants to merge 2 commits into from

Commits on Oct 14, 2024

  1. Fix documentation 

    The documentation of binary_cross_entropy_with_logit says that it
expects the target to be of type usize which is wrong and yields an
error at runtime due to dtype mismatch in the multiplication step.
    @BeneSim
    BeneSim committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    712adde View commit details
    Browse the repository at this point in the history

  2. Fix NaN loss for sigmoid(x) == 1 

    In the current implementation of binary_cross_entropy_with_logit the
loss will actually be NaN due to taking the log(0) which occurs for high
logits passing through a sigmoid and an affine transformation:

inp.affine(-1., 1.)?.log()?
^      ^              ^
|      |              |
1.0    |              |
       0.0            |
                      NaN

The proposed implementation is actually taken more or less directly from
pytorch
https://github.com/pytorch/pytorch/blob/41977a05314bbf537e1c5d6cf5916a368d1907d9/aten/src/ATen/native/Loss.cpp#L362
    @BeneSim
    BeneSim committed Oct 14, 2024
    Configuration menu
    Copy the full SHA
    fa07a09 View commit details
    Browse the repository at this point in the history