Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mixed Precision in LuxLib rules #392

Closed
willtebbutt opened this issue Nov 26, 2024 · 2 comments
Closed

Mixed Precision in LuxLib rules #392

willtebbutt opened this issue Nov 26, 2024 · 2 comments
Labels
enhancement New feature or request

Comments

@willtebbutt
Copy link
Member

willtebbutt commented Nov 26, 2024

@penelopeysm discovered in TuringLang/docs#559 (comment) that Mooncake's current rule for matrix-matrix multiplication in LuxLib doesn't successfully handle the case that the two input arrays contain numbers at different precisions.

It really shouldn't be too hard to handle this properly -- we would just need to re-write the current from_rrule implementations of the various variants of matrix-matrix multiplication found around here to have proper Mooncake rules which

  1. define is_primitive for arrays whose elements are subtypes of IEEEFloat, and
  2. define the rrule!s in such a way that ensures the correct element-type is adhered to.
  3. add a method of rrule!! which is a catch-all for all other element types, which always errors with some kind of sensible error message that users can make use of to know how to modify their code.

Note: this is also a great opportunity to ensure excellent performance in Lux.jl -- the current implementations of the rules involve more allocations than are really needed, because we do not increment the gradients in-place.

@willtebbutt willtebbutt added the enhancement New feature or request label Nov 26, 2024
@yebai
Copy link
Contributor

yebai commented Nov 26, 2024

Could this be a potential duplicate of #196? In that PR, I explicitly suggested supporting mixed precision, but I agree it is better to refrain from performing any promotion implicitly.

@willtebbutt
Copy link
Member Author

Hmmm good point. I'll move this there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants