Skip to content

Add Flash IPA#47

Open
AntonOresten wants to merge 1 commit intomainfrom
flash-ipa
Open

Add Flash IPA#47
AntonOresten wants to merge 1 commit intomainfrom
flash-ipa

Conversation

@AntonOresten
Copy link
Member

@AntonOresten AntonOresten commented Jan 28, 2026

Adds an experimental version of Flash IPA that doesn't use factorized pair representations, but instead only uses the full $O(L^2)$ pair features to create the pair bias, that gets passed into an attention function that supports it.

The "head_dim" is not the same dim that gets used in the MHA. That instead becomes head_dim + (3+1+1)*num_query_points, so if the attention function is a flash attention that only supports dimensions that are power of two, the hidden size may need to be odd-ish. Might also need to distinguish between $d_{qk}$ and $d_v$.

In pseudo-pseudo-code, it should roughly be:

image

@codecov
Copy link

codecov bot commented Jan 28, 2026

Codecov Report

❌ Patch coverage is 0% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 29.60%. Comparing base (79ab829) to head (9ea6bc1).

Files with missing lines Patch % Lines
src/ipa/FlashIPA.jl 0.00% 35 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #47      +/-   ##
==========================================
- Coverage   30.79%   29.60%   -1.20%     
==========================================
  Files          43       44       +1     
  Lines         867      902      +35     
==========================================
  Hits          267      267              
- Misses        600      635      +35     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments