feat: positional encoding blog #2483
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Right now links are to my personal site, will change over to the |
pcuenca
approved these changes
Nov 22, 2024
assets/you-could-have-designed-SOTA-positional-encoding/thumbnail.png
Outdated
Show resolved
Hide resolved
pcuenca
reviewed
Nov 22, 2024
pcuenca
approved these changes
Nov 25, 2024
| As we can see, without any positional information, the output of a (multi | ||
| headed) self attention operation is **identical for the same token in | ||
| different positions**, despite the tokens clearly representing distinct entities. Let's begin designing a method of enhancing self attention with positional information, such that it can determine relationships between words encoded by | ||
| their positions. |
There was a problem hiding this comment.
This is great. I wonder if we should compare it with the actual embeddings from the model (with positional embedding applied) to show they are different, but it's probably not necessary.
There was a problem hiding this comment.
I wanted to do this, but I wanted to keep the code samples as short as possible. If you're happy I'll leave it for now
Co-authored-by: Pedro Cuenca <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Congratulations! You've made it this far! Once merged, the article will appear at https://huggingface.co/blog. Official articles
require additional reviews. Alternatively, you can write a community article following the process here.
Preparing the Article
You're not quite done yet, though. Please make sure to follow this process (as documented here):
mdfile. You can also specifyguestororgfor the authors.Here is an example of a complete PR: #2382
Getting a Review
Please make sure to get a review from someone on your team or a co-author.
Once this is done and once all the steps above are completed, you should be able to merge.
There is no need for additional reviews if you and your co-authors are happy and meet all of the above.
Feel free to add @pcuenca as a reviewer if you want a final check. Keep in mind he'll be biased toward light reviews
(e.g., check for proper metadata) rather than content reviews unless explicitly asked.