fix: tokens exceeded with too many related memories. #38
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
不建议让用户将最大token设置为模型的上下文窗口，发给模型的提示与模型的回复会一并算在上下文窗口，如果给4096的gpt3.5发送4ktoken会导致模型只能回复96token的内容 |
|
感谢提醒，就这个PR想解决的核心问题而言，内置一个合适的最大token数我觉得没有问题。如果考虑到后续的扩展性，我觉得这个还是应当交给用户来设置，以应对不同模型的情况，可以加一下关于这个的注释来告诉用户应该怎么设置，然后在配置模板文件里设置一个合适的默认值 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
有的时候会出现单条记忆比较长的情况，这样几条相关记忆加上去，token上限就超出了，导致不能正常发送prompt。加了一个简单的处理，如果检测到当前的几条相关记忆加上去后会超出最大token数限制，那么就按照相似度从低到高删除记忆。