Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[cli] Add analysis job to evaluate retrieval success #82

Open
bdb-dd opened this issue Jun 28, 2024 · 0 comments
Open

[cli] Add analysis job to evaluate retrieval success #82

bdb-dd opened this issue Jun 28, 2024 · 0 comments
Labels
status/draft Status: When you create an issue before you have enough info to properly describe the issue.

Comments

@bdb-dd
Copy link
Collaborator

bdb-dd commented Jun 28, 2024 •
edited
Loading

Description

In our first iteration of the main pipeline, we used instructor and prompt engineering techniques to return a boolean flag for the "i_dont_know" state.

Since streaming and output validation are not compatible, we have to evaluate retrieval success in a separate stage.
@bdb-dd bdb-dd added the status/draft Status: When you create an issue before you have enough info to properly describe the issue. label Jun 28, 2024
@renovate renovate bot mentioned this issue Jul 30, 2024
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
status/draft Status: When you create an issue before you have enough info to properly describe the issue.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bdb-dd