Search with Context Similarity #2

sshivaditya2019 · 2024-10-05T20:38:33Z

Resolves #50

Results for Database fetching backfilling:

A total of 146 issues were identified.
A comprehensive total of 1,238 comments was collected, including comments from pull requests (PRs), PR reviews, and comments specifically related to the identified issues.
Embeddings were generated using Voyage AI for enhanced data analysis.
The data was then converted into CSV format and loaded into Supabase for further use.

sshivaditya2019 · 2024-10-12T04:18:47Z

QA:

Question Answering based on Retrieval
Task Explain and Code Parsing
Does not hallucinate and create information

Models Used:

Claude 3.5 Sonnet: This model performed well with context lengths of up to 160K, showing a notable advantage in coding tasks and comprehension.
OpenAI o1-mini: In contrast to Sonnet, this model tended to hallucinate frequently when dealing with context lengths exceeding 100K.

0x4007 · 2024-10-12T08:08:23Z

QA:

Question Answering based on Retrieval

Task Explain and Code Parsing

Does not hallucinate and create information

Models Used:

Claude 3.5 Sonnet: This model performed well with context lengths of up to 160K, showing a notable advantage in coding tasks and comprehension.

This aligns with my expectations. Claude is really good at dealing with fine grained comprehension and working with code. I use it as my primary model over ChatGPT inside of my cursor IDE.

OpenAI o1-mini: In contrast to Sonnet, this model tended to hallucinate frequently when dealing with context lengths exceeding 100K.

I haven't done extensive testing regarding context lengths but I generally use o1 for higher level more complex tasks.

For example the most recent interesting use was when I bootstrapped both the "sync-configs-agent" tool as well as the "rpc-handler" tool in my github org.

I give a detailed prompt using my voice to provide context, and then after I'll paste in context that's relevant. I'll have o1 preview do it's thing and be like 80-90% to completion. I use Claude for the remainder.

I also know that mini has a larger usable context window but may be less capable than preview. So I would use it for "in between" tasks which I can pump in a ton of context from an already made codebase but still have large ish sweeping changes recommended.

References:

0x4007 · 2024-10-15T05:26:38Z

Hows this coming along?

sshivaditya2019 · 2024-10-15T06:11:18Z

Hows this coming along?

I'm currently trying to adjust the prompt to include the verbose parameter (v = 1). I've experimented with various prompting techniques, including words like "brevity," which typically help in reducing verbosity.

However, none of these approaches seem to be effective with sonnet. The output either becomes too unimaginative, lacking creativity in using resources and context, or it fails to cut off entirely. The current version works well. I can merge it into main and think of a better prompt later.

0x4007 · 2024-10-16T06:02:42Z

Okay merge and lets test.

0x4007

Looks like a pretty solid implementation

0x4007 · 2024-10-16T06:05:34Z

src/types/plugin-inputs.ts

+export const pluginSettingsSchema = T.Object({
+  model: T.String({ default: "o1-mini" }),
+  openAiBaseUrl: T.Optional(T.String()),
+  similarityThreshold: T.Number({ default: 0.1 }),


Can you explain to me what the similarity threshold is for?

Similarity levels for the similarity search with issues and comments range from 0 to 1 (Unit Normalized), where 0 indicates the best match and 1 represents the farthest or worst match.

This could benefit from clarifying that this is calculating the difference with subtraction, so closer to 0 difference means more similar. Or to make it more intuitive, maybe reverse it.

1 is the most similar and 0 is least similar, so you want 90% similarity threshold (0.9) thats a lot more intuitive for a config.

Fixed, Inverted the scale, the parameters would range from 0 to 1 instead now. If someone enters 0.9 it would mean 90% similar now.

0x4007 · 2024-10-16T06:06:08Z

src/types/gpt.ts

Maybe rename to llm.d.ts

Renamed File

src/types/github.ts

0x4007 · 2024-10-16T06:07:15Z

src/types/env.ts

 export const envSchema = T.Object({
+  OPENAI_API_KEY: T.String(),
+  UBIQUITY_OS_APP_NAME: T.String(),


Perhaps this should default to "UbiquityOS"

Added the default value.

0x4007 · 2024-10-16T06:07:44Z

src/helpers/issue.ts

+ * @returns The content of the README file as a string.
+ */
+export async function pullReadmeFromRepoForIssue(params: FetchParams): Promise<string | undefined> {
+  let readme = undefined;


Suggested change

let readme = undefined;

let readme;

Removed Initialization to undefined.

src/handlers/ask-gpt.ts

src/adapters/voyage/helpers/embedding.ts

src/adapters/openai/helpers/completions.ts

package.json

0x4007 · 2024-10-17T05:59:09Z

src/handlers/ask-gpt.ts

+    model,
+    rerankedText,
+    formattedChat,
+    ["typescript", "github", "cloudflare worker", "actions", "jest", "supabase", "openai"],


Handling Ground Truths: They are indicating that the system uses “ground truths” — meaning predefined correct examples or comments that the system relies on for determining context. Even if the query (or comment) doesn’t provide enough context, the system tries not to make assumptions. For example, if the query asks about “types” in a code snippet without specifying a language, the system shouldn’t assume it’s referring to Python.

Hard coding these things is the wrong approach then. This needs to be dynamic in a new task.

0x4007 · 2024-10-17T06:01:24Z

src/handlers/ask-gpt.ts

+  if (!text) {
+    return "";
+  }
+  return text.replace(/[^a-zA-Z0-9\s]/g, "");


You sure you want to remove formatting clues such as bullet point lists, and the syntax for images? You'll just be left with URLs

You're also removing the block quote indicator which certainly changes the meaning of the corpus (quoting somebody else doesn't mean you agree.)

This seems like the regex needs to be a lot more comprehensive.

I removed this because it was only used with issues and comments. The goal was to eliminate just the emojis, as I thought they caused the LLM to produce strange Unicode values in the results. I believe the newer models don’t have this issue either, so it’s unnecessary.

0x4007

You marked my comments as "resolved" but didn't implement the requested changes.

tests/main.test.ts

sshivaditya2019 · 2024-10-18T02:08:48Z

You marked my comments as "resolved" but didn't implement the requested changes.

Could you please clarify which changes I may have overlooked?

0x4007 · 2024-10-18T02:19:07Z

src/adapters/openai/helpers/completions.ts

+    if (answer && answer.content && res.usage) {
+      return { answer: answer.content, tokenUsage: { input: res.usage.prompt_tokens, output: res.usage.completion_tokens, total: res.usage.total_tokens } };
+    }
+    return { answer: "", tokenUsage: { input: 0, output: 0, total: 0 } };


Returning an empty string always seems like a bad idea. This seems to make more sense to throw an error

It throws an error at the UI level, displaying the message No answer from OpenAI. Sample

0x4007 · 2024-10-18T02:20:36Z

src/adapters/supabase/helpers/comment.ts

+export interface CommentType {
+  id: string;
+  plaintext: string;
+  markdown?: string;


Optional seems wrong unless its an optimization to save tokens

0x4007 · 2024-10-18T02:21:19Z

src/adapters/supabase/helpers/comment.ts

+      query_text: query,
+      query_embedding: embedding,
+      threshold: threshold,
+      max_results: 10,


Is ten optimal

There are ten issues and ten comments. Voyage AI performs excellently in this regard, consistently providing relevant issues. I believe ten is sufficient given the extensive local context.

0x4007 · 2024-10-18T02:23:42Z

src/handlers/ask-llm.ts

+}
+
+/**
+ * Asks GPT a question and returns the completions


Might be good to find and replace all GPT instances in the code base with LLM

0x4007 · 2024-10-18T02:24:47Z

src/handlers/ask-llm.ts

+    model,
+    rerankedText,
+    formattedChat,
+    ["typescript", "github", "cloudflare worker", "actions", "jest", "supabase", "openai"],


In case we haven't already: we should make another task for dynamic ground truths

0x4007 · 2024-10-18T02:31:25Z

src/helpers/issue.ts

+    const links: string[] = [];
+    inputString = inputString.replace(/https?:\/\/\S+/g, (match) => {
+      links.push(match);
+      return `__LINK${links.length - 1}__`;


This seems wrong but i dont know the full context of how its used.

It removes duplicate sentences and phrases from the context and works reasonably well, fitting nearly ~250K of context in o1-mini. However, a downside is the loss of context regarding references, as it retains links and only some punctuation.

0x4007 · 2024-10-18T02:33:02Z

src/helpers/issue.ts

+ * @param params - The parameters required to fetch the README, including the context with octokit instance.
+ * @returns The content of the README file as a string.
+ */
+export async function pullReadmeFromRepoForIssue(params: FetchParams): Promise<string | undefined> {


Mixed feelings on this. They fall out of date so fast. Its useful reference but might be worth warning the LLM that theres a good chance that it is out of date information.

They typically offer useful context about a repository, even if it's slightly outdated. This information can help users with their queries and provide some setup guidance.

0x4007 · 2024-10-18T02:34:02Z

src/plugin.ts

 import { Context } from "./types";
+import { askQuestion } from "./handlers/ask-llm";
+import { addCommentToIssue } from "./handlers/add-comment";
+import { LogLevel, LogReturn, Logs } from "@ubiquity-dao/ubiquibot-logger";


Does this package still work? I thought we deleted it and rebranded to something like

@ubiquity-os/ubiquity-os-logger

I think this was installed before the purge. Will update this to the new logger version.

0x4007 · 2024-10-18T02:36:09Z

My last batch of comments is intended to be handled async because the pull is good enough to test in beta with

sshivaditya2019 · 2024-10-18T02:57:19Z

@0x4007 This is the config I used

plugins:
  - name: test-app
    id: test-app
    uses:
      - plugin: http://localhost:5000
        runsOn: ["issue_comment.created"]
        with: 
          model: "openai/o1-mini"
          openAiBaseUrl: "https://openrouter.ai/api/v1"

Locally the env file (.dev.vars) has to be configured with:

OPENAI_API_KEY=""
SUPABASE_URL=""
SUPABASE_KEY=""
VOYAGEAI_API_KEY=""

I can deploy it on the workers using my credentials if necessary.

gentlementlegen · 2024-10-18T08:39:20Z

@sshivaditya2019 For the Supabase, does it need a brand new instance or does it share with https://github.com/ubiquity-os-marketplace/text-vector-embeddings/ ? Also the deployment script does not upload the VOYAGEAI_API_KEY to the worker, which I think is wanted.

sshivaditya2019 · 2024-10-18T13:57:22Z

It utilizes the same database as text-vector-embeddings. I’ll make the necessary updates to the deployment script, but other than that, it should be a simple worker deployment. If you need the VOYAGEAI_API_KEY, I can send it through Telegram or another method. I believe it would be better to use my Supabase, as I have already backfilled the issues and comments. I can provide the CSVs for that if you’d like to set up your Supabase with it.

gentlementlegen · 2024-10-18T14:43:57Z

@sshivaditya2019 Sounds good, please poke me in telegram (@the_mentlegen)

Keyrxng added 30 commits July 12, 2024 14:49

chore(deps): openAi

3de383f

chore: settings config

bf8d492

chore: remove supabase

d8de447

feat: add commnt with diff styles

12cbcc4

feat: simple openai chat fn

b9c1ab5

feat: issue related functions

7643b3f

fix: improved context issue filtering

bfac23d

chore: types and plugin entry

554f3f8

feat: issue utils

55d5b2a

feat: chat ready

bd790b5

fix: cspell, eslint

6063f36

feat: ubiquibot-logger

1d51869

fix: ignore all bot comments

6b0333b

chore: use string arrays, remove never configs

a72c97b

feat: deeper linked context fetching

c7b6605

chore: types and eslint ignore .wrangler

94d65e5

chore: simplify main handler

154a9b2

feat: comments handler

2b86ab2

chore: improved context handling

7e6582b

chore: refactor chat formatting, remove no diff error log

8a0a796

refactor: optimizing

834a570

chore: remove env and init tests

c684530

chore: test env setup

c08b9d0

refactor: handle PluginInputs separately for better tests

93e9cd4

chore: setup tests

042bcc0

chore: chat history and linked context tests

6ce964d

chore: remove depth

64bf785

chore: diff comments from logs

f724b85

chore: fix test

053856e

ci: knip

7bd0557

sshivaditya added 2 commits October 11, 2024 23:31

fix: tests

dd4c334

fix: tests and knip

e37f585

sshivaditya2019 marked this pull request as ready for review October 12, 2024 03:41

Update package.json

a55fb00

0x4007 approved these changes Oct 16, 2024

View reviewed changes

sshivaditya added 3 commits October 17, 2024 01:10

fix: type rename and add tsx

0f00dd4

fix: knip

6c7f136

feat: added instruction to the embedding

9583286

0x4007 reviewed Oct 17, 2024

View reviewed changes

0x4007 requested changes Oct 17, 2024

View reviewed changes

0x4007 mentioned this pull request Oct 17, 2024

Handling Ground Truths #3

Open

sshivaditya added 2 commits October 17, 2024 02:17

fix: tests

baec08a

fix: inverted the scale on similarity threshold

1edbd21

0x4007 reviewed Oct 17, 2024

View reviewed changes

tests/main.test.ts Outdated Show resolved Hide resolved

fix: removed jest commas

b417cd5

0x4007 approved these changes Oct 18, 2024

View reviewed changes

0x4007 merged commit e63b9ec into ubiquity-os-marketplace:development Oct 18, 2024
2 checks passed

ubiquity-os-beta bot mentioned this pull request Dec 8, 2024

Search ubiquity-os/plugins-wishlist#50

Closed

Search with Context Similarity #2

Search with Context Similarity #2

Conversation

sshivaditya2019 commented Oct 5, 2024 • edited Loading

sshivaditya2019 commented Oct 12, 2024 • edited Loading

0x4007 commented Oct 12, 2024 • edited Loading

0x4007 commented Oct 15, 2024

sshivaditya2019 commented Oct 15, 2024

0x4007 commented Oct 16, 2024

0x4007 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0x4007 left a comment

Choose a reason for hiding this comment

sshivaditya2019 commented Oct 18, 2024

Choose a reason for hiding this comment

sshivaditya2019 Oct 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0x4007 commented Oct 18, 2024

sshivaditya2019 commented Oct 18, 2024

gentlementlegen commented Oct 18, 2024 • edited Loading

sshivaditya2019 commented Oct 18, 2024 • edited Loading

gentlementlegen commented Oct 18, 2024

sshivaditya2019 commented Oct 5, 2024 •

edited

Loading

sshivaditya2019 commented Oct 12, 2024 •

edited

Loading

0x4007 commented Oct 12, 2024 •

edited

Loading

sshivaditya2019 Oct 18, 2024 •

edited

Loading

gentlementlegen commented Oct 18, 2024 •

edited

Loading

sshivaditya2019 commented Oct 18, 2024 •

edited

Loading