Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(EAI-152) check for change to chunkAlgoHash when updating embeddings #580

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
Open

(EAI-152) check for change to chunkAlgoHash when updating embeddings #580

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
6816ee7
check for change in chunkAlgoHash
yakubova92 Dec 16, 2024
d688f5c
add pages with changed chunking to arg for next step
yakubova92 Dec 16, 2024
47c3b92
pulling pages to be updated based on embedded content
yakubova92 Dec 17, 2024
11bac3a
update embeddings if chunk algo changes, regardless of page changes
yakubova92 Dec 20, 2024
5425929
update mockEmbeddedContentStore to have newly added method
yakubova92 Dec 20, 2024
e358ca0
teardown in afterEach
yakubova92 Dec 20, 2024
57091b2
chunkAlgoHash creation moved up in function chain, value passed down
yakubova92 Dec 20, 2024
42ab655
move chunkAlgoHashes out of global scope
yakubova92 Dec 20, 2024
5872f56
change implementation - query for data sources that use an old chunk …
yakubova92 Jan 15, 2025
963870a
cleanup and lint
yakubova92 Jan 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 32 additions & 15 deletions packages/mongodb-rag-core/src/contentStore/updateEmbeddedContent.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,21 @@ export interface EmbedConcurrencyOptions {
createChunks?: number;
}

const chunkAlgoHashes = new Map<string, string>();
yakubova92 marked this conversation as resolved.
Show resolved Hide resolved

const getHashForFunc = (f: ChunkFunc, o?: Partial<ChunkOptions>): string => {
const data = JSON.stringify(o ?? {}) + f.toString();
const existingHash = chunkAlgoHashes.get(data);
if (existingHash) {
return existingHash;
}
const hash = createHash("sha256");
hash.update(data);
const digest = hash.digest("hex");
chunkAlgoHashes.set(data, digest);
return digest;
};

/**
(Re-)embeddedContent the pages in the page store that have changed since the given date
and stores the embeddedContent in the embeddedContent store.
Expand Down Expand Up @@ -48,6 +63,23 @@ export const updateEmbeddedContent = async ({
sourceNames ? ` in sources: ${sourceNames.join(", ")}` : ""
}`
);
// find changed chunkingHashes
const allPages = await pageStore.loadPages({sources: sourceNames});
for (const page of allPages) {
const existingContent = await embeddedContentStore.loadEmbeddedContent({
page,
});
const chunkAlgoHash = getHashForFunc(chunkPage, chunkOptions);
if (existingContent[0].chunkAlgoHash !== chunkAlgoHash) {
logger.info(
`Chunking algorithm has changed for ${page.sourceName}: ${page.url}. Deleting existing embedded content to force an update.`
);
await embeddedContentStore.deleteEmbeddedContent({
page,
});
}
}

await PromisePool.withConcurrency(concurrencyOptions?.processPages ?? 1)
.for(changedPages)
.process(async (page, index, pool) => {
Expand All @@ -73,21 +105,6 @@ export const updateEmbeddedContent = async ({
});
};

const chunkAlgoHashes = new Map<string, string>();

const getHashForFunc = (f: ChunkFunc, o?: Partial<ChunkOptions>): string => {
const data = JSON.stringify(o ?? {}) + f.toString();
const existingHash = chunkAlgoHashes.get(data);
if (existingHash) {
return existingHash;
}
const hash = createHash("sha256");
hash.update(data);
const digest = hash.digest("hex");
chunkAlgoHashes.set(data, digest);
return digest;
};

export const updateEmbeddedContentForPage = async ({
page,
store,
Expand Down