Skip to content

Commit 43ddd02

Browse files
authored
Merge branch 'mongodb-developer:main' into mongo-feed
2 parents f05d82c + ddf02d0 commit 43ddd02

File tree

152 files changed

+37727
-1
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

152 files changed

+37727
-1
lines changed

README.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ This section contains examples of use cases that are commonly seen in industry-f
4040
| **MongoDB and BuildShip Agents** | MongoDB Atlas, BuildShip low-code, Anthropic | [![GitHub](https://img.shields.io/badge/GitHub-View_on_GitHub-blue?logo=GitHub)](https://github.com/mongodb-developer/GenAI-Showcase/blob/main/snippets/low-code/BuildShip.md) | This guide demonstrates how to create an AI agent for handling rental booking workflows using BuildShip's no-code platform with MongoDB Aggregation and Insert integrations. |
4141
| **Realtime Voice OpenAI API and Atlas Vector Search Rental Agent** | OpenAI, Atlas , Python , React | [![GitHub](https://img.shields.io/badge/GitHub-View_on_GitHub-blue?logo=GitHub)](https://github.com/Pash10g/voice-openai-mongo-rentals-agent)|Sample application that use OpenAI Realtime API and MongoDB Atlas vector/CRUD as tools for a full voice based rental assistant. |
4242
| **MongoDB and n8n.io AI template** | n8n AI automation, MongoDB, JS | [template](https://n8n.io/workflows/2554-mongodb-ai-agent-intelligent-movie-recommendations/)| A sample workflow to start of when building AI context aware flows with [n8n platform](https://n8n.io)|
43-
43+
| **MongoMP - Music Platform** | Next.js 14, MongoDB Atlas, JWT | [![GitHub](https://img.shields.io/badge/GitHub-View_on_GitHub-blue?logo=GitHub)](apps/mongo-mp)| A music streaming platform with AI-powered recommendations using MongoDB's vector search capabilities|
4444

4545
## Evaluations
4646
- [Evaluating your LLM applications](https://github.com/mongodb-developer/GenAI-Showcase/blob/main/notebooks/evals/ragas-evaluation.ipynb)
@@ -90,6 +90,7 @@ AI Investment Researcher | MongoDB, CrewAI and LangChain | [![Open In Colab](htt
9090
|Smolagents MongoDB tools integration | Smolagents by HF, MongoDB, OpenAI | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mongodb-developer/GenAI-Showcase/blob/main/notebooks/agents/smolagents_hf_with_mongodb.ipynb) | |
9191
|Multi-Agent Order Management System | Smolagents by HF, MongoDB, DeepSeek | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mongodb-developer/GenAI-Showcase/blob/main/notebooks/agents/smolagents_multi-agent_micro_agents.ipynb) | |
9292
|Beyond Vectors: Augment LLM Capabilities with MongoDB Aggregation Framework and CrewAI | CrewAI, Python, OpenAI, MongoDB | [GitHub Repo](https://github.com/ranfysvalle02/mdb-agg-crewai) |[![View Article](https://img.shields.io/badge/View%20Article-blue)](https://www.mongodb.com/developer/products/mongodb/augment-llm-capabilities-with-mdb-aggregation/) | |
93+
|Self-Reflecting Gift Agent with Haystack and MongoDB Atlas | Haystack, Python, OpenAI, MongoDB |[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/mongodb-developer/GenAI-Showcase/blob/main/notebooks/agents/self_reflecting_gift_agent_haystack.ipynb) | |
9394

9495

9596
## ML

apps/graph_rag_demo/.env

Whitespace-only changes.

apps/graph_rag_demo/.gitignore

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# Ignore Mac system files
2+
.DS_store
3+
4+
# Ignore node_modules folder
5+
node_modules
6+
7+
# Ignore all text files
8+
*.txt
9+
10+
# Ignore files related to API keys
11+
./.env
12+
./.gitignore
13+
14+
# Ignore SASS config files
15+
.sass-cache
16+
#Ignore PDF Files
17+
PDF_KG
18+
.sass-cache
19+

apps/graph_rag_demo/README.md

+100
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# kg_graph__rag_mongo
2+
This guide explores how to leverage MongoDB's capabilities to create and manipulate graph representations using both Python and Node.js. By utilizing these two popular programming languages, we can demonstrate the versatility of MongoDB in different development environments, showcasing how to perform essential Graph RAG to represent and analyze graphs can provide valuable insights into complex relationships and interactions within data combining the Graph and RAG data in MongoDB.
3+
4+
5+
## Prep Steps
6+
1. Set Up Your MongoDB Database:
7+
8+
Set up a Atlas a [cloud-based MongoDB instance of MongoDB.](https://www.mongodb.com/docs/atlas/tutorial/create-new-cluster/)
9+
10+
2. Install Required Libraries:
11+
For Python:
12+
Please do a regular installation of Anaconda for your Operating System using the [doc](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html)
13+
Once conda is installed, open up a shell with the Conda CLI, change to this directory and create a new environment with the packages listed in the requirements.txt by executing the following statement:
14+
15+
```bash
16+
conda env create -f environment.yaml
17+
conda activate kg-demo
18+
```
19+
For Node.js:
20+
For the Node JS files we have a package.json and you will be able to use npm install to install all the packages needed. Our Code was tested on Node version v20.15.0 and npm version 10.7.0
21+
22+
3. Now open the .env file in the directory and populate the following three variables. Please note that the .env file may be empty when you clone this repository
23+
OPENAI_API_KEY1=
24+
ATLAS_CONNECTION_STRING=
25+
PDF =
26+
27+
4. Create the MongoDB Atlas Database:
28+
Please note that ATLAS_CONNECTION_STRING and OPENAI_API_KEY1 should already be created in the environment file.
29+
Please create a new MongoDB Atlas database called <code>langchain_db</code>
30+
Create two collections named <code>knowledge_graph</code> and <code>nodes_relationships</code>
31+
32+
5. Download and install **MongoDB Compass** for your platform by following the steps mentioned [here](https://www.mongodb.com/docs/compass/current/install/). Please ensure you install a compass version 1.40.0 or higher. Once installed, connect to your Atlas cluster by following the link [here](https://www.mongodb.com/docs/compass/current/connect/).
33+
34+
6. Create vector index on the Compass UI for the <code>embedding</code> field for <code>knowledge_graph</code> collection. [Please refer to this document](https://www.mongodb.com/docs/compass/current/indexes/create-vector-search-index/). You can use the following json document for the index defination. Please name the vector index as <code>vector_index</code>
35+
Sample:
36+
```json
37+
{
38+
"fields": [ {
39+
"type": "vector",
40+
"path": "embedding",
41+
"numDimensions": 1536,
42+
"similarity": "euclidean"
43+
} ]
44+
}
45+
```
46+
After this create an Atlas Search Index on the <code>knowledge_graph</code> collection. [Please refer this document](https://www.mongodb.com/docs/compass/current/indexes/create-search-index/). Please name the Atlas Search Index as <code>default</code>
47+
48+
49+
50+
51+
## Running the application
52+
53+
### If running for the first time
54+
We have to setup the data in the collection. Please perform the following steps:
55+
56+
1. Open an Anaconda Shell and navigate to the project directory.
57+
2. Activate the kg-demo environment by issuing the below commands:
58+
```bash
59+
conda activate kg-demo
60+
```
61+
3. Run the <code>data_insert.py file</code> by issuing the following command:
62+
```bash
63+
python data_insert.py
64+
```
65+
4. Now open a OS shell which should have `node` installed. Navigate to the project directory and issue the following commands in the same order:
66+
```bash
67+
node addEmbeddings.js
68+
```
69+
**Please note that if the above command does not end after 10-15 seconds, please terminate it using Ctrl+C**
70+
And then
71+
```bash
72+
node addTags.js
73+
```
74+
5. Now return to the Anaconda shell opened in Step 1 where you should already be there in the project directory and in the kg-demo environment and run the following command:
75+
```bash
76+
python driver_code.py
77+
```
78+
This will ask for a question which you want to ask. You can give a question like **How is social support related to aging?**.
79+
It will then ask about the Spanning Tree depth. You can give it a value of 2 or 3 for an optimal performance.
80+
Enjoy
81+
82+
### On Subsequent Runs
83+
Data is already prepared. We just need to chat and ask questions. Please perform the following steps:
84+
85+
1. Open an Anaconda Shell and navigate to the project directory.
86+
2. Activate the kg-demo environment by issuing the below commands:
87+
```bash
88+
conda activate kg-demo
89+
```
90+
3. Run the following command:
91+
```bash
92+
python driver_code.py
93+
```
94+
This will ask for a question which you want to ask. You can give a question like **How is social support related to aging?**.
95+
It will then ask about the Spanning Tree depth. You can give it a value of 2 or 3 for an optimal performance.
96+
97+
The answer for this question should be something as below:
98+
99+
**Social Support:Social factor Stress:Condition Diet:Lifestyle factor Physical Health:Condition Work Environment:Environment Job Satisfaction:Emotional state Aging:Condition Job Satisfaction:Condition Productivity:Condition Social Support:Activity Immune System:Biological system Cortisol:Hormone Cognitive Function:Function Immune System:System Diet:Activity Burnout:Condition Work Performance:Condition Heart Disease:Disease Cognitive Function:Condition Sleep Quality:Condition Inflammation:Biological process Social Support:Condition Productivity:Outcome Employee Turnover:Outcome Work Environment:Factor Diabetes:Disease Genetics:Biological factor Physical Activity:Activity Diet:Condition Obesity:Condition Heart Disease:Condition Social Support:Factor Social Relationships:Condition Sleep Quality:Health aspect Inflammation:Condition Diet:Factor Memory:Condition Blood Pressure:Condition Exercise:Activity Depression:Condition Anxiety:Condition Mental Health:Aspect Learning:Condition Sleep Quality:Aspect Stress:Concept Diet:Behavior Physical Health:Health aspect Anxiety:Emotional state Anxiety:Mental condition Depression:Mental condition Depression:Emotional state Physical Activity:Behavior Job Satisfaction:Psychological factor Mental Health:Health aspect Mental Health:Condition -----------
100+
Social support is related to aging through its impact on stress. Social support reduces stress, and since stress accelerates aging, having social support can indirectly slow down the aging process by reducing the level of stress experienced by individuals.**
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.

apps/graph_rag_demo/addEmbeddings.js

+56
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
2+
import { MongoDBAtlasVectorSearch } from "@langchain/mongodb";
3+
import { OpenAIEmbeddings } from "@langchain/openai";
4+
import { MongoClient } from "mongodb";
5+
import { PDFLoader } from "langchain/document_loaders/fs/pdf";
6+
import dotenv from "dotenv";
7+
dotenv.config();
8+
9+
const client = new MongoClient(process.env.ATLAS_CONNECTION_STRING, appname="devrel.showcase.apps.graph_rag_demo");
10+
process.env.OPENAI_API_KEY = process.env.OPENAI_API_KEY1;
11+
12+
async function run() {
13+
try {
14+
await client.connect();
15+
console.log('Connected to MongoDB successfully');
16+
const database = client.db("langchain_db");
17+
const collection = database.collection("knowledge_graph");
18+
const dbConfig = {
19+
collection: collection,
20+
indexName: "vector_index", // The name of the Atlas search index to use.
21+
textKey: "chunks", // Field name for the raw text content. Defaults to "text".
22+
embeddingKey: "embedding", // Field name for the vector embeddings. Defaults to "embedding".
23+
};
24+
// Ensure that the collection is empty
25+
await collection.deleteMany({});
26+
const pdfArray = [
27+
'./PDF_KG/Diet, Stress and Mental Healt.pdf',
28+
'./PDF_KG/Effect of Stress Management Interventions on Job Stress among nurses working in critical care units.pdf',
29+
'./PDF_KG/Factors contributing to stress among parents of children with autism.pdf',
30+
'./PDF_KG/Level of physical activity, well-being, stress and self rated health in persons with migraine and co existing tension-type headache and neck pain.pdf',
31+
'./PDF_KG/Stress and Blood Pr ess and Blood Pressure During Pr e During Pregnancy Racial Diff egnancy Racial Differences.pdf',
32+
'./PDF_KG/Stress and Headache.pdf',
33+
'./PDF_KG/THE IMPACT OF STRESSFUL LIFE EVENTS ON RELAPSE OF GENERALIZED ANXIETY DISORDER.pdf',
34+
'./PDF_KG/where are we at with stress and headache.pdf'
35+
]
36+
37+
// Load and split the sample data
38+
pdfArray.forEach(async (pdfname) => {
39+
console.log("Starting sync...", pdfname);
40+
const loader = new PDFLoader(pdfname);
41+
const data = await loader.load();
42+
const textSplitter = new RecursiveCharacterTextSplitter({
43+
chunkSize: 1000,
44+
chunkOverlap: 200,
45+
});
46+
const docs = await textSplitter.splitDocuments(data);
47+
await MongoDBAtlasVectorSearch.fromDocuments(docs, new OpenAIEmbeddings(), dbConfig);
48+
console.log("Ending sync...", pdfname);
49+
})
50+
} catch (error){
51+
console.log(error)
52+
}
53+
}
54+
run().catch(console.dir);
55+
56+

apps/graph_rag_demo/addTags.js

+69
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
import { MongoClient } from "mongodb";
2+
import dotenv from "dotenv";
3+
dotenv.config();
4+
5+
async function findAllDocuments(dbName, collectionName1, collectionName2) {
6+
const client = new MongoClient(process.env.ATLAS_CONNECTION_STRING, appname="devrel.showcase.apps.graph_rag_demo");
7+
try {
8+
await client.connect();
9+
console.log('Connected to MongoDB');
10+
11+
// Access the database
12+
const db = client.db(dbName);
13+
14+
// Access the source and destination collections
15+
const sourceCollection = db.collection(collectionName1);
16+
const destinationCollection = db.collection(collectionName2);
17+
18+
// Find all documents in the source collection
19+
const documents = await sourceCollection.find({}, { projection: { _id: 1 } }).toArray()
20+
console.log(`Found ${documents.length} documents in source collection`);
21+
22+
// Insert documents into the destination collection
23+
if (documents.length > 0) {
24+
for (let i = 0; i < documents.length; i++) {
25+
const idRepalce = documents[i]._id.replace(":", ' as a ')
26+
console.log(idRepalce)
27+
const agg = [
28+
{
29+
$search: {
30+
index: "default",
31+
text: {
32+
path: "chunks",
33+
query: idRepalce,
34+
},
35+
},
36+
},
37+
{
38+
$addFields: {
39+
tags: {
40+
$cond: {
41+
if: { $isArray: "$tags" },
42+
then: { $concatArrays: [ "$tags", [ {tagName: documents[i]._id, score: { $meta: "searchScore" }} ] ] },
43+
else: [{tagName: documents[i]._id, score: { $meta: "searchScore" }}]
44+
}
45+
}
46+
}
47+
},
48+
{
49+
$merge: {
50+
into: collectionName2,
51+
whenMatched: "merge",
52+
whenNotMatched: "discard"
53+
}
54+
}
55+
]
56+
const newdocuments = await destinationCollection.aggregate(agg).toArray();
57+
console.log('Documents copied to destination collection', newdocuments.chunks);
58+
}
59+
} else {
60+
console.log('No documents found in the source collection');
61+
}
62+
} catch (err) {
63+
console.error('Error connecting to the database or finding documents:', err);
64+
} finally {
65+
client.close();
66+
}
67+
}
68+
69+
findAllDocuments('langchain_db', 'nodes_relationships', 'knowledge_graph').catch(console.dir);

apps/graph_rag_demo/build_graph.py

+31
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
from collections import defaultdict
2+
def addEdge(graph,u,v):
3+
graph[u].append(v)
4+
def generate_edges(graph):
5+
edges = []
6+
7+
# for each node in graph
8+
for node in graph:
9+
10+
# for each neighbour node of a single node
11+
for neighbour in graph[node]:
12+
13+
# if edge exists then append
14+
edges.append((node, neighbour))
15+
return edges
16+
17+
def build_graph(level_dict):
18+
graph = defaultdict(list)
19+
relationship_names = defaultdict(set)
20+
for level in level_dict.keys():
21+
#print(f'Level is {level}')
22+
for source,targets in level_dict[level].items():
23+
#print(f'Source is {source} and targets are {targets}')
24+
for _,relationships in targets.items():
25+
#print(f'Target Node {target_node} and relationships {relationships}')
26+
for target_node, edges in relationships.items():
27+
addEdge(graph,source,target_node)
28+
relationship_key = (source,target_node)
29+
for edge in edges:
30+
relationship_names[relationship_key].add(edge)
31+
return graph,relationship_names

apps/graph_rag_demo/data_insert.py

+65
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
from langchain_core.documents import Document
2+
from langchain_community.graphs.graph_document import GraphDocument, Node, Relationship
3+
from pymongo import MongoClient
4+
from dotenv import load_dotenv
5+
from pprint import pprint
6+
import os
7+
import json
8+
from nodes_relationships import nodes,links
9+
10+
def build_lookup_map():
11+
quick_lookup = {}
12+
for key in links.keys():
13+
relationship = links[key]
14+
source_node = relationship.source
15+
lookup_key = str(source_node.id)+":"+str(source_node.type)
16+
lookup_content = quick_lookup.get(lookup_key,"empty")
17+
if lookup_content != "empty":
18+
quick_lookup.get(lookup_key).append(relationship)
19+
else:
20+
quick_lookup[lookup_key] = [relationship]
21+
return quick_lookup
22+
23+
def create_mongo_documents():
24+
mongo_documents = []
25+
quick_lookup = build_lookup_map()
26+
for key in nodes.keys():
27+
node = nodes[key]
28+
id = str(node.id)+":"+str(node.type)
29+
type = node.type
30+
rel = quick_lookup.get(id,None)
31+
relationships = set()
32+
targets = {}
33+
if rel!=None:
34+
for relationship in rel:
35+
target_id = str(relationship.target.id)+":"+str(relationship.target.type)
36+
relationships.add(target_id)
37+
target_type = targets.get(target_id,None)
38+
if target_type != None:
39+
targets[target_id].append(relationship.type)
40+
else:
41+
targets[target_id] = [relationship.type]
42+
mongo_documents.append({"_id":id,"type":type,"relationships":list(relationships),"targets":targets})
43+
else:
44+
mongo_documents.append({"_id":id,"type":type,"relationships":[],"targets":{}})
45+
return mongo_documents
46+
47+
def mongo_insert():
48+
mongo_documents = create_mongo_documents()
49+
try:
50+
uri = os.getenv("ATLAS_CONNECTION_STRING")
51+
print(uri)
52+
client = MongoClient(uri)
53+
database = client["langchain_db"]
54+
collection = database["nodes_relationships"]
55+
for doc in mongo_documents:
56+
collection.insert_one(doc)
57+
except Exception as e:
58+
print(e)
59+
finally:
60+
client.close()
61+
if __name__=="__main__":
62+
load_dotenv()
63+
print("Inserting Documents")
64+
mongo_insert()
65+
print("Successfully Inserted Documents")

0 commit comments

Comments
 (0)