Skip to content

Commit 9d82ea7

Browse files
haroon0xharoon0x
committed
update readme
1 parent 470b1a4 commit 9d82ea7

File tree

3 files changed

+157
-21
lines changed

3 files changed

+157
-21
lines changed

‎misc/demo.mp4‎

4.05 MB
Binary file not shown.

‎src/README.md‎

Lines changed: 137 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,141 @@
1-
<h1 align="center">Pocket Flow Project Template: Agentic Coding</h1>
1+
# CrawlWise: GEO Agent System
22

3-
<p align="center">
4-
<a href="https://github.com/The-Pocket/PocketFlow" target="_blank">
5-
<img
6-
src="./assets/banner.png" width="800"
7-
/>
8-
</a>
9-
</p>
3+
CrawlWise is an end-to-end Generative Engine Optimization (GEO) agent system that audits and improves webpages for SEO and AI-readiness. It combines a FastAPI backend, a modern React/Tailwind frontend, the [PocketFlow](https://github.com/The-Pocket/PocketFlow) agent framework, and the [AlchemystAI LLM Proxy](https://getalchemystai.com) for robust, production-grade content analysis and generation.
104

11-
This is a project template for Agentic Coding with [Pocket Flow](https://github.com/The-Pocket/PocketFlow), a 100-line LLM framework, and your editor of choice.
5+
---
126

13-
- We have included the [.cursorrules](.cursorrules), [.clinerules](.clinerules), and [.windsurfrules](.windsurfrules) files to let Cursor AI (also Cline, or Windsurf) help you build LLM projects.
14-
15-
- Want to learn how to build LLM projects with Agentic Coding?
7+
## 🚀 What is CrawlWise?
8+
9+
- **Input:** A user submits a public URL (plus optional keywords and tone).
10+
- **Process:**
11+
1. The backend crawls the page (using [Crawl4AI](https://pypi.org/project/crawl4ai/)), extracts content and metadata.
12+
2. PocketFlow orchestrates a multi-step agent pipeline:
13+
- **Audit Node:** Analyzes structure, SEO/GEO issues, and provides recommendations (via AlchemystAI LLM).
14+
- **Enhancement Node:** Generates improved intro, meta tags, and FAQ (via AlchemystAI LLM).
15+
3. The frontend displays results in a clean, animated UI.
16+
- **Output:** A structured JSON with audit and improvement suggestions.
17+
18+
---
19+
20+
## 🛠️ Tech Stack
21+
22+
- **Backend:** FastAPI, [PocketFlow](https://github.com/The-Pocket/PocketFlow), [Crawl4AI](https://pypi.org/project/crawl4ai/), [LangChain](https://python.langchain.com/), [AlchemystAI LLM Proxy](https://getalchemystai.com)
23+
- **Frontend:** React, TailwindCSS
24+
- **Agent Orchestration:** [PocketFlow](https://github.com/The-Pocket/PocketFlow)
25+
- **LLM Provider:** [AlchemystAI](https://getalchemystai.com)
26+
27+
---
28+
29+
## 🏗️ Architecture
30+
31+
```mermaid
32+
graph TD
33+
A[User (Frontend)] -->|Submits URL| B[FastAPI Backend]
34+
B --> C[Crawl4AI Crawler]
35+
C --> D[Extracted Content & Metadata]
36+
D --> E[PocketFlow Agent Pipeline]
37+
E --> F[Audit Node (LLM via AlchemystAI)]
38+
E --> G[Enhancement Node (LLM via AlchemystAI)]
39+
F --> H[Audit JSON]
40+
G --> I[Improvements JSON]
41+
H & I --> J[API Response]
42+
J -->|Display Results| A
43+
F & G --> K[AlchemystAI LLM Proxy]
44+
K -.->|LLM Calls| F
45+
K -.->|LLM Calls| G
46+
```
47+
48+
---
49+
50+
## 🌐 Key Links
51+
52+
- **PocketFlow Agent Framework:** [https://github.com/The-Pocket/PocketFlow](https://github.com/The-Pocket/PocketFlow)
53+
- **AlchemystAI LLM Proxy:** [https://getalchemystai.com](https://getalchemystai.com)
54+
55+
---
56+
57+
## ⚡ Quickstart
58+
59+
1. **Clone the repo**
60+
```bash
61+
git clone <your-fork-or-this-repo>
62+
cd CrawlWise
63+
```
64+
2. **Set up environment**
65+
- Create a `.env` file in `src/` with your AlchemystAI API key:
66+
```env
67+
ALCHEMYST_API_KEY=sk-... # Get yours at https://getalchemystai.com
68+
```
69+
3. **Install backend dependencies**
70+
```bash
71+
cd src
72+
pip install -r requirements.txt
73+
# or
74+
pip install .
75+
```
76+
4. **Run the backend**
77+
```bash
78+
uvicorn main:app --reload
79+
```
80+
5. **Run the frontend**
81+
```bash
82+
cd frontend
83+
npm install
84+
npm run dev
85+
```
86+
87+
---
88+
89+
## 🔑 Get an AlchemystAI API Key
90+
- Sign up at [https://getalchemystai.com](https://getalchemystai.com)
91+
- Copy your API key and add it to your `.env` as `ALCHEMYST_API_KEY`
92+
93+
---
94+
95+
## 📡 Example API Usage
96+
97+
**Request:**
98+
```json
99+
POST /api/v1/audit
100+
{
101+
"agent": "seo",
102+
"url": "https://example.com/your-blog-post",
103+
"keywords": ["LLM SEO", "AI search"],
104+
"tone": "professional"
105+
}
106+
```
107+
108+
**Response:**
109+
```json
110+
{
111+
"audit": {
112+
"structure": ["Missing H1", "Too many H3s"],
113+
"issues": ["No meta title", "No structured FAQ section"],
114+
"recommendations": ["Add a meta title using keywords", "Use structured data"]
115+
},
116+
"improvements": {
117+
"intro": "In this article, we explore how to use large language models for modern SEO...",
118+
"meta": {
119+
"title": "Mastering SEO for LLMs in 2025",
120+
"description": "Learn how to optimize your content for AI-driven search engines using LLM strategies."
121+
},
122+
"faqs": [
123+
{
124+
"question": "What is Generative Engine Optimization (GEO)?",
125+
"answer": "GEO is the practice of optimizing your content for inclusion in AI-generated answers."
126+
}
127+
]
128+
}
129+
}
130+
```
131+
132+
---
133+
134+
## 🤖 How It Works (Agent Pipeline)
135+
136+
- **PocketFlow** orchestrates the agent pipeline:
137+
1. **CrawlExtractNode:** Fetches and parses the webpage.
138+
2. **AuditContentNode:** Audits content/metadata for SEO/GEO (via LLM).
139+
3. **GenerateEnhancementsNode:** Generates improved intro, meta, FAQ (via LLM).
140+
- All LLM calls are routed through the **AlchemystAI Proxy** for reliability and cost efficiency.
16141

17-
- Check out the [Agentic Coding Guidance](https://the-pocket.github.io/PocketFlow/guide.html)
18-
19-
- Check out the [YouTube Tutorial](https://www.youtube.com/@ZacharyLLM?sub_confirmation=1)

‎src/nodes.py‎

Lines changed: 20 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -74,9 +74,16 @@ def exec(self, inputs):
7474
)
7575
system_prompt = (
7676
"You are CrawlWise, an autonomous AI agent for Generative Engine Optimization (GEO). "
77-
"Only output valid JSON. Do not include any explanations, markdown, or extra text. "
78-
"For an audit, analyze the content and metadata for: structural issues, keyword presence, E-E-A-T signals, and FAQ section presence. "
79-
"Output a JSON with keys: structure (list), issues (list), recommendations (list)."
77+
"Your task is to audit webpage content and metadata for SEO and GEO readiness. "
78+
"You must output ONLY valid, minified JSON (no markdown, no code block, no explanations, no extra text). "
79+
"The JSON must have these keys: "
80+
'"structure" (list of strings), '
81+
'"issues" (list of strings), '
82+
'"recommendations" (list of strings). '
83+
"If any field is missing, output an empty list for it. "
84+
"Do not include any comments, trailing commas, or text outside the JSON object. "
85+
"Example output: {\"structure\":[],\"issues\":[],\"recommendations\":[]}. "
86+
"If you do not follow these instructions, your output will be rejected."
8087
)
8188
import json
8289
response = call_llm(prompt, system_prompt)
@@ -112,9 +119,16 @@ def exec(self, inputs):
112119
)
113120
system_prompt = (
114121
"You are CrawlWise, an autonomous AI agent for Generative Engine Optimization (GEO). "
115-
"Only output valid JSON. Do not include any explanations, markdown, or extra text. "
116-
"For improvements, generate: a rewritten introduction, optimized meta title and description, and a FAQ section. "
117-
"Output a JSON with keys: intro (str), meta (dict with title/description), faqs (list of dicts with question/answer)."
122+
"Your task is to generate content improvements for a webpage. "
123+
"You must output ONLY valid, minified JSON (no markdown, no code block, no explanations, no extra text). "
124+
"The JSON must have these keys: "
125+
'"intro" (string), '
126+
'"meta" (object with keys "title" (string) and "description" (string)), '
127+
'"faqs" (list of objects, each with keys "question" (string) and "answer" (string)). '
128+
"If any field is missing, output an empty string, object, or list as appropriate. "
129+
"Do not include any comments, trailing commas, or text outside the JSON object. "
130+
"Example output: {\"intro\":\"\",\"meta\":{\"title\":\"\",\"description\":\"\"},\"faqs\":[]}. "
131+
"If you do not follow these instructions, your output will be rejected."
118132
)
119133
import json
120134
response = call_llm(prompt, system_prompt)

0 commit comments

Comments
 (0)