Experiment embedding ,dual chain retrieval, HotFlip upranking and inducing sequence generator

[hieunguyen-cyber]

Phase 1 — Diagram-only Summary

Extracted directly from the experiment diagram.

---

1. Experiment Embedding

• Dataset: BigCodeBench
• Models: BGE-M3 + mGTE
• Fusion: Average embeddings from both models
• Output: Query embeddings and Document embeddings

---

2. Dual-chain Retrieval (2-way)

• Compute cosine similarity between all query–document pairs
• For each query → select Top-5 documents
• Merge all Top-5 results → form Targeted Document Set (TDS)
• Build proxy query mapping: for each document in TDS, list queries where it appears in Top-5

---

3. HotFlip Upranking

• Apply HotFlip optimization on targeted documents
• Perform token-level edits guided by similarity gradients
• Goal: Uprank documents in retrieval results (increase cosine similarity with proxy queries)

---

4. Inducing Sequence Generator

• Generate inducing paragraphs/comments/docstrings with embedded URLs
• Reference method: web cloaking induction for LLMs
• Use LaTeX generator to produce polished PDF files
• Inject 0.01% – 3% of poisoned documents into the clean RAG database

---

Evaluation

• Compute Precision@k (before vs after attack)
• Measure ASR (Attack Success Rate) — rate of agent follow/quote on malicious links

[hieunguyen-cyber]

A Guide on Data Processing and Dual Chain Retrieval Pipeline

Step 1: Download Data

• Download the dataset from the internet.
• The data should be in query-code form.

Step 2: Preprocess the Dataset

• Based on the format and code in preprocess_bigcodebench.py, write a new preprocessing script for your dataset.
• Ensure the script processes all data and creates a folder named after your dataset.
• The output should be in JSON format.
• Assign IDs to each document and query in the format: <dataset_name>_<dataset_version>_<document/query>_<idx>.
• The JSON files should have the same name as their IDs.
• This corresponds to the following directory structure:

data/
├── processed/
│   └── <dataset_name>/
│       └── <version>/
│           ├── queries/
│           │   ├── <query_id>.json
│           │   └── ...
│           └── documents/
│               ├── <document_id>.json
│               └── ...
└── embedding/
    └── <dataset_name>/
        └── <version>/
            ├── queries/
            │   ├── <query_id>.npy
            │   └── ...
            └── documents/
                ├── <document_id>.npy
                └── ...

Step 3: Split the Queries Set

• Run the split.py script to split the queries into sets (e.g., for uprank and test).

Step 4: Calculate Embeddings

• Run the embedding_calculator.py file to compute embeddings for queries and documents.

Step 5: Run the Chain Scripts

• Run first_chain.py followed by dual_chain.py to perform the dual chain retrieval.