Ludwig/liquidation docs (#718)

* merge


Former-commit-id: 510ea9c

* brontes-exports


Former-commit-id: 82776b2

* Delete brontes-exports/builder-info/03-27 directory (#537)



Former-commit-id: 21cae97

* test


Former-commit-id: 666f144

* :test


Former-commit-id: 39a67ef

* new range


Former-commit-id: b91dbd6

* teesting post run


Former-commit-id: 5bb0b0b

* export


Former-commit-id: 2ea1677

* mid run


Former-commit-id: 5b95638

* mid run update


Former-commit-id: d688799

* updated data


Former-commit-id: e4a473d

* checking cex data fix


Former-commit-id: 0025287

* testing


Former-commit-id: 1097f8d

* Delete brontes-exports directory (#585)



Former-commit-id: 794a489

* Ludwig/markout review (#624)

* wip

* make clippy happy

* wip

* finished review, added docs

Former-commit-id: 2282560

* first draft


Former-commit-id: f2759d54d02d76e71e709eaec22eb3d0e8b4edcd

* create cex-quotes feature

Former-commit-id: 85ad5f1926a0796bc00c13d292059ac5c297d4c5

* reduced quality paramater for optimistic to 65%


Former-commit-id: db271cdb5af9ca5362a9aa9d30280478be783362

* add note on builder pnl calc limitations


Former-commit-id: 86108d828f13acba04625f6b54b51cbac2bca373

* todo update main


Former-commit-id: d03565bd35fe6e9d6b125fc501b7985a26a91358

* updated docs

* deleted tree.json

---------

Co-authored-by: root <[email protected]>

book/architecture/inspectors.md

@@ -72,6 +72,8 @@ After processing the inspector results, we [calculate the block builder’s PnL]
   - **Proposer Payments:** ETH paid by the builder to the block proposer.
   - **Transaction Sponsorship:** ETH spent by the builder to [sponsor](https://titanbuilder.substack.com/p/titan-tech-teatime-1) transactions within the block.
 
+> **Note:** Some builders secretly refund parts of priority fees to searchers or order flow generators (tg bots for example). We can't track these kickbacks without knowing the addresses involved. If you have this information, please share it to help us improve our calculations.
+
 ### Step 4: Store Results
 
 Finally the resulting [`MevBlock`](./database/schema/mev_blocks.md#mevblock-fields) and [`Vec<Bundles>`](./database/schema/mev_blocks.md#bundle-fields) are written to the database in the `MevBlocks` table.

book/installation/installation.md

@@ -4,7 +4,7 @@ Brontes runs on Linux and macOS. Currently it must be installed from source, tho
 
 ## Hardware Requirements
 
-Running Brontes with Reth in the background requires hardware that's more robust than what might typically be recommended for Reth alone. This ensures optimal performance and reliability, especially for those aiming to keep up with the blockchain tip.
+Running Brontes with Reth in the background requires hardware that's more robust than what might typically be recommended for Reth alone.
 
 - See [reth installation guide](https://paradigmxyz.github.io/reth/installation/installation.html) for more details on Reth's hardware requirements.
 

book/intro.md

@@ -6,27 +6,41 @@ _Documentation for Brontes users and developers._
 <img alt="Twitter Follow" src="https://img.shields.io/twitter/follow/SorellaLabs?style=social">
 </a>
 
-**Brontes** is a blazingly fast and modular blockchain analytics pipeline, designed with a focus on systematically identifying **MEV**.
+**Brontes** is a _blazingly_ fast and modular blockchain analytics pipeline, designed to systematically identify **MEV**.
 
 <div style="text-align: center;">
     <img src="https://raw.githubusercontent.com/0xvanbeethoven/brontes-img/main/Brontes.png" alt="Brontes" style="border-radius: 20px; width: 400px; height: auto;">
 </div>
 
-## What is Brontes?
+## Why Brontes?
 
-[Brontes](https://github.com/SorellaLabs/brontes), developed by [Sorella Labs](https://twitter.com/Sorellalabs), is an advanced blockchain analytics pipeline built on top of [Reth](https://github.com/paradigmxyz/reth/). It transforms raw Ethereum data into a structured, analyzable format, enriched with off-chain data. Brontes provides a pipelined and modular framework for complex analytics, enabling data scientists, and researchers to focus on their analysis and methodology without being burdened by the intricacies of data preprocessing.
+[Brontes](https://github.com/SorellaLabs/brontes), developed by [Sorella Labs](https://twitter.com/Sorellalabs), is a blockchain analytics pipeline built on top of [Reth](https://github.com/paradigmxyz/reth/). It addresses a critical challenge in blockchain research: the overwhelming flood of data and tedious preprocessing that often derail researchers from their core focus.
 
-## Why Brontes?
+**Key features:**
+
+- Transforms raw Ethereum data into a structured, analyzable format
+- Enhances analysis with off-chain data (metadata, CEX prices, p2p data...)
+- Provides a modular framework to easily implement user-defined inspectors for custom analytics
+
+Blockchain data analysis, especially at the trace level, can overwhelm even seasoned researchers. While a few masochists might find satisfaction in the chore of data preprocessing and normalization, most of us are captivated by the intellectual challenge of crafting innovative analytical techniques.
+
+Our Inspector Framework allows you to focus on developing and applying novel methodologies. By eliminating initial hurdles, Brontes frees you to immerse yourself in creative analysis rather than getting bogged down in preprocessing.
+
+## Who is this for?
 
-Analyzing blockchain data, particularly at the transaction trace level, is an overwhelming and time-consuming process. The sheer volume of data and the effort required for data classification, and normalization often hinder the ability of data scientists to focus on pioneering analysis and developing new methodologies.
+Brontes is designed for:
 
-Albeit a few masochists might revel in the painstaking but necessary prep work, the rest of us don't exactly find joy in it; the true thrill emerges in the exploration and refinement of new methodologies. This insight is what shapes our Inspector Framework and Brontes at its core. Our aim is to strip away the burden of these initial steps, giving you the freedom to leap straight into pioneering analysis.
+- Blockchain researchers and data scientists
+- MEV analysts and strategists
+- DeFi protocol developers
+- Anyone working with large-scale Ethereum data
 
 ## Navigating This Book
 
-- To install Brontes, refer to the [installation guide](./installation/installation.md).
-- To run Brontes, follow the instructions in the [run guide](./run/run_brontes.md).
-- To learn about how brontes works under the hood, see the [architecture doc](./architecture/intro.md).
+- **Installation**: Get started with our [step-by-step guide](./installation/installation.md)
+- **Running Brontes**: Follow our [quick-start instructions](./run/run_brontes.md)
+- **Under the Hood**: Explore Brontes' [architecture](./architecture/intro.md)
+- **MEV Identification**: Dive into our [mev-inspector methodologies](./mev_inspectors/intro.md)
 
 ## Licensing and Community Involvement
 

book/mev_inspectors/atomic-arb.md

@@ -2,9 +2,10 @@
 
 The Atomic Arbitrage Inspector is designed to detect and analyze the profitability of various forms of atomic arbitrage.
 
+//TODO: Add transfer to swap creation to identify arbs on missing exchanges
 **What is an atomic arbitrage?**
 
-An atomic arbitrage is a type of arbitrage that involves multiple trades that are executed atomically and result in a profit for the trader. Typically, atomic arbitrages involve arbitraging price differences between different liquidity pools.
+An atomic arbitrage is a type of arbitrage that involves multiple trades that are executed in a single transaction and result in a profit for the trader. Typically, these involve arbitraging price differences between different liquidity pools.
 
 ## Methodology
 
@@ -14,7 +15,7 @@ The inspector retrieves transactions in the block that involve `swap`, `transfer
 
 ### Step 2: Identify and Classify Potential Atomic Arbitrages
 
-In this step, we analyze the sequence of swaps within each transaction to identify and categorize potential atomic arbitrages.
+In this step, we analyze the sequence of swaps within each transaction to identify and categorize potential arbitrages.
 
 #### Classification Criteria
 
@@ -32,17 +33,10 @@ We categorize atomic arbitrages into four distinct types:
 3. **Stablecoin**: Arbitrages involving stablecoin pairs
 4. **Long Tail**: Complex patterns not fitting the above categories
 
-The arbitrage type will determine the filtering conditions applied in the subsequent steps.
+The arbitrage type will determine the filtering conditions applied subsequent steps.
 
 > **Note:** This is by no means a comprehensive list of atomic arbitrage types. If you have discovered atomic arbitrages that do not fit these criteria, please let us know. We would love to expand our classification to include new patterns and improve our analysis.
 
-We classify atomic arbitrages in these distinct types:
-
-- Triangle,
-- CrossPair
-- StablecoinArb,
-- LongTail
-
 #### For Zero or One Swap
 
 - Not considered an arbitrage opportunity. We move to the next transaction.

book/mev_inspectors/cex_dex.md

@@ -1,6 +1,90 @@
 # Cex-Dex Inspector
 
-The cex-dex inspector operates in
+The Cex-Dex inspector identifies arbitrage between centralized and decentralized exchanges. While on-chain DEX trades are visible, CEX trades must be inferred. Using available CEX trade data, the inspector estimates likely CEX trade prices. This allows for approximating the full arbitrage strategy and its potential profitability.
+
+**What is Cex-Dex Arbitrage?**
+
+Cex-Dex arbitrage involves exploiting price differences between centralized and decentralized exchanges. This arbitrage opportunity arises because centralized exchange operate in near continuous time while decentralized exchange operate in discrete time, updating only when a new block is produced. This causes the DEX price to lag behind the CEX price, creating opportunities for arbitrage.
+
+## Methodology
+
+### Step 1: Retrieve Relevant Transactions
+
+The inspector retrieves transactions in the block that involve `swap`, `transfer`, `eth_transfer`, `FlashLoan`, `batch_swap` or `aggregator_swap` actions on decentralized exchanges.
+
+### Step 2: Identify Potential Cex-Dex Arbitrage
+
+For each transaction:
+
+1. Filter out solver settlement and DeFi automation transactions.
+2. Extract DEX swaps and transfers from the transaction.
+3. If no DEX swaps are found, attempt to convert transfers to swaps for labeled CEX-DEX bots.
+4. Filter out triangular arbitrage (where the first and last token in the swap sequence are the same).
+
+### Step 3: Estimate Centralized Exchange Prices
+
+For each DEX swap, the inspector estimates corresponding CEX prices using two methods:
+
+#### A. Time Window Volume Weighted Average Markout (VWAM)
+
+1. Set a default time window around the block timestamp.
+2. Collect all trades within this window and calculate the total volume.
+3. If volume is insufficient, dynamically extend the time window.
+4. Apply a bi-exponential decay function to weight trades based on their temporal proximity to the block timestamp.
+5. Calculate an Adjusted Volume Weighted Average Price (VWAP) using these weights.
+
+#### B. Optimistic VWAP
+
+1. Collect all trades within a set time window.
+2. Sort trades by price and select the most favorable trades up to the required volume.
+3. Calculate VWAP based on these selected trades.
+
+### Step 4: Calculate Potential Arbitrage Profits
+
+For each swap and CEX price estimate:
+
+1. Calculate the price difference between DEX and CEX.
+2. Estimate potential profit for both buying on DEX and selling on CEX, and vice versa.
+3. Calculate profits using both mid-price and ask-price scenarios.
+
+### Step 5: Aggregate and Analyze Results
+
+1. Calculate profits for each CEX individually and for a global VWAM across all exchanges.
+2. Determine the most profitable route across all exchanges.
+3. Calculate optimistic profits based on the Optimistic VWAP.
+
+### Step 6: Account for Gas Costs
+
+Subtract the transaction's gas cost from the calculated profits for each scenario.
+
+### Step 7: Validate and Filter Potential Arbitrages
+
+A transaction is considered a valid Cex-Dex arbitrage if it meets any of the following conditions:
+
+1. Profitable based on global VWAM or optimistic estimates.
+2. Profitable on multiple exchanges.
+3. Executed by an address with significant history of Cex-Dex arbitrage (>40 previous trades).
+4. Labeled as a known Cex-Dex arbitrageur.
+5. Is a private transaction with direct builder payment.
+6. Uses a known MEV contract.
+7. Shows significant profit on a single exchange (excluding stable coin pairs).
+
+### Step 8: Handle Edge Cases and Outliers
+
+1. Filter out high-profit outliers (>$10,000 profit) on specific exchanges (Kucoin, Okex) to avoid false positives.
+2. Apply stricter validation for stable coin pair arbitrages.
+
+### Step 9: Prepare Final Output
+
+For validated Cex-Dex arbitrages, compile detailed information including:
+
+1. Transaction details (hash, gas costs, etc.)
+2. DEX swap details
+3. Estimated CEX prices and trade details
+4. Calculated profits for various scenarios (global VWAM, per-exchange, optimistic)
+5. Time windows used for price estimations
+
+The inspector outputs this information as a `Bundle` containing `BundleData::CexDex` for further analysis and reporting.
 
 ## Markout with trades
 

book/mev_inspectors/jit-liquidity.md

@@ -135,18 +135,18 @@ For non standard JIT patterns, we employ a recursive strategy:
 
 For confirmed JIT bundles:
 
-1. Calculate searcher revenue: Balance deltas of searcher addresses & sibling address (e.g piggy bank address) if applicable
+1. Calculate searcher revenue: Balance deltas for searcher addresses & sibling address (e.g piggy bank address) if applicable, using the all mint, burn and collect actions.
 2. Calculate searcher cost: Sum of gas costs for all attacker transactions
 3. Profit = Revenue - Cost
-4. Applying a maximum profit threshold
+4. Filter false positives using the maximum profit threshold
 
 ### Step 5: Generate JIT Bundle
 
 For confirmed opportunities:
 
-1. Construct a `JitLiquidity` structure with detailed transaction information
-2. Create a `Bundle` with a summary header and `JitLiquidity` data
-3. For recursive analyses, deduplicate results and keep the largest JIT bundle
+1. Construct a `JitLiquidity` type
+2. Create a `Bundle` with a summary `BundleHeader` and `JitLiquidity` data
+3. For recursive analyses, deduplicate results, by retaining the largest JIT bundle when multiple JITs with overlapping transaction sets are detected.
 
 ### Step 6: Identify JIT CexDex
 

book/mev_inspectors/liquidation.md

@@ -1 +1,54 @@
 # Liquidation Inspector
+
+The Liquidation Inspector is designed to detect and analyze the profitability of liquidation events.
+
+**What is a Liquidation?**
+
+A liquidation occurs when a borrower's collateral is forcibly sold to repay their outstanding debt, typically when the collateral's value falls below a certain threshold.
+
+## Methodology
+
+### Step 1: Retrieve Relevant Transactions
+
+The inspector retrieves transactions in the block that involve `swap` or `liquidation` actions.
+
+### Step 2: Identify Potential Liquidations
+
+For each relevant transaction, we:
+
+1. Split the actions into swaps and liquidations.
+2. Filter out transactions with no liquidation events.
+
+### Step 3: Analyze Liquidation Candidates
+
+For each potential liquidation, we:
+
+1. Collect all addresses involved in the transaction.
+2. Calculate the balance changes (deltas) for all actions in the transaction.
+
+### Step 4: Calculate Profitability
+
+We apply specific criteria to determine the profitability of each liquidation:
+
+1. Calculate USD value of token transfers using DEX pricing data.
+2. Compute gas costs for the transaction.
+3. Determine profitability by subtracting gas costs from revenue.
+4. Apply a maximum profit threshold to filter out unrealistic opportunities.
+
+### Step 5: Generate Liquidation Bundle
+
+For confirmed liquidation opportunities:
+
+1. Construct a `Liquidation` structure containing:
+
+   - Liquidation transaction hash
+   - Liquidation swaps
+   - Liquidation events
+   - Gas details
+
+2. Create a `Bundle` with:
+   - A header summarizing key information (profit, gas used, transaction hash)
+   - The detailed `Liquidation` data
+
+> **Note on Pricing:**
+> The inspector uses DEX pricing data to value token transfers. If reliable pricing data is unavailable, the liquidation is flagged, and profit is set to zero to avoid false positives.

book/run/run_brontes.md

@@ -10,7 +10,7 @@ brontes run
 
 ### Required Parameters
 
-When running Brontes, you must a start block and end block
+When running Brontes, you must input a start block and end block
 
 - **Start Block**: The block number from which Brontes begins processing.
 - **End Block**: The block number at which Brontes stops processing. If omitted, Brontes will run historically and continue at the tip until manually stopped, provided you have access to the metadata API.

crates/bin/Cargo.toml

@@ -99,11 +99,8 @@ tikv-jemallocator = { version = "0.5.0", features = [
 [features]
 default = [
   "brontes-core/default",
-  "brontes-database/default",
   "brontes-classifier/default",
-  "brontes-inspect/default",
   "jemalloc",
-  "cex-dex-markout",
 ]
 
 jemalloc = ["brontes-metrics/jemalloc"]
@@ -112,10 +109,10 @@ dhat-heap = []
 
 sorella-server = ["local-reth", "local-clickhouse"]
 
-cex-dex-markout = [
-  "brontes-types/cex-dex-markout",
-  "brontes-database/cex-dex-markout",
-  "brontes-inspect/cex-dex-markout",
+cex-dex-quotes = [
+  "brontes-types/cex-dex-quotes",
+  "brontes-database/cex-dex-quotes",
+  "brontes-inspect/cex-dex-quotes",
 ]
 
 

crates/bin/src/cli/db/init.rs

@@ -27,22 +27,22 @@ pub struct Init {
     ///     CexTrades
     #[arg(long, short, requires = "init_libmdbx", value_delimiter = ',')]
     pub tables_to_init:         Option<Vec<Tables>>,
-    #[cfg(not(feature = "cex-dex-markout"))]
+    #[cfg(feature = "cex-dex-quotes")]
     /// The sliding time window (BEFORE) for cex prices relative to the block
     /// timestamp
     #[arg(long = "price-tw-before", default_value = "0.5")]
     pub cex_time_window_before: f64,
-    #[cfg(not(feature = "cex-dex-markout"))]
+    #[cfg(feature = "cex-dex-quotes")]
     /// The sliding time window (AFTER) for cex prices relative to the block
     /// timestamp
     #[arg(long = "price-tw-after", default_value = "1.0")]
     pub cex_time_window_after:  f64,
-    #[cfg(feature = "cex-dex-markout")]
+    #[cfg(not(feature = "cex-dex-quotes"))]
     /// The sliding time window (BEFORE) for cex trades relative to the block
     /// timestamp
     #[arg(long = "trades-tw-before", default_value = "0.5")]
     pub cex_time_window_before: f64,
-    #[cfg(feature = "cex-dex-markout")]
+    #[cfg(not(feature = "cex-dex-quotes"))]
     /// The sliding time window (AFTER) for cex trades relative to the block
     /// timestamp
     #[arg(long = "trades-tw-after", default_value = "2.0")]
@@ -94,9 +94,9 @@ impl Init {
             task_executor
                 .spawn_critical("init", async move {
                     let mut tables = Tables::ALL.to_vec();
-                    #[cfg(not(feature = "cex-dex-markout"))]
+                    #[cfg(feature = "cex-dex-quotes")]
                     tables.retain(|t| !matches!(t, Tables::CexTrades));
-                    #[cfg(feature = "cex-dex-markout")]
+                    #[cfg(not(feature = "cex-dex-quotes"))]
                     tables.retain(|t| !matches!(t, Tables::CexPrice));
 
                     let multi = MultiProgress::default();

crates/bin/src/cli/run.rs

@@ -167,9 +167,9 @@ impl RunArgs {
             .inspectors
             .as_ref()
             .map(|f| {
-                #[cfg(not(feature = "cex-dex-markout"))]
+                #[cfg(feature = "cex-dex-quotes")]
                 let cmp = Inspectors::CexDex;
-                #[cfg(feature = "cex-dex-markout")]
+                #[cfg(not(feature = "cex-dex-quotes"))]
                 let cmp = Inspectors::CexDexMarkout;
                 f.len() == 1 && f.contains(&cmp)
             })

crates/brontes-database/brontes-db/Cargo.toml

@@ -145,9 +145,8 @@ brontes-types.workspace = true
 alloy-primitives.workspace = true
 
 [features]
-default = ["cex-dex-markout"]
 sorella-server = ["local-reth", "local-clickhouse"]
-cex-dex-markout = ["brontes-types/cex-dex-markout"]
+cex-dex-quotes = ["brontes-types/cex-dex-quotes"]
 tests = ["brontes-pricing/tests"]
 local-reth = [
   "brontes-core/local-reth",

crates/brontes-database/brontes-db/src/clickhouse/cex_config.rs

@@ -12,7 +12,7 @@ impl CexDownloadConfig {
     }
 }
 
-#[cfg(feature = "cex-dex-markout")]
+#[cfg(not(feature = "cex-dex-quotes"))]
 impl Default for CexDownloadConfig {
     fn default() -> Self {
         Self {
@@ -28,7 +28,7 @@ impl Default for CexDownloadConfig {
     }
 }
 
-#[cfg(not(feature = "cex-dex-markout"))]
+#[cfg(feature = "cex-dex-quotes")]
 impl Default for CexDownloadConfig {
     fn default() -> Self {
         Self {