SEO Middleware for Next.js

SEO Middleware is a lightweight Next.js package designed to improve SEO scores for traditional SERPs (a.k.a. Google, Bing, Yahoo, etc.) and modern AI agents and bots (a.k.a. ChatGPT, Grok, Perplexity, etc.).

This middleware is powered by ostr.io pre-rendering engine and paired with the global CDN and smart caching layer. seo-middleware-nextjs establishes access to Pre-rendering Engine that significantly increases crawling budget, improves SEO scores, link previews, and Web Vitals/Lighthouse performance metrics (like TTFB) — all without changes in the existing codebase.

SEO middleware intelligently reroutes bot traffic to ostr.io rendering endpoints, minimizing server load, reducing database queries, and lowering infrastructure costs.

Why SEO Middleware?

• 🕸 Executes JavaScript — get rendered HTML page and its content; Outperforms SSR and allows to reduce server load by disabling SSR;
• 🏎️ Expands Crawl Budget — Improves timings for dynamic and static pages via advanced CDN and caching;
• 🚀 Boosts Web Vitals and Lighthouse scores;
• 🎛️ Improves TTFB, LCP, INP, CLS, and other Web Vitals and LightHouse metrics positively enhancing overall SEO score;
• 🖥 Supports PWAs and SPAs;
• 📱 Supports mobile-like crawlers;
• 💅 Supports styled-components;
• ⚡️ Supports AMP (Accelerated Mobile Pages);
• 🤓 Works with Content-Security-Policy and other complex front-end security rules;
• 📦 This package implemented in strict TypeScript and exports all necessary types;
• ❤️ Search engines and social network crawlers love optimized pages that delivered in blazingly-fast manner;
• 📱 Consistent link previews in messaging apps, like iMessage, Messages, Facebook, Slack, Telegram, WhatsApp, Viber, VK, Twitter, and other apps;
• 💻 Image, title, and description previews for links posted at social networks, like Facebook, X/Twitter, Instagram, and other social networks.

Related packages and docs

• Detailed Pre-rendering Engine Documentation
• Cloud Integrations
  • CloudFlare Worker Integration
  • Netlify Integration
• Web Server Integrations
  • Nginx Integration
  • Apache Integration
• Related Packages
  • Node.js Integration via NPM package
  • Meteor.js Integration via Atmosphere Package

ToC

• Installation
• Basic usage
• API
  • Constructor new SEOMiddleware()
  • TS Types
• Speed-up rendering
• Detect request from Pre-rendering engine during runtime
• Detect type of Pre-rendering engine
• Debugging

About Package

This package works as an "edge function", intercepting requests from crawlers and social media bots received by Next.js application. It seamlessly proxies those requests to a pre-rendering engine, which returns fully-rendered static HTML — optimized for indexing and rich previews.

Built with developers in mind, seo-middleware-nextjs is lightweight, well-structured, and easy to customize. Whether you're scaling a production app or prototyping, it's designed to be hackable and flexible enough to fit any project. By offloading bot traffic to the pre-rendering engine, it helps reduce backend load and improve server performance effortlessly.

Tip

This package was originally developed for pre-rendering engine by ostr.io. But it's not limited to, and can proxy-pass requests to any other rendering-endpoint.

Installation

Install seo-middleware-nextjs package from NPM:

# using npmjs
npm install seo-middleware-nextjs --save

# using yarn
yarn add seo-middleware-nextjs

Usage

Setup SEO middleware in few lines of code, see this ready to copy-paste middleware.ts file

Basic usage

Start with creating middleware.ts in /src/ or root directory of Next.js project, then import seo-middleware-nextjs package:

import { SEOMiddleware } from 'seo-middleware-nextjs';

Register middleware handle:

// inside middleware.ts file:
import { SEOMiddleware } from 'seo-middleware-nextjs';

const seoMiddleware = new SEOMiddleware({
  auth: 'Basic dGVzdDp0ZXN0'
});

// EXPORT MIDDLEWARE FUNCTION
export const middleware = seoMiddleware.createMiddleware();

// EXPORT config.matcher
// @see https://nextjs.org/docs/app/api-reference/file-conventions/middleware#matcher
export const config = {
  matcher: '/((?!api|_next/static|_next/image|_next/webpack-hmr|\\.well-known|favicon.ico).*)'
};

Important

config.matcher can not be set to variable or changed/assigned during runtime; It can hold array of paths or RegExp-alike string, see official Next.js middleware matcher docs for more details

Tip

The API credentials and instructions can be found in "Integration Guide" of Pre-rendering Panel, — click on the name of a website, then on Integration Guide

API

Create new SEOMiddleware instance in /src/middleware.ts file.

Constructor

new SEOMiddleware(opts: SEOMiddlewareOptions);

• opts {SEOMiddlewareOptions} - Configuration options
• opts.auth {string} - Auth string in the next format: Basic xxx...xxx. Can be set via an environment variables: SPIDERABLE_SERVICE_AUTH or PRERENDER_SERVICE_AUTH or OSTR_AUTH.
• opts.renderingEndpoint {string} - Valid URL to Pre-rendering engine. Default: https://render.ostr.io
• opts.keepGetQuery {boolean} — Toggle support for get-query. Default true; It's safe to set to false if your app doesn't use get-query for its functionality
• opts.supportEscapedFragment {boolean} — Toggle support for ?_escaped_fragment_= get query. Default true (recommended to keep it true)
• opts.retries {number} — Amount of retries sending request to pre-rendering engine in case of the failure. Default: 2
• opts.ignoredPaths {RegExp|false} - Regular Expression with paths that should get ignored by SEO Middleware. Default false
• opts.botAgents {string[]} - An array of strings (case insensitive) with additional User-Agent names of crawlers that needs to get intercepted. The default list includes all modern crawler's User-Agents, see import {BOT_AGENTS} for more datils
• opts.ignoredExtensions {string[]} — An array of strings with static files extensions that should get ignored by SEO Middleware. The default value holds all modern files extensions
• opts.logger {Console|Winston|Pino} — Logger facility for info messages, warnings, and errors. Default: console
• opts.debug {boolean} — Enable debug logs. Default: false

// inside middleware.ts file:
import { SEOMiddleware } from 'seo-middleware-nextjs';

// INITIATE SEOMiddleware CLASS
const seoMiddleware = new SEOMiddleware({
  auth: 'Basic dGVzdDp0ZXN0',
  renderingEndpoint: 'https://render.ostr.io',
  ignoredPaths: /\/images\/|\/uploads\//,
  supportEscapedFragment: true,
  keepGetQuery: true,
  retries: 4,
});

// EXPORT MIDDLEWARE FUNCTION
export const middleware = seoMiddleware.createMiddleware();

// EXPORT config.matcher
// THIS ONE EXCLUDES ALL STATIC AND API ENDPOINTS
// ADD ANY OTHER URLs THAT SHOULD BE EXCLUDED FROM PRE-RENDERING
// OR USE IS AS "ONLY" PATHs MATCHING, EX.: '/(articles/?.*|products/?.*|only-this-path)' {string}
// @see https://nextjs.org/docs/app/api-reference/file-conventions/middleware#matcher
export const config = {
  matcher: '/((?!api|_next/static|_next/image|_next/webpack-hmr|\\.well-known).*)'
};

Tip

We provide various options for renderingEndpoint as "Rendering Endpoints", each endpoint has its own features to fit different project needs

Types

Full list of exported types

export declare const IGNORE_EXTENSIONS: string[];
export declare const BOT_AGENTS: string[];
export type SEOMiddlewareLogger = Pick<Console, 'debug' | 'info' | 'warn' | 'error' | 'log'>;

export interface SEOMiddlewareOptions {
  auth?: string;
  supportEscapedFragment?: boolean;
  keepGetQuery?: boolean;
  ignoredPaths?: false | RegExp;
  logger?: SEOMiddlewareLogger;
  ignoredExtensions?: string[];
  botAgents?: string[];
  renderingEndpoint?: string;
  retries?: number;
  debug?: boolean;
}

export declare class SEOMiddleware {
  auth: string;
  supportEscapedFragment: boolean;
  keepGetQuery: boolean;
  ignoredPaths: false | RegExp;
  logger: SEOMiddlewareLogger;
  ignoredExtensions: Set<string>;
  botAgents: AhoCorasick;
  renderingEndpoint: string;
  renderingBase: string;
  retries: number;
  debug: boolean;
  constructor(opts: SEOMiddlewareOptions);
  createMiddleware(): (req: NextRequest) => Promise<Response>;
}

Configuration via env.vars

Authentication token can be set via various environment variables:

OSTRIO_AUTH='Basic xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
PRERENDER_SERVICE_AUTH='Basic xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
SPIDERABLE_SERVICE_AUTH='Basic xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

Speed-up rendering

To speed-up rendering, JS-runtime should tell to the Pre-rendering engine when the page is ready. Initialize early in application's code window.IS_RENDERED = false, and once the page is ready set window.IS_RENDERED variable to true. Example:

// At the beginning of the client's bundle:
window.IS_RENDERED = false;

afterPageAndAssetsAreLoaded(() => {
  // Once page and assets are ready:
  window.IS_RENDERED = true;
});

For more details on IS_RENDERED variable and other available optimizations see this documentation

Detect request from Pre-rendering engine during runtime

Pre-rendering engine will set window.IS_PRERENDERING global variable to true. Detecting requests from pre-rendering engine is easy as:

if (window.IS_PRERENDERING) {
  // This request is coming from Pre-rendering engine
}

For more details on IS_PRERENDERING variable see this documentation

Detect type of the Pre-rendering engine

Like browsers, — crawlers and bots may request page as "mobile" (small screen touch-devices) or as "desktop" (large screens without touch-events) the pre-rendering engine supports these two types. For cases when content needs to get optimized for different screens pre-rendering engine will set window.IS_PRERENDERING_TYPE global variable to desktop or mobile

if (window.IS_PRERENDERING_TYPE === 'mobile') {
  // This request is coming from "mobile" web crawler and "mobile" pre-rendering engine
} else if (window.IS_PRERENDERING_TYPE === 'desktop') {
  // This request is coming from "desktop" web crawler and "desktop" pre-rendering engine
} else {
  // This request is coming from user
}

For more details on IS_PRERENDERING_TYPE variable see this documentation

Debugging

1. To enable debug messages set opts.debug = true or DEBUG=true environment variable.
2. Remove export config.matcher to exclude its misconfiguration

Additionally:

• Set auth to Basic dGVzdDp0ZXN0 (credentials for testing)
• Set renderingEndpoint to https://render-bypass.ostr.io avoiding cached results, read more about rendering endpoints

import { SEOMiddleware } from 'seo-middleware-nextjs';

const seoMiddleware = new SEOMiddleware({
  auth: 'Basic dGVzdDp0ZXN0',
  renderingEndpoint: 'https://render-bypass.ostr.io',
  debug: true,
});

export const middleware = seoMiddleware.createMiddleware();

---

Read Full Pre-rendering Engine Documentation