Move request formatting to backend by sjmonson · Pull Request #478 · vllm-project/guidellm

sjmonson · 2025-11-21T23:04:59Z

Summary

Splits the RequestFormatter into multiple separate components across data loading, data preprocessing, and the backend. Necessary step for next attempt at multi-turn.

Details

Removed the RequestFormatter preprocessor
Added "Encode" preprocessors for audio, video, and image
Added a "Finalizer" stage after preprocessors to convert from column ordered data to request object
Added most of previous formatter code to backend

Test Plan

Run all know benchmark well-lit pathways

Related Issues

Resolves #

"I certify that all code in this PR is my own, except as noted below."

Use of AI

Includes AI-assisted code completion
Includes code generated by an AI application
Includes AI-generated tests (NOTE: AI written tests should have a docstring that includes ## WRITTEN BY AI ##)

sjmonson · 2026-01-26T21:26:57Z

A note on --request-type: Request type only really makes sense in the context of the openai_http backend. For upcoming backends (such as #178 and #548) I imagine request_type will be completely ignored. Ideally we would not give openai_http any special treatment and require request_type to either be specified as a --backend-kwarg (basically how its implemented now without the special --request-type shorthand) or as entirely separate backend for each request-type. For now I have kept --request-type to avoid user confusion.

src/guidellm/backends/openai/http.py

jaredoconnell

Looks good so far. I tested the default paths with a simple local benchmark, and that didn't break anything.

I have a few minor comments.

I'll give a final approving review once the tests are passing.

src/guidellm/data/loaders.py

src/guidellm/__main__.py

src/guidellm/settings.py

src/guidellm/backends/openai/request_formatter.py

jaredoconnell

The code looks good, and I tested the happy path. I didn't test edge cases or more advanced usage of the backend features.

We may want to wait for Mark's approval, too, before merging.

src/guidellm/backends/openai/http.py

src/guidellm/data/preprocessors/encoders.py

src/guidellm/backends/openai/http.py

markurtz

I think this all loks reasonable to move towards. There are a few things I'd like to do a pass over / discuss in a follow up, will connect later on that

Signed-off-by: Samuel Monson <smonson@redhat.com>

Fix schema tests Drop unused backend mock Move openai tests to match new structure Rewrite OpenAI backend tests Drop HuggingFace test that relies on network Fix failing backend tests Remove remaining references to request_type in tests Unify request_formatter and response_handler tests More unit test fixes Signed-off-by: Samuel Monson <smonson@redhat.com>

Signed-off-by: Samuel Monson <smonson@redhat.com>

jaredoconnell

I have no blocking comments. It looks good and I re-tested a simple benchmark.

src/guidellm/data/preprocessors/encoders.py

src/guidellm/__main__.py

sjmonson force-pushed the feat/split_formatter branch from 1e80237 to 687f702 Compare November 21, 2025 23:11

sjmonson force-pushed the feat/split_formatter branch from 8dbc7f5 to cef2e60 Compare December 9, 2025 18:23

sjmonson mentioned this pull request Dec 15, 2025

Add embeddings endpoint support #501

Open

26 tasks

sjmonson force-pushed the feat/split_formatter branch 2 times, most recently from c98234b to 28ece45 Compare January 26, 2026 16:43

sjmonson marked this pull request as ready for review January 27, 2026 20:18

sjmonson requested a review from jaredoconnell January 27, 2026 20:18

jaredoconnell reviewed Jan 27, 2026

View reviewed changes

src/guidellm/backends/openai/http.py Outdated Show resolved Hide resolved

sjmonson force-pushed the feat/split_formatter branch from 01c2dce to dbbd18a Compare January 28, 2026 18:20

jaredoconnell reviewed Jan 28, 2026

View reviewed changes

src/guidellm/data/loaders.py Show resolved Hide resolved

src/guidellm/__main__.py Show resolved Hide resolved

src/guidellm/settings.py Outdated Show resolved Hide resolved

src/guidellm/backends/openai/request_formatter.py Outdated Show resolved Hide resolved

sjmonson requested a review from markurtz January 29, 2026 15:15

sjmonson changed the title ~~Split up RequestFormatter~~ Move request formatting to backend Jan 29, 2026

sjmonson force-pushed the feat/split_formatter branch 3 times, most recently from 1ecb2a1 to 3f05211 Compare January 29, 2026 19:46

sjmonson requested a review from jaredoconnell January 29, 2026 21:11

jaredoconnell previously approved these changes Jan 30, 2026

View reviewed changes

src/guidellm/backends/openai/http.py Outdated Show resolved Hide resolved

src/guidellm/data/preprocessors/encoders.py Show resolved Hide resolved

src/guidellm/backends/openai/http.py Outdated Show resolved Hide resolved

sjmonson dismissed jaredoconnell’s stale review via 7da5b8e February 2, 2026 16:33

markurtz previously approved these changes Feb 3, 2026

View reviewed changes

sjmonson added 10 commits February 3, 2026 10:18

Split formatter across multiple stages

d15cdba

Signed-off-by: Samuel Monson <smonson@redhat.com>

Rework entrypoint to match new preprocess and finalizer flow

ec14fc3

Signed-off-by: Samuel Monson <smonson@redhat.com>

Migrate remaining code off of requests.arguments

39b0061

Signed-off-by: Samuel Monson <smonson@redhat.com>

Fix naming for finalizer registry

3f7b6ed

Signed-off-by: Samuel Monson <smonson@redhat.com>

Move OpenAI backend to subdir

3cc6094

Signed-off-by: Samuel Monson <smonson@redhat.com>

Move request formatting to separate file

52dce13

Signed-off-by: Samuel Monson <smonson@redhat.com>

Move request_type handling to backend

3f2538e

Signed-off-by: Samuel Monson <smonson@redhat.com>

Ignore args on backend failure

600f1fc

Signed-off-by: Samuel Monson <smonson@redhat.com>

Pass all columns to backend

9140d6f

Signed-off-by: Samuel Monson <smonson@redhat.com>

Move openai backend to openai/http.py

3d99c2f

Signed-off-by: Samuel Monson <smonson@redhat.com>

sjmonson added 9 commits February 3, 2026 10:18

Ensure model name is always set

6a1bf66

Signed-off-by: Samuel Monson <smonson@redhat.com>

Rename request-type to request-format

0da47f3

Signed-off-by: Samuel Monson <smonson@redhat.com>

Unify request_formatter and response_handler

e8844e9

Signed-off-by: Samuel Monson <smonson@redhat.com>

Move legacy API name handling to backend

f821d30

Signed-off-by: Samuel Monson <smonson@redhat.com>

Add parser for json lists

bec1ccd

Signed-off-by: Samuel Monson <smonson@redhat.com>

Improve various builtin docs

32db9a9

Signed-off-by: Samuel Monson <smonson@redhat.com>

Address review comment regarding endpoint names

c695ba7

Signed-off-by: Samuel Monson <smonson@redhat.com>

Add option for global preprocessor args

efa02cf

Signed-off-by: Samuel Monson <smonson@redhat.com>

sjmonson force-pushed the feat/split_formatter branch from 6dab76f to efa02cf Compare February 3, 2026 15:18

Fix startup failure

6a9e461

Signed-off-by: Samuel Monson <smonson@redhat.com>

sjmonson dismissed markurtz’s stale review via 6a9e461 February 3, 2026 18:26

Move base64 audio encoding to backend

974ba77

Signed-off-by: Samuel Monson <smonson@redhat.com>

jaredoconnell approved these changes Feb 3, 2026

View reviewed changes

src/guidellm/data/preprocessors/encoders.py Show resolved Hide resolved

src/guidellm/__main__.py Show resolved Hide resolved

src/guidellm/__main__.py Show resolved Hide resolved

sjmonson merged commit 4575a75 into main Feb 3, 2026
16 checks passed

sjmonson deleted the feat/split_formatter branch February 3, 2026 21:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move request formatting to backend#478

Move request formatting to backend#478
sjmonson merged 21 commits intomainfrom
feat/split_formatter

sjmonson commented Nov 21, 2025 •

edited

Loading

Uh oh!

sjmonson commented Jan 26, 2026 •

edited

Loading

Uh oh!

Uh oh!

jaredoconnell left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jaredoconnell left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

markurtz left a comment

Uh oh!

jaredoconnell left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

sjmonson commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Test Plan

Related Issues

Use of AI

Uh oh!

sjmonson commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

jaredoconnell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jaredoconnell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

markurtz left a comment

Choose a reason for hiding this comment

Uh oh!

jaredoconnell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sjmonson commented Nov 21, 2025 •

edited

Loading

sjmonson commented Jan 26, 2026 •

edited

Loading