Skip to content

Conversation

@chilo-ms
Copy link
Contributor

Description

The newly added two APIs, CreateCustomOpDomains() and GetNumCustomOpDomains, are used when running inference on a model that contains EP-specific custom operations.

Workflow:

  1. The EP implements these functions to supply a list of OrtCustomOpDomain instances.
  2. The application either 1) calls SessionOptionsAppendExecutionProvider_V2() with an OrtEpDevice containing
    the plugin EP's factory or 2) enables auto ep selection.
  3. Then ORT either 1) SessionOptionsAppendExecutionProvider_V2() appends the provided OrtCustomOpDomains to the
    session options or 2) registers the OrtCustomOpDomains from the selected EP devices.

As a result, any session created from these session options will have these custom op domains registered
in ORT, ensuring that the custom ops are properly recognized and validated when the model is loaded.

Plugin EPs can provide two types of custom ops:

  1. A full OrtCustomOp with a concrete kernel implementation

    • This Example EP demonstrates this approach.
    • In GetCapability(), it calls EpGraphSupportInfo_AddSingleNode() to inform ORT
      that the custom node should NOT be fused or compiled. Instead, ORT should invoke
      the custom node's Compute() function at runtime.
  2. A "placeholder" OrtCustomOp with an empty kernel implementation

    • A compile-based Plugin EP can supply an OrtCustomOp whose CustomKernel::Compute()
      does nothing. The purpose is to satisfy model validation during model loading by
      registering the custom op as a valid operator in the session.
    • In GetCapability(), the EP should call EpGraphSupportInfo_AddNodesToFuse() to
      notify ORT that this custom node should be fused and compiled by the EP.
    • In Compile(), the EP executes its compiled bits to perform inference for
      the fused custom node.

Motivation and Context

Currently, the provider-bridge TRT RTX EP and TRT EP supports registering custom op domain list in session option so
that it can run model contains TRT specific custom ops.

This PR adds the same feature for plugin EP.

@chilo-ms
Copy link
Contributor Author

This PR is identical to #26759, as for some reasons i can't find the unresolved comment in the conversation and resolve it, it's blocking that PR to be merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants