[POC][OV] Support OpenVINO as Keras 3 backend

[rkazants]

Details: Support OpenVINO as Keras 3 backend. This is inference-only backend. In order to switch on this, define environment variable as follows: os.environ["KERAS_BACKEND"] = "openvino" or use set_backend.

Here is an example how it works:
Install OpenVINO pip install openvino -U

import os
os.environ["KERAS_BACKEND"] = "openvino"
import numpy as np
import keras

# build a simple network
x1 = keras.Input(shape=(2,), dtype='float32')
x2 = keras.Input(shape=(2,), dtype='float32')
add = keras.layers.Add()([x1, x2])
sigmoid = keras.activations.sigmoid(add)
model = keras.Model(inputs={'x1': x1, 'x2': x2}, outputs={'sigmoid': sigmoid})
model.summary()

# infer using OpenVINO
out = model.predict({'x1': np.array([[1.0, 2.0]], dtype=np.float32),
                     'x2': np.array([[3.0, 4.0]], dtype=np.float32)})
print('openvino out = ', out)

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[fchollet]

Thanks for the PR!

[fchollet]

This should not be backend-specific.

[rkazants]

Hi @fchollet, please advice how to avoid this code. Keras common requires it to be of KerasTensor type during calling op-by-op. How to relax this check and allow to pass ov type object into ops wrappers?

[fchollet]

All you need is for backend.is_tensor(x) to return True for the types you consider to be tensors.

[rkazants]

@fchollet, I redefined it for my backend but openvino.runtime.Output does not have dtype attribute and exception is raised here: https://github.com/keras-team/keras/blob/master/keras/src/dtype_policies/dtype_policy.py#L200
We retrieve dtype using method get_element_type. I think backend.standardize_dtype should accept x instead of x.dtype.
How do you think I can solve this exception? Thanks for your help in advance.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 0.35907% with 555 lines in your changes missing coverage. Please review.

Project coverage is 81.05%. Comparing base (00c904f) to head (fa6b461).
Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
keras/src/backend/openvino/numpy.py	0.00%	178 Missing ⚠️
keras/src/backend/openvino/core.py	0.00%	117 Missing ⚠️
keras/src/backend/openvino/trainer.py	0.00%	116 Missing ⚠️
keras/src/backend/openvino/nn.py	0.00%	52 Missing ⚠️
keras/src/backend/openvino/__init__.py	0.00%	24 Missing ⚠️
keras/src/backend/openvino/math.py	0.00%	18 Missing ⚠️
keras/src/backend/openvino/random.py	0.00%	14 Missing ⚠️
keras/src/backend/openvino/linalg.py	0.00%	12 Missing ⚠️
keras/src/backend/openvino/rnn.py	0.00%	7 Missing ⚠️
keras/src/backend/openvino/image.py	0.00%	4 Missing ⚠️
... and 5 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #19727      +/-   ##
==========================================
- Coverage   81.98%   81.05%   -0.93%     
==========================================
  Files         514      525      +11     
  Lines       47120    47738     +618     
  Branches     7401     7455      +54     
==========================================
+ Hits        38633    38696      +63     
- Misses       6701     7251     +550     
- Partials     1786     1791       +5     

Flag	Coverage Δ
keras	80.91% <0.35%> (-0.93%)	⬇️
keras-jax	64.16% <0.17%> (-0.71%)	⬇️
keras-numpy	59.17% <0.35%> (-0.66%)	⬇️
keras-tensorflow	65.14% <0.17%> (-0.73%)	⬇️
keras-torch	64.11% <0.17%> (-0.71%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[gbaned]

Hi @rkazants Can you please resolve the conflicts? Thank you!

[rkazants]

Hi @rkazants Can you please resolve the conflicts? Thank you!

I am working on this PR. I am trying to resolve comments.

Best regards,
Roman

[gbaned]

Hi @rkazants Any update on this PR? Please. Thank you!

[github-actions]

This PR is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

[rkazants]

I am just back from vacation. I plan to continue from the next week.

[github-actions]

This PR is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

[gbaned]

Hi @rkazants Any update on this PR? Please. Thank you!

[rkazants]

Hi @rkazants Any update on this PR? Please. Thank you!

Hi @gbaned, please anticipate the update early this week. Sorry for the delay.

Thanks,
Roman

[rkazants]

@fchollet, please review. all other comments are addressed.

[rkazants]

@fchollet,
Also, I would suggest separate integration tests for openvino backend. I want to have test suite for each keras operation where it creates one layer model and test inference results. As reference, we can use numpy backend's results, for example. This way we will add more op coverage support step-by-step and validate new functionality. How do you think?

Thanks,
Roman

[rkazants]

Hi @fchollet, look forward to receive your review:)
I am ready to address your comments asap these days.

Thanks in advance,
Roman

[rkazants]

@fchollet, kindly reminder about review.

Thank you,
Roman

[rkazants]

Hi @fchollet, any update? Please inform me if any concerns.

Thanks,
Roman

[rkazants]

Hi @fchollet, @gbaned,

Could you please take a look? Let me know if any minimal scope of supported ops are required for this PR. I will add it here.

Thanks,
Roman

[fchollet]

Sorry, only seeing this now. It looks better! It seems the set of ops currently implemented is quite small, however. What kinds of end-to-end workflows are currently working? What level of op coverage do you think you can achieve?

[fchollet]

All you need is for backend.is_tensor(x) to return True for the types you consider to be tensors.

[rkazants]

Sorry, only seeing this now. It looks better! It seems the set of ops currently implemented is quite small, however. What kinds of end-to-end workflows are currently working? What level of op coverage do you think you can achieve?

Hi @fchollet, currently we support predict method only for inference. evaluate is not supported because it requires to support loss function and run it on the same backend. That is not convenient to do with our backend because eager execution is absent.

In future, from my quick glance we will be able to cover solid number of operations because OpenVINO opset is quite well developed and support a lot of TF, PyTorch, ONNX models from different hubs (TF, ONNX, HF, PyTorch, etc,) for inference and we permanently develop it to cover TF, PyTorch, ONNX opsets. You can find all OV operations here: https://docs.openvino.ai/2024/documentation/openvino-ir-format/operation-sets/operation-specs.html

Let me know what amount of operations I should cover in this PR to get it merged. So we will have the base OpenVINO support here and continue ops coverage support in future PRs. For the base OpenVINO support, I propose to support operations required by BERT model without preprocessor from Keras Hub, for example, and demonstrate that it works. Example below. Will it be sufficient for merge?

import os
os.environ["KERAS_BACKEND"] = "openvino"

import keras_nlp

features = {
    "token_ids": np.ones(shape=(2, 12), dtype="int32"),
    "segment_ids": np.array([[0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0]] * 2),
    "padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]] * 2),
}

# Load a BERT model.
classifier = keras_nlp.models.BertClassifier.from_preset(
    "bert_base_en_uncased", 
    num_classes=2,
    preprocessor=None
)

classifier.predict(features)

In the description of this PR, I demonstrated very simple model inference. Please check.

[fchollet]

Let me know what amount of operations I should cover in this PR to get it merged. So we will have the base OpenVINO support here and continue ops coverage support in future PRs. For the base OpenVINO support, I propose to support operations required by BERT model without preprocessor from Keras Hub, for example, and demonstrate that it works. Example below. Will it be sufficient for merge?

We will have to run only a very similar subset of unit tests on CI for OpenVINO since we cannot skip every test that isn't passing (too many). So we would have to create basically an OpenVINO-specific integration test that checks precisely the end-to-end workflows that are intended to be supported.

Are you going to add support for CV workflows? I would recommend also having support for conv/pooling ops.

[rkazants]

We will have to run only a very similar subset of unit tests on CI for OpenVINO since we cannot skip every test that isn't passing (too many). So we would have to create basically an OpenVINO-specific integration test that checks precisely the end-to-end workflows that are intended to be supported.

Are you going to add support for CV workflows? I would recommend also having support for conv/pooling ops.

Yes, I will add separate integration tests for OpenVINO to cover currently supported ops. For this PR, let me enable MobileNetV3 from KerasCV that will require to support conv/pooling operations.

Summarizing it, my to-do list for this PR looks as follows:

1. Enable BertClassifier model from KerasNLP
2. Enable MobileNet model from KerasCV
3. Add integration tests for OpenVINO to cover supported ops

@fchollet, will it be fine and sufficient for this PR merge?

Thanks a lot,
Roman

[fchollet]

@fchollet, will it be fine and sufficient for this PR merge?

Yes, that sounds good to me!