Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Addition of 4 pytorch onnx-export models. #278

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

Addition of 4 pytorch onnx-export models. #278

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
a45c418
try adding the 4 pytorch models with onnx export
saienduri Jul 3, 2024
9b9f7b1
comment out jobs that we don't care about for validation purposes now
saienduri Jul 3, 2024
10d44e7
try with added flag
saienduri Jul 3, 2024
a5c0f77
update mobilebert shape output
saienduri Jul 3, 2024
a088614
remove all expected fails to see if works as intended
saienduri Jul 3, 2024
a0d2d52
fix all shapes
saienduri Jul 3, 2024
7501f7b
fix remote download json files
saienduri Jul 3, 2024
8411a4a
update tolerances and xfails
saienduri Jul 3, 2024
bb90c56
xfails
saienduri Jul 3, 2024
3d9bd2c
update xfail
saienduri Jul 3, 2024
34aff57
uncomment out tests now that verified
saienduri Jul 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 8 additions & 9 deletions iree_tests/configs/models_cpu_llvm_task.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,19 @@
"config_name": "cpu_llvm_task",
"iree_compile_flags" : [
"--iree-hal-target-backends=llvm-cpu",
"--iree-llvmcpu-target-cpu-features=host"
"--iree-llvmcpu-target-cpu-features=host",
"--iree-input-demote-i64-to-i32"
],
Comment on lines 3 to 7
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also added the --iree-input-demote-i64-to-i32 flag to the cpu models config file so that we are in line with what we run in e2eshark for any cpu code generation

I'm not sure I agree with this. I'm worried about the number of flags we have developers using, and I'd like to have more visibility into which models require which flags and why. One way we can do that is by having users of the test suite test with default flags.

In this case, the --iree-input- flags are sort of special in that they are generic across backends so I do see a case for using some of them... maybe in a _compatibility_flags test suite config, then we could diff the XFAILs between that and the _default_flags config.

"iree_run_module_flags": [
"--device=local-task"
],
"skip_compile_tests": [
"pytorch/models/sdxl-scheduled-unet-3-tank",
"pytorch/models/sdxl-vae-decode-tank",
"pytorch/models/sdxl-prompt-encoder-tank"
],
"skip_compile_tests": [],
"skip_run_tests": [],
"expected_compile_failures": [
"pytorch/models/opt-125M", // TODO(#17344): need to regenerate .mlirbc
"pytorch/models/resnet50", // TODO(#17344): need to regenerate .mlirbc
"pytorch/models/direct/opt-125M", // TODO(#17344): need to regenerate .mlirbc
"pytorch/models/direct/resnet50", // TODO(#17344): need to regenerate .mlirbc
],
"expected_run_failures": []
"expected_run_failures": [
"pytorch/models/onnx-export/mobilebert-uncased",
]
Comment on lines +17 to +19
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This commit adds testing for 4 models that we have noticed to be unstable and want to keep passing through CI

Can we add models that we think are stable too? :) I'd like to prefetch all models from e2eshark and get them running here if possible, rather than cherrypick a few after we notice issues.

}
20 changes: 9 additions & 11 deletions iree_tests/configs/models_gpu_rocm_gfx90a.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,25 +2,23 @@
"config_name": "gpu_rocm",
"iree_compile_flags" : [
"--iree-hal-target-backends=rocm",
"--iree-rocm-target-chip=gfx90a",
"--iree-opt-const-eval=false",
"--iree-codegen-transform-dialect-library=${IREE_TEST_PATH_EXTENSION}/attention_and_matmul_spec.mlir"
"--iree-rocm-target-chip=gfx90a"
],
"iree_run_module_flags": [
"--device=hip"
],
"skip_compile_tests": [
"pytorch/models/sdxl-scheduled-unet-3-tank",
"pytorch/models/sdxl-vae-decode-tank",
"pytorch/models/sdxl-prompt-encoder-tank"
],
"skip_compile_tests": [],
"skip_run_tests": [],
"expected_compile_failures": [
"pytorch/models/opt-125M", // TODO(#17344): need to regenerate .mlirbc
"pytorch/models/resnet50",
"pytorch/models/direct/opt-125M", // TODO(#17344): need to regenerate .mlirbc
"pytorch/models/direct/resnet50",
// error: 'builtin.module' op failed to run transform dialect passes
// (might need to drop the iree-codegen-transform-dialect-library flag)
Comment on lines 15 to 16
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can drop this comment since the attention_and_matmul_spec is removed.
I also saw error: a handle passed as operand #0 and consumed by this operation points to a payload entity more than once and updated the comment in my draft PR: #277

How about we just leave the reason comments off? Also on my PR there, if we run with -rA (instead of -rpfE, see https://docs.pytest.org/en/stable/how-to/output.html), we'll get full error output for even XFAIL'd tests in CI logs, so having an possibly outdated comment in the source isn't as useful.

"sharktank/llama/open-llama-3b-v2-f16"
"sharktank/llama/open-llama-3b-v2-f16",
"pytorch/models/onnx-export/mit-b0",
"pytorch/models/onnx-export/mobilebert-uncased",
"pytorch/models/onnx-export/t5-base",
"pytorch/models/onnx-export/t5-large"
],
"expected_run_failures": []
}
0 ...s/pytorch/models/opt-125M/opt-125M.mlirbc → ...ch/models/direct/opt-125M/opt-125M.mlirbc
View file
Open in desktop
File renamed without changes.
0 ...dels/opt-125M/real_weights_data_flags.txt → ...rect/opt-125M/real_weights_data_flags.txt
View file
Open in desktop
File renamed without changes.
0 ...orch/models/opt-125M/splat_data_flags.txt → ...dels/direct/opt-125M/splat_data_flags.txt
View file
Open in desktop
File renamed without changes.
0 ...tests/pytorch/models/opt-125M/splats.irpa → ...ytorch/models/direct/opt-125M/splats.irpa
View file
Open in desktop
File renamed without changes.
0 ...s/pytorch/models/opt-125M/test_cases.json → ...ch/models/direct/opt-125M/test_cases.json
View file
Open in desktop
File renamed without changes.
0 ...dels/resnet50/real_weights_data_flags.txt → ...rect/resnet50/real_weights_data_flags.txt
View file
Open in desktop
File renamed without changes.
0 ...s/pytorch/models/resnet50/resnet50.mlirbc → ...ch/models/direct/resnet50/resnet50.mlirbc
View file
Open in desktop
File renamed without changes.
0 ...orch/models/resnet50/splat_data_flags.txt → ...dels/direct/resnet50/splat_data_flags.txt
View file
Open in desktop
File renamed without changes.
0 ...tests/pytorch/models/resnet50/splats.irpa → ...ytorch/models/direct/resnet50/splats.irpa
View file
Open in desktop
File renamed without changes.
0 ...s/pytorch/models/resnet50/test_cases.json → ...ch/models/direct/resnet50/test_cases.json
View file
Open in desktop
File renamed without changes.
3 changes: 3 additions & 0 deletions iree_tests/pytorch/models/onnx-export/mit-b0/model.mlirbc
View file
Open in desktop
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These models are run through the onnx mode and pytorch framework where the pytorch model is exported to onnx, imported to torch IR using the onnx importer, and then fed to IREE to lower to linalg and run the module.

That seems rather roundabout... I guess ONNX there is used as a serialization format?

IMO if we're adding tests for these models we should add both the direct and onnx-export versions.

Git LFS file not shown
3 changes: 3 additions & 0 deletions iree_tests/pytorch/models/onnx-export/mit-b0/real_weights_data_flags.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
--input=1x3x512x512xf32=@inference_input.0.bin
--expected_output=1x1000xf32=@inference_output.0.bin
--expected_f32_threshold=0.01f
13 changes: 13 additions & 0 deletions iree_tests/pytorch/models/onnx-export/mit-b0/test_cases.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"file_format": "test_cases_v0",
"test_cases": [
{
"name": "real_weights",
"runtime_flagfile": "real_weights_data_flags.txt",
"remote_files": [
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/mit-b0/inference_input.0.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/mit-b0/inference_output.0.bin"
]
}
]
}
3 changes: 3 additions & 0 deletions iree_tests/pytorch/models/onnx-export/mobilebert-uncased/real_weights_data_flags.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
--input=1x128xi64=@inference_input.0.bin
--expected_output=1x2xf32=@inference_output.0.bin
--expected_f32_threshold=0.01f
14 changes: 14 additions & 0 deletions iree_tests/pytorch/models/onnx-export/mobilebert-uncased/test_cases.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"file_format": "test_cases_v0",
"test_cases": [
{
"name": "real_weights",
"runtime_flagfile": "real_weights_data_flags.txt",
"remote_files": [
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/mobilebert-uncased/inference_input.0.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/mobilebert-uncased/inference_output.0.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/mobilebert-uncased/model.mlirbc"
]
}
]
}
54 changes: 54 additions & 0 deletions iree_tests/pytorch/models/onnx-export/t5-base/real_weights_data_flags.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
--input=1x512xi64=@inference_input.0.bin
--input=1x512xf32=@inference_input.1.bin
--input=1x512xi64=@inference_input.2.bin
--expected_output=1x512x768xf32=@inference_output.0.bin
--expected_output=1x12x512x64xf32=@inference_output.1.bin
--expected_output=1x12x512x64xf32=@inference_output.2.bin
--expected_output=1x12x512x64xf32=@inference_output.3.bin
--expected_output=1x12x512x64xf32=@inference_output.4.bin
--expected_output=1x12x512x64xf32=@inference_output.5.bin
--expected_output=1x12x512x64xf32=@inference_output.6.bin
--expected_output=1x12x512x64xf32=@inference_output.7.bin
--expected_output=1x12x512x64xf32=@inference_output.8.bin
--expected_output=1x12x512x64xf32=@inference_output.9.bin
--expected_output=1x12x512x64xf32=@inference_output.10.bin
--expected_output=1x12x512x64xf32=@inference_output.11.bin
--expected_output=1x12x512x64xf32=@inference_output.12.bin
--expected_output=1x12x512x64xf32=@inference_output.13.bin
--expected_output=1x12x512x64xf32=@inference_output.14.bin
--expected_output=1x12x512x64xf32=@inference_output.15.bin
--expected_output=1x12x512x64xf32=@inference_output.16.bin
--expected_output=1x12x512x64xf32=@inference_output.17.bin
--expected_output=1x12x512x64xf32=@inference_output.18.bin
--expected_output=1x12x512x64xf32=@inference_output.19.bin
--expected_output=1x12x512x64xf32=@inference_output.20.bin
--expected_output=1x12x512x64xf32=@inference_output.21.bin
--expected_output=1x12x512x64xf32=@inference_output.22.bin
--expected_output=1x12x512x64xf32=@inference_output.23.bin
--expected_output=1x12x512x64xf32=@inference_output.24.bin
--expected_output=1x12x512x64xf32=@inference_output.25.bin
--expected_output=1x12x512x64xf32=@inference_output.26.bin
--expected_output=1x12x512x64xf32=@inference_output.27.bin
--expected_output=1x12x512x64xf32=@inference_output.28.bin
--expected_output=1x12x512x64xf32=@inference_output.29.bin
--expected_output=1x12x512x64xf32=@inference_output.30.bin
--expected_output=1x12x512x64xf32=@inference_output.31.bin
--expected_output=1x12x512x64xf32=@inference_output.32.bin
--expected_output=1x12x512x64xf32=@inference_output.33.bin
--expected_output=1x12x512x64xf32=@inference_output.34.bin
--expected_output=1x12x512x64xf32=@inference_output.35.bin
--expected_output=1x12x512x64xf32=@inference_output.36.bin
--expected_output=1x12x512x64xf32=@inference_output.37.bin
--expected_output=1x12x512x64xf32=@inference_output.38.bin
--expected_output=1x12x512x64xf32=@inference_output.39.bin
--expected_output=1x12x512x64xf32=@inference_output.40.bin
--expected_output=1x12x512x64xf32=@inference_output.41.bin
--expected_output=1x12x512x64xf32=@inference_output.42.bin
--expected_output=1x12x512x64xf32=@inference_output.43.bin
--expected_output=1x12x512x64xf32=@inference_output.44.bin
--expected_output=1x12x512x64xf32=@inference_output.45.bin
--expected_output=1x12x512x64xf32=@inference_output.46.bin
--expected_output=1x12x512x64xf32=@inference_output.47.bin
--expected_output=1x12x512x64xf32=@inference_output.48.bin
--expected_output=1x512x768xf32=@inference_output.49.bin
--expected_f32_threshold=0.01f
65 changes: 65 additions & 0 deletions iree_tests/pytorch/models/onnx-export/t5-base/test_cases.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
{
"file_format": "test_cases_v0",
"test_cases": [
{
"name": "real_weights",
"runtime_flagfile": "real_weights_data_flags.txt",
"remote_files": [
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_input.0.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_input.1.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_input.2.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.0.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.1.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.2.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.3.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.4.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.5.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.6.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.7.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.8.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.9.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.10.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.11.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.12.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.13.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.14.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.15.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.16.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.17.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.18.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.19.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.20.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.21.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.22.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.23.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.24.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.25.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.26.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.27.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.28.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.29.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.30.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.31.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.32.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.33.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.34.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.35.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.36.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.37.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.38.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.39.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.40.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.41.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.42.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.43.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.44.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.45.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.46.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.47.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.48.bin",
"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/inference_output.49.bin",
Comment on lines +57 to +60
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

o_o that's a lot of outputs. Fine for now, but let's file an issue to use archive files (.zip or .tar.gz). If we use something like Hugging Face repositories, we could also lean on their download API to fetch an entire repo

"https://sharkpublic.blob.core.windows.net/sharkpublic/shark-test-suite/iree-tests/pytorch/models/onnx-export/t5-base/model.mlirbc"
]
}
]
}
Loading