Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incompatible version of Polygaist is installed if build-with-polygeist.sh is used #122

Open
geexie opened this issue Jan 12, 2022 · 18 comments
Open

Incompatible version of Polygaist is installed if build-with-polygeist.sh is used #122

geexie opened this issue Jan 12, 2022 · 18 comments

Comments

@geexie
Copy link

geexie commented Jan 12, 2022

Does this script is still valid? It checks out version of Polygeist from polygeist-version.txt which seems to be incompatible and miss some required headers so I'm getting the error

[10/33] Building CXX object tools/mlir/tools/polymer/lib/Target/CMakeFiles/obj.PolymerTargetOpenScop.dir/OpenScop/ConvertFromOpenScop.cc.o
FAILED: tools/mlir/tools/polymer/lib/Target/CMakeFiles/obj.PolymerTargetOpenScop.dir/OpenScop/ConvertFromOpenScop.cc.o 
/usr/bin/c++  -DGTEST_HAS_RTTI=0 -DMLIR_CUDA_CONVERSIONS_ENABLED=0 -DMLIR_INCLUDE_TESTS -DMLIR_ROCM_CONVERSIONS_ENABLED=0 -D_DEBUG -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/mlir/tools/polymer/lib/Target -I/home/geek/py/base/Polygeist-polymer/mlir/tools/polymer/lib/Target -Iinclude -I/home/geek/py/base/Polygeist-polymer/llvm/include -I/home/geek/py/base/Polygeist-polymer/mlir/include -Itools/mlir/include -Itools/mlir/tools/polymer/pluto/include -I/home/geek/py/base/Polygeist-polymer/mlir/tools/polymer/include -Itools/mlir/tools/polymer/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wno-comment -fdiagnostics-color -ffunction-sections -fdata-sections -O3     -fno-exceptions -fno-rtti -UNDEBUG -std=c++14 -MD -MT tools/mlir/tools/polymer/lib/Target/CMakeFiles/obj.PolymerTargetOpenScop.dir/OpenScop/ConvertFromOpenScop.cc.o -MF tools/mlir/tools/polymer/lib/Target/CMakeFiles/obj.PolymerTargetOpenScop.dir/OpenScop/ConvertFromOpenScop.cc.o.d -o tools/mlir/tools/polymer/lib/Target/CMakeFiles/obj.PolymerTargetOpenScop.dir/OpenScop/ConvertFromOpenScop.cc.o -c /home/geek/py/base/Polygeist-polymer/mlir/tools/polymer/lib/Target/OpenScop/ConvertFromOpenScop.cc
/home/geek/py/base/Polygeist-polymer/mlir/tools/polymer/lib/Target/OpenScop/ConvertFromOpenScop.cc:31:10: fatal error: mlir/Dialect/Arithmetic/IR/Arithmetic.h: No such file or directory
 #include "mlir/Dialect/Arithmetic/IR/Arithmetic.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
[11/33] Building CXX object tools/mlir/tools/polymer/lib/Transforms/CMakeFiles/obj.PolymerTransforms.dir/Reg2Mem.cc.o
FAILED: tools/mlir/tools/polymer/lib/Transforms/CMakeFiles/obj.PolymerTransforms.dir/Reg2Mem.cc.o 
/usr/bin/c++  -DGTEST_HAS_RTTI=0 -DMLIR_CUDA_CONVERSIONS_ENABLED=0 -DMLIR_INCLUDE_TESTS -DMLIR_ROCM_CONVERSIONS_ENABLED=0 -D_DEBUG -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/mlir/tools/polymer/lib/Transforms -I/home/geek/py/base/Polygeist-polymer/mlir/tools/polymer/lib/Transforms -Iinclude -I/home/geek/py/base/Polygeist-polymer/llvm/include -I/home/geek/py/base/Polygeist-polymer/mlir/include -Itools/mlir/include -Itools/mlir/tools/polymer/pluto/include -I/home/geek/py/base/Polygeist-polymer/mlir/tools/polymer/include -Itools/mlir/tools/polymer/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wno-comment -fdiagnostics-color -ffunction-sections -fdata-sections -O3     -fno-exceptions -fno-rtti -UNDEBUG -std=c++14 -MD -MT tools/mlir/tools/polymer/lib/Transforms/CMakeFiles/obj.PolymerTransforms.dir/Reg2Mem.cc.o -MF tools/mlir/tools/polymer/lib/Transforms/CMakeFiles/obj.PolymerTransforms.dir/Reg2Mem.cc.o.d -o tools/mlir/tools/polymer/lib/Transforms/CMakeFiles/obj.PolymerTransforms.dir/Reg2Mem.cc.o -c /home/geek/py/base/Polygeist-polymer/mlir/tools/polymer/lib/Transforms/Reg2Mem.cc
/home/geek/py/base/Polygeist-polymer/mlir/tools/polymer/lib/Transforms/Reg2Mem.cc:15:10: fatal error: mlir/Dialect/Arithmetic/IR/Arithmetic.h: No such file or directory
 #include "mlir/Dialect/Arithmetic/IR/Arithmetic.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

Which version of Polygeist should I compile against of?
``
@wsmoses
Copy link

wsmoses commented Jan 12, 2022

CI appears to use daeb952678b1d46c867c94ed2f20d78e1a3d5ec5

That said @kumasento we should probably rebase this onto polygeist/mlir main.

@kumasento
Copy link
Owner

Thanks @geexie and @wsmoses . Sorry for the inconvenience.

I've populated the commit ID from what the CI uses. Hope this will help.

@geexie
Copy link
Author

geexie commented Jan 14, 2022 •
edited
Loading

@wsmoses, @kumasento thanks for the link! do you actually build Polygeist to get Polymer working? On you CI you just build LLVM and then Polymer. I'm a bit confused about relation ship of those 2 projects and repos (Polygeist and Polymer)

@kumasento
Copy link
Owner

@wsmoses, @kumasento thanks for the link! do you actually build Polygeist to get Polymer working? On you CI you just build LLVM and then Polymer. I'm a bit confused about relation ship of those 2 projects and repos (Polygeist and Polymer)

Thanks for the Q. It depends on what your input is. If it is MLIR, then just building Polymer + LLVM is fine. But if you need to pass in C/C++ code, you need Polygeist to translate them into MLIR first.

Hope this helps.

@wsmoses
Copy link

wsmoses commented Jan 14, 2022

Generalizing the answer from Ruizhe a bit further:

Polygeist is the name of the overall project as well as the repo for the frontend and raising pipelines (Eg takes C/C++, and also converts cfg to loops, loops to affine, etc).

This repo is a component of the overall framework that does the bidirectional conversion to/from openscop.

@geexie
Copy link
Author

geexie commented Jan 18, 2022

@wsmoses, thanks for your comment. This means that if I want just go from c++ to affine the Polygeist is perfect for this task and if I want a full pipeline with conversion and polyhedral tuning let's say run some c++ workload I need to use Polygeist to generate MLIR and than pass this MLIR to Polymer in 2 separate commands

@wsmoses
Copy link

wsmoses commented Jan 18, 2022

Correct!

@paldebjit
Copy link

@kumasento @wsmoses I was trying to follow the steps in CI. I got the following error:

[dp638@zhang-26 build]$ cat /work/zhang-x1/users/dp638/Tools/MLIR_Tools/polymer/build/CMakeFiles/CMakeError.log
Performing C++ SOURCE FILE Test CXX_SUPPORTS_CUSTOM_LINKER failed with the following output:
Change Dir: /work/zhang-x1/users/dp638/Tools/MLIR_Tools/polymer/build/CMakeFiles/CMakeTmp

Run Build Command(s):/work/zhang-x1/users/dp638/Tools/Ninja/ninja cmTC_30574 && [1/2] Building CXX object CMakeFiles/cmTC_30574.dir/src.cxx.o
clang-14: warning: argument unused during compilation: '-fuse-ld=lld' [-Wunused-command-line-argument]
[2/2] Linking CXX executable cmTC_30574
FAILED: cmTC_30574 
: && /work/zhang-x1/users/dp638/Tools/MLIR_Tools/Polygeist/llvm-project/install/bin/clang++ -fuse-ld=lld  CMakeFiles/cmTC_30574.dir/src.cxx.o -o 
cmTC_30574   && :
clang-14: error: invalid linker name in argument '-fuse-ld=lld'
ninja: build stopped: subcommand failed.


Source file was:
int main() { return 0; }

I have built llvm-project inside Polygeist at the commit daeb952678b1d46c867c94ed2f20d78e1a3d5ec5. Any suggestion what could be causing this? My gcc version is the following: gcc version 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC).

@kumasento
Copy link
Owner

kumasento commented Jan 27, 2022 via email

@paldebjit
Copy link

@kumasento Here is a screenshot of my setup. It matches the CI setup (checkout-wise). However, polymer compilation still fails. I would appreciate your suggestions. Thanks once again.
Screenshot from 2022-01-27 17-06-24

@kumasento
Copy link
Owner

kumasento commented Jan 27, 2022 via email

@paldebjit
Copy link

I am trying Polygeist daeb952678b1d46c867c94ed2f20d78e1a3d5ec5 and also trying to checkout the LLVM that is linked to that checkout of Polygeist. I will post my findings soon.

@kumasento
Copy link
Owner

#125 should do the trick

@wsmoses
Copy link

wsmoses commented Jan 28, 2022

The lld issue is weird and shouldn't occur regardless. That in particular feels like some messed up cmake cache/config? In any case once #125 lands try a clean build

@paldebjit
Copy link

paldebjit commented Jan 29, 2022 •
edited
Loading

@kumasento I was able to build polymer-opt and polymer-translate using the daeb952678b1d46c867c94ed2f20d78e1a3d5ec5 commit. I will try #125 later as I guess this build will meet my need. I have one more question for you.

The polymer-opt has a pass --extract-scop-stmt. In my understanding that is needed to transform the optimized Pluto OpenSCOP to a MLIR function format. However, what I want is to extract the OpenSCOP for a given MLIR code in (Affine Dialect for the time being) which I guess is done by polymer-translate with --export-scop option. However, when I use polymer-translate --export-scop matmul.mlir, I get an error

polymer-translate --export-scop matmul.mlir 
matmul.mlir:2:8: error: custom op 'memref.alloc' is unknown
  %A = memref.alloc() : memref<64x64xf32>

Maybe because matmul is not entirely in affine dialect? I would appreciate your suggestions in this regard.

The --extract-scop-stmt with --pluto-opt does spit out an OpenSCOP in the terminal but I guess that is post pluto optimization.

@kumasento
Copy link
Owner

Hi @paldebjit, I'm not quite sure about your question: are you trying to dump MLIR from Pluto-optimised (or original?) OpenScop, or generate OpenScop from an MLIR program?

In either case, polymer-translate should be the right tool for you.

  • --import-scop translates OpenScop to MLIR
  • --export-scop translates MLIR to OpenScop

However, polymer-translate hasn't been maintained for a while, and it could possibly be obsolete.

The error you saw is due to that polymer-translate is not updated for a while and the memref dialect is not registered by it yet. It should be an easy fix, and I would do that soon this weekend.

But if you're in a hurry, you could try printing it out to the terminal. The whole pluto transformation is in this function:

polymer/lib/Transforms/PlutoTransform.cc

Lines 72 to 133 in b2f5469

static mlir::FuncOp plutoTransform(mlir::FuncOp f, OpBuilder &rewriter,
std::string dumpClastAfterPluto,
bool parallelize = false, bool debug = false,
int cloogf = -1, int cloogl = -1,
bool diamondTiling = false) {
LLVM_DEBUG(dbgs() << "Pluto transforming: \n");
LLVM_DEBUG(f.dump());
PlutoContext *context = pluto_context_alloc();
OslSymbolTable srcTable, dstTable;
std::unique_ptr<OslScop> scop = createOpenScopFromFuncOp(f, srcTable);
if (!scop)
return nullptr;
if (scop->getNumStatements() == 0)
return nullptr;
// Should use isldep, candl cannot work well for this case.
context->options->silent = !debug;
context->options->moredebug = debug;
context->options->debug = debug;
context->options->isldep = 1;
context->options->readscop = 1;
context->options->identity = 0;
context->options->parallel = parallelize;
context->options->unrolljam = 0;
context->options->prevector = 0;
context->options->diamondtile = diamondTiling;
if (cloogf != -1)
context->options->cloogf = cloogf;
if (cloogl != -1)
context->options->cloogl = cloogl;
PlutoProg *prog = osl_scop_to_pluto_prog(scop->get(), context);
pluto_schedule_prog(prog);
pluto_populate_scop(scop->get(), prog, context);
if (debug) { // Otherwise things dumped afterwards will mess up.
fflush(stderr);
fflush(stdout);
}
osl_scop_print(stderr, scop->get());
const char *dumpClastAfterPlutoStr = nullptr;
if (!dumpClastAfterPluto.empty())
dumpClastAfterPlutoStr = dumpClastAfterPluto.c_str();
mlir::ModuleOp m = dyn_cast<mlir::ModuleOp>(f->getParentOp());
SmallVector<DictionaryAttr> argAttrs;
f.getAllArgAttrs(argAttrs);
mlir::FuncOp g = cast<mlir::FuncOp>(createFuncOpFromOpenScop(
std::move(scop), m, dstTable, rewriter.getContext(), prog,
dumpClastAfterPlutoStr));
g.setAllArgAttrs(argAttrs);
pluto_context_free(context);
return g;
}

@paldebjit
Copy link

Hi @kumasento thanks. I was able to compile. The issue currently is on my side, my code base depends on LLVM-13 which does not have Arithmetic dialect and the mlir::FlatAffineValueConstraints whereas many of your code fragments do use Arith dialect and the that MLIR class. I am trying to backtrack and see which commit of your code I can reliably use along with my LLVM 13 based codebase. Thanks once again for your kind help. I appreciate it very mcuh.

@kumasento
Copy link
Owner

TBH I don't know if that is viable: changes in Polymer are not just version related, they may improve functionality as well.

The latest commit that doesn't use FlatAffineValueConstraint is 9a59b6b3ace91e4914f2532930a205678b1799fe.

I'm curious if there is a way to make a codebase compatible with multiple MLIR versions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@wsmoses @geexie @kumasento @paldebjit