Feature/openqxd #1414

chaoos · 2023-11-06T17:36:54Z

Adds interface openQxD <--> QUDA
Adds capability for C* boundaries in 1,2 or 3 spatial directions
Fixed CPU reordering
Adds openQCD gamma matrix basis (and basis transformations)
Adds order class for spinors
Adds order class for clover field (U(1) clover field is transfered from openQxD)
Adds order class for gauge field
Changed all the comments in quda.h to C89 style comments
Rebased on your current develop branch

The rank formula was updated to the openQxD layout: rank = coords[0]*dims[1]*dims[2]*dims[3] + coords[1]*dims[2]*dims[3] + coords[2]*dims[3] + coords[3]

The forloop was corrected (typo in i counter)

corrected to i<=3

trying to manually couple indexing correctly

Currently, the reordering is done inside OpenQxd

…eature/openqxd

Probably it will continue to work for spinor fields... To be confirmed..

In check3, only half the sites are loaded onto quda (odd ones on whole lattice), then in the save function, only the odd ones in the first half get reloaded!

… feature/openqxd

properties

subobject"

Sw term is zero, but QUDA should not complain then

…wiki/Coding-Conventions-and-Style#coding-style-and-clang-format

chaoos · 2024-02-27T11:03:26Z

Thank you for this PR, this is a great contribution to QUDA. I have done an initial visual review at this point, and left a bunch of comments. My only real concern of note is the demotion of the interface to being C89 instead of C99, but those concerns may be unwarranted.

Outside of those comments, somethings that will need to be done:
* [ ]  Apply clang-format (see [here](https://github.com/lattice/quda/wiki/Coding-Conventions-and-Style#coding-style-and-clang-format))

* [ ]  Create a wiki page of how to use OpenQ*D with QUDA, similar for example to what we have for [MILC}(https://github.com/lattice/quda/wiki/MILC-with-QUDA)
Is there any specific functionality that is added to QUDA that we should be adding to the internal tests?

Dear Kate,

I dealt with the two points:

see commit 568dd37 for the clang-formatting
The page for the wiki is written, but I cannot commit it, permission issue. I attach the patch here openqxd.zip

Best,
Roman

maddyscientist · 2024-02-28T05:01:25Z

Hi Roman,

I've added you to the lattice github group, so you should have permission to edit the wiki now (and push to branches in the lattice org).

Thx

maddyscientist · 2024-02-28T05:07:14Z

@Jenkins ok to test

interface)

chaoos · 2024-02-28T15:24:09Z

I was able to commit to the wiki, thanks for giving me access!
The CI/CD also seems to be fixed.

havogt · 2024-02-28T15:51:47Z

cscs-ci run

maddyscientist

I've had a review of the latest code. This is a great contribution, but a few things left outstanding to be resolved. I have left some comments, but beyond those:

The C89 changes are of concern, since we will be embracing a C99 interface, e.g., use of bool.
The new openQCD interface is rather opaque to me. Can this be better documented?
Thanks for the wiki additions. Is there a test input file or equivalent to allow us to run a test with this?

maddyscientist · 2024-03-22T21:46:59Z

include/dirac_quda.h

@@ -2606,6 +2606,8 @@ namespace quda {
          && (pc_type == QUDA_MATPC_EVEN_EVEN || pc_type == QUDA_MATPC_EVEN_EVEN_ASYMMETRIC))
        return true;

+      if (dirac_type == QUDA_WILSON_DIRAC || dirac_type == QUDA_CLOVER_DIRAC) return true;


This addition here looks bogus: the Wilson operator is not Hermitian. Is this addition intentional?

This is in the class DiracG5M (i.e. gamma5 times the Dirac operator), which is Hermitian. I had to change this, since the eigensolver didn't accept a non-Hermitian operator else. So this should be correct (only for gamma5*D and D*gamma5 though).

maddyscientist · 2024-03-22T21:49:50Z

include/instantiate_dslash.h

@@ -38,6 +38,20 @@ namespace quda
 #else
      errorQuda("QUDA_RECONSTRUCT=%d does not enable reconstruct-8/9", QUDA_RECONSTRUCT);
 #endif
+#ifdef BUILD_OPENQCD_INTERFACE


@kostrzewa noting that we could augment this for tmLQCD after this is merged to give recon 9/13 support, which I guess will help your perf?

chaoos · 2024-03-26T15:35:40Z

The C89 changes are of concern, since we will be embracing a C99 interface, e.g., use of bool.

Dear Kate,

The C99 thing is ok from our side. Should I just revert all changes I made in quda.h regarding this?
We are currently in the phase of documenting, so give me some days for this. Do you want more comments in the code (I think every function/method has a javadoc-decoration), or conceptual things in the wiki?
Regarding testing of openqxd functionality; we have some testing on the side of openqxd. If you want to test this in this repo, then you need to clone and compile the openqxd repo and run the tests from there. We are working on making this repo public, but there is still some politics going on: that's not in my power.

maddyscientist · 2024-04-12T18:51:53Z

@chaoos apologies for the slow reply at my end. GTC and other travel got in the way.....

If you can revert the C99 -> C89 changes that would be great. It would also narrow the scope of the PR which would also be helpful.

Regarding documentation, the basic rule is that all new functions should have doxygen markup next to their declarations, together with overall conceptual documentation in the wiki. If you can write up the documentation to include cloning the openq*d and compiling the code to work with QUDA that would be great. Understood that the repo isn't public yet, and that's fine for now. Does that sound ok to you?

not do any transfer of fields)

mathiaswagner and others added 30 commits September 26, 2022 12:39

initial structure (completely untested and still a lot of things WIP)

ed966db

compilation fixes and plaquette dummy

71c1897

Updated rank formula

efc24b8

The rank formula was updated to the openQxD layout: rank = coords[0]*dims[1]*dims[2]*dims[3] + coords[1]*dims[2]*dims[3] + coords[2]*dims[3] + coords[3]

Typo corrected

61b8a02

The forloop was corrected (typo in i counter)

Typo corrected

a40b032

corrected to i<=3

fix rankFromCoords to consider BLK order

0a69472

some fixes for openQCD testing

c29aa66

Still not working,

d3d4022

trying to manually couple indexing correctly

Now working, todo: add ipt directly in quda

9e1b7ad

Currently, the reordering is done inside OpenQxd

TODO: comments & planning

aec93e3

hardwire quda rankfromcoords

1f0e573

hardwire test 2

6271fac

Merge branch 'feature/openqxd' of github.com:fernandezdlg/quda into f…

ef1e5af

…eature/openqxd

ignore build folder

8550ed0

rankFromCoords for OpenQxD gauge fields working.

c9d0f6c

Probably it will continue to work for spinor fields... To be confirmed..

OpenQCD spinor interface from TIFR (need adjusting

301378b

commented out openqcd spinors sections for compile

240d614

spinors feature OK (still needs correct indexing)

6202ac4

openqcd dslash correction + load color spinor

cf3a04a

load-save in spinors+gauge + format

16b7df0

small correction on load fct

12f22f4

ordering done, lexicographical io w openqxd

d535512

added openqcd dirac order

f538e53

dslash quda := NO dslash operator applied

568d1cd

strange bad indexing behaviour (check3 log)

dc8c99f

small corrections + comments

ecc2177

In check3, only half the sites are loaded onto quda (odd ones on whole lattice), then in the save function, only the odd ones in the first half get reloaded!

cleanup and test oddeven

8b1830d

latest test

f738510

formatting

ad8f689

corrected over-include enum + cmmnts

189896e

chaoos requested review from a team as code owners December 15, 2023 17:48

Roman Gruber added 15 commits February 5, 2024 17:44

added packed formats

1b1a55c

Merge branch 'feature/openqxd' of https://github.com/chaoos/quda into…

c2c8ef6

… feature/openqxd

dynamic t_boundary

dd92262

enable RECONSTRUCT_8/9/12/13/NO for openqxd interface

20e59ba

fixed eigensolver, when cuda_prec_eigensolver is not set in the infile

d40d6e5

only transfer/recalc gauge/clover field according to openqxd's flag DB

6c7bb9d

added void pointer to QudaInvertParam struct for arbitrary additinal

005aed9

properties

changed solver interface to only deal with solver section IDs

ad1ab82

refactored interface

c1fec52

removed ptr in getHandle

d356fa0

removed ptr in getHandle (2)

a80dd55

fixed clang complaining "error: suggest braces around initialization of

466d702

subobject"

in openQCD sometimes we test with gauge field all set to unity, then the

c79cb4a

Sw term is zero, but QUDA should not complain then

remove trailing whitespaces

6edc863

applied clang-format as described in https://github.com/lattice/quda/…

568dd37

…wiki/Coding-Conventions-and-Style#coding-style-and-clang-format

Roman Gruber added 3 commits February 28, 2024 12:43

eigensolver iterface rewritten to use identifiers too (as inverter

3150761

interface)

fixed compiler error in CI/CD

0752a23

clang-format

ca54f7e

fixed eigensolver deallocation

9d9ace7

maddyscientist requested changes Mar 22, 2024

View reviewed changes

added openQCD_qudaDw_NoLoads (wrapper around Dirac operator that does

f63a507

not do any transfer of fields)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/openqxd #1414

Feature/openqxd #1414

chaoos commented Nov 6, 2023 •

edited

Loading

chaoos commented Feb 27, 2024

maddyscientist commented Feb 28, 2024

maddyscientist commented Feb 28, 2024

chaoos commented Feb 28, 2024

havogt commented Feb 28, 2024

maddyscientist left a comment

maddyscientist Mar 22, 2024

chaoos Mar 26, 2024 •

edited

Loading

maddyscientist Mar 22, 2024

chaoos commented Mar 26, 2024

maddyscientist commented Apr 12, 2024

Feature/openqxd #1414

Are you sure you want to change the base?

Feature/openqxd #1414

Conversation

chaoos commented Nov 6, 2023 • edited Loading

chaoos commented Feb 27, 2024

maddyscientist commented Feb 28, 2024

maddyscientist commented Feb 28, 2024

chaoos commented Feb 28, 2024

havogt commented Feb 28, 2024

maddyscientist left a comment

Choose a reason for hiding this comment

maddyscientist Mar 22, 2024

Choose a reason for hiding this comment

chaoos Mar 26, 2024 • edited Loading

Choose a reason for hiding this comment

maddyscientist Mar 22, 2024

Choose a reason for hiding this comment

chaoos commented Mar 26, 2024

maddyscientist commented Apr 12, 2024

chaoos commented Nov 6, 2023 •

edited

Loading

chaoos Mar 26, 2024 •

edited

Loading