Clarification on how to use the EPU command

[souravmat-git]

Hello,

After reviewing the SEACAS documentation, I found that I can join parallel meshes using the EPU (Exodus Parallel Unification) program. I installed and built SEACS on a Linux cluster. I was also able to run the tests, and all tests successfully passed.

However, I was curious about how to call the EPU command to combine parallel meshes. For instance,
epu -auto `/where/my/distributed/mesh/is/ filename.e.<#of_processors>.<0000>

If I run this, then I get an error that the EPU command is not found.

I would really appreciate it if someone could clarify its usage.

Sourav
[editted to add URL to SEACAS documentation]

[gsjaardema]

You need to somehow add the path to the epu command to your environment, or you can add the explicit path to the beginning of the epu command. For example, if the epu binary is at /usr/local/seacas/bin/epu, then you could do:

export PATH=$PATH:/usr/local/seacas/bin
epu -auto /where/my/distributed/mesh/is/ filename.e

or

/usr/local/seacas/bin/epu -auto `/where/my/distributed/mesh/is/ filename.e

Note that with recent versions of epu, you don't need to add the <#of_processors>.<0000> as long as you only have one set of decomposed files in the directory.

Hope that is clear and helps...

[souravmat-git]

Thanks! It was easy and fast.
Specifically, merging a 3D exodus file size of 3.4 GB took around 331.90 s.

Is it possible to run this command in parallel to handle even more large file sizes?

Thanks,
Sourav

[gsjaardema]

If the seacas installation was built with parallel-enabled, then there is a pepu executable which can be run in parallel. If you run it on N ranks, then it will run N epu's simultaneously each working on <#files>/N files. It will then run a single copy that joins the N files into a single file.

There is also a epup script which does something similar, but doesn't use MPI and instead uses something called gnu parallel which runs multiple copies of applications depending on the capabilities of the system (multi-core, multi-cpu). It then runs one last time and joins the individual files into a single file.

Another option is that the io_shell program built with SEACAS can, if built in parallel, read N files and output 1 file. You could either run it on all of the original files, or you could do either pepu or epup and get down to N files and then run io_shell on N ranks to go N->1.

Hope that helps and isn't too confusing.

[souravmat-git]

Thanks for sharing this!

I could use both pepu and epup commands and control the number of processors by setting the subcycle variable. They reduce the execution time compared to just using epu.

However, the usage of the io_shell program is unclear to me since I could not find any documentation.
I would be glad if you could point me to some documentation regarding this.

Sourav

[gsjaardema]

However, the usage of the io_shell program is unclear to me since I could not find any documentation. I would be glad if you could point me to some documentation regarding this.

io_shell started as a program for testing the IOSS library and would read a file and then write the exact same file. This has evolved a little and now io_shell has some additional functionality. However, for the joining of parallel decomposed file-per-rank files, the basic functionality is to just read a file and write the same file. However, in parallel we make use of the auto-join capability in which we read a file-per-rank and then output a single file using parallel io. For this case, the syntax is simply:

mpiexec -np {N} {path_to_io_shell}/io_shell --compose mpiio {input_file_name.e} {composed_file_name.e}

The mpiio argument to --compose is not needed, but due to a bug in the optional argument processing, it is needed unless there is another argument following --compose. You can see other options using io_shell --help

[souravmat-git]

Thank you for answering the questions!

Sourav

[gsjaardema]

And just for full disclosure....

Most/many systems have a limit on the number of files that an application can have open simultaneously--typically about 1024. If you are running epu in serial mode on a set of files larger than that, then epu will automatically go into a subcyclemode where it will join a subset S of the files. such that S < 1024 so it can keep the files open throughout the run. It will when run on the next subset of size S until it has accessed all the files. It will then run again and join the N/S files into a single file.