CMSSW 15 won't support microarch-v2

[mmascher]

Impact of the new feature
It was decided at the O&C week that starting from CMSSW 15 we won't build CMSSW for microarch v2. v1 has already been excluded and is not supported anymore. We need to make sure that jobs that will have a requirement for microarch x86_64-v[34] when they need to run on CMSSW15.

Additional context
Microarch can be found on the slots as:

[mmascher@vocms4100 ~]$ condor_status -af Microarch | sort | uniq -c
     15 x86_64-v1
  12810 x86_64-v2
 113836 x86_64-v3
  38921 x86_64-v4

[mmascher]

Adding an email from antonio about this:

Hi Phat,

Thanks, we discussed this in the SI meeting today. Marco will open an issue, where we have to discuss basically 2 items:

• how (and who) should be doing the mapping between CMSSW versions and desired micro arch? In my opinion, this should happen in WM, which translates the request constraints into actual resource requirements.

• where should we implement the matchmaking constraint for the micro arch, as general condition on the pilots, or rather as additional requirements for the jobs? We discussed it could be done in WM, if this is exclusive to production jobs, or rather insert it as a general condition in the slot matching requirements, if we expect this to somehow also affect analysis jobs in the future.

Cheers,
Antonio

[srimanob]

Thanks @mmascher

More discussion is ongoing as sites show concern if their people can't analyze Run 3 data with newer CMSSW. The current proposal is at the end of the comment. (2) is on WM+SI part

==========

(1) Building CMSW with Dual Version Compatibility (v2 + v3):

• We continue to build CMSW with both v2 and v3, allowing cmssw/scram to select the version that best matches the available CPU architecture.
• This approach enables analyzers using resources that only support v2 to still access and analyze data with the latest CMSSW versions.
• All external dependencies are built with v2, maintaining compatibility for systems limited to this architecture. The situation will be the same as HLT at this moment.

(2) Central production:

• By implementing a matching condition, jobs submitted with CMSSW_15_0_0 (and later) are directed to sites that support v3-only CPUs.
• Sites restricted to v2 CPUs will continue to receive jobs for Run-2 and Run-3 campaigns (2022, 2023, 2024) as long as we maintain the current campaign setup. This ensures that all sites remain actively engaged in ongoing data processing without requiring immediate hardware upgrades.

(3) Validation:

• Starting with CMSSW_15_0_0_pre1, we switch from v2 to v3 for validation and maintain validated v3 versions moving forward.
• The High-Level Trigger (HLT) will benefit from this by not needing to perform separate validation for v3, simplifying the validation process and reducing redundancy.

[smuzaffar]

Hello, as @srimanob mentioned, we discussed this in detail during yesterday's core sw meeting and, as @srimanob mentioned, CMSSW_15_0_X will support dual micro-arch for x86_64 with x86-64-v3 being the default micro-architecture.

How can we make sure that Central Production/RelVals use x86-64-v3 compatible resources for 15.0.X and above release cycle? In future, SCRAM can set an environment variable to expose default micro-arch of a release so that Central Production/RelVals can use that in the job requirement.

[belforte]

@mmascher (there's no hook for Antonio here). Please link to the discussion issue in SI mentioned in the mail from Antonio

Of course support is needed for analysis jobs as well.

Now we multiple threads on this same topic (this, the one in CRABClient mentioned above, the SI one, discussion in CMSSW https://indico.cern.ch/event/1482012/ ). I'd rather like a single document where people comment and which quickly converge to the specification for the solution.
Can SW-Core do that ? @smuzaffar

[smuzaffar]

@belforte , why not just use this issue as a single doc then :-)

About "mapping between CMSSW versions and desired micro arch", I think it will be easier if SCRAM set this via an environment variable. This way you do not need to hardcode this mapping in WM. e.g in CMSSW_15_0_X scram can set CMSSW_DEFAULT_MICRO_ARCH=x86-64-v3 and Relvals/Central production use it to request resources. For Central production/Relvals we prefer to use default microarch.

For analysis jobs, where some users might want to use v2 resources, SCRAM can set CMSSW_MICRO_ARCHS env to comma separated list of micro-archs and crab can make use of that to request the correct resources.

In case these env variable are not set ( e.g for CMSSW_14_2 and earlier) then just do what we are doing now.

FYI @makortel

[belforte]

Assuming that CMSSW_MICRO_ARCHS is set after cmsenv, adding a classAd MY.DESIRED_MICROARCH="x86-64-v2,x86-64-v3,..." is easy.
And I presume that's what SI will give us. I do not want to make site lists based on their hardware. Also some sites may have a mix, right ? @mmascher do you agree ?

I worry about the overall picture. Of course we should try to make it fully transparent for as many users as possible. Hopefully only those who develop for CMSSW_15 on v2 machines will have to do something. So I have to worry about what other changes to do in CRAB besides the above.

my questions are more like

• does this affect where DM (and CRAB tape recall) place datasets ?
• are those v2 machines grouped at specific sites or scattered all over ?
• are there sites which are mostly/only v2 ?
• what about jobs which run at sites other then where the data is ?
• will it matter which release was a dataset produced with ? Should we at least make sure that all data produced with CMSSW_15+ land at sites with e.g. >50% v3 machines ?

[makortel]

One question to WMCore (say @amaltaro @anpicci) came up in the core software meeting yesterday: does WMCore (production, not CRAB) run cmsRun from a release area, or does it create a developer area (cmsrel CMSSW_X_Y_Z / scram project CMSSW_X_Y_Z) first?

[makortel]

Another question to PdmV (@AdrianoDee @DickyChant @miquork): on what sites are the RelVals being run nowadays?

In yesterday's core software meeting we had a feeling that the solution on WMCore side to limit production (including RelVal) jobs to x86-64-v3 might not be in place by CMSSW_15_0_0_pre1 (scheduled to be built on December 10), where core software is presently planning to switch the default from v2 to v3 (and where this change would be validated). We assumed PdmV would be able to add the necessary resource matchmaking criteria at the submission time, but then wondered if already the set of sites (e.g. FNAL?) used for RelVals would, in practice, guarantee x86-64-v3-only hardware.

[AdrianoDee]

@makortel currently only at FNAL. We could re-include CERN now that the data taking has ended. But we have no urgency or specific need to do so (FNAL resources are more than enough).

[belforte]

stupid Q. What about v4 ? Will v3-built exe run there ?

[smuzaffar]

stupid Q. What about v4 ? Will v3-built exe run there ?

yes. Any exe build for vN should be able to run on vN+x ( where x>=0). So v2 exec should be able to run on v2, v3 and v4

[makortel]

stupid Q. What about v4 ? Will v3-built exe run there ?

yes. Any exe build for vN should be able to run on vN+x ( where x>=0). So v2 exec should be able to run on v2, v3 and v4

Yeah, today we run v2 binaries on all v2, v3, and v4 hardware.

[belforte]

I know that what Matti asked was not a question meant for CRAB. But since CRAB wrapper set ups the env. for cmsRun with same tools as WMA does, I looked at the code and confirmed my observation that a developer area is created and then the environment settings from scramv1 run -sh are copied into the wrapper memory and passed to subprocesses as needed. Of course Alan can still correct me if I got it wrong.

[makortel]

Thanks @belforte. So both WMA and CRAB behave similarly in this regard.

@smuzaffar Reading your slide 5 again from yesterday, do I understand correctly that in the case that a job lands on a v2-only node and sets up the developer area, that developer area would set up the full multi-arch behavior including the selection of best microarchitecture (v2 in this case) by scram?

[smuzaffar]

Yes @makortel , (though not implemented yet but) idea is that if one creates a dev area on v2-only node then scram should automatically

• enable multi-microarch support when scram project is called
• set env for best microarch and in this case it should be v2

All cmssw jobs should work fine as long as any part (shared libs/plugins) of cmssw was not (re)build in dev area before submitting the job. So if Central Production/Relval jobs just create cmssw dev area and run cmsDriver/cmsRun then it should work fine (regardless of the micro-arch of the node where it runs).

Problem I see is for crab jobs where user do checkout part of cmssw and build and submit. If someone submits a job with v3 only builds (default microarch) and it then lands on v2 node then all the user libs will be ignored.

[mapellidario]

Stefano's conclusions are correct

confirmed my observation that a developer area is created and then the environment settings from scramv1 run -sh are copied into the wrapper memory and passed to subprocesses as needed.

However, the technical details are slightly different. WMCore runs cmssw from WMCore/WMSpec/Steps/Executors/CMSSW.py, and uses Scram.py for the pset tweak and a bash script inside a popen for cmsRun. An example of the arguments is [1].

Long story short, wmcore creates a developer area

WMCore/src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py

Line 447 in 83c08ff

$SCRAM_COMMAND project $SCRAM_PROJECT $CMSSW_VERSION

then loads into the env the output of scramv1 runtime -sh

WMCore/src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py

Line 454 in 83c08ff

eval `$SCRAM_COMMAND runtime -sh`

then executes cmsRun

WMCore/src/python/WMCore/WMSpec/Steps/Executors/CMSSW.py

Line 461 in 83c08ff

$EXECUTABLE -j $JOB_REPORT $CONFIGURATION 2>&1 &

---

[1]

ref: https://cmsweb.cern.ch/reqmgr2/config?name=cmsunified_task_BTV-Run3Summer22NanoAODv12-00044__v1_T_241118_130440_5190

.steps.cmsRun1.application.section_('controls')
.steps.cmsRun1.application.section_('multicore')
.steps.cmsRun1.application.multicore.eventStreams = 0
.steps.cmsRun1.application.multicore.numberOfCores = 4
.steps.cmsRun1.application.section_('configuration')
.steps.cmsRun1.application.configuration.cacheName = 'reqmgr_config_cache'
.steps.cmsRun1.application.configuration.configId = 'fe55d6061c26d6e60fd635da09b033a0'
.steps.cmsRun1.application.configuration.retrieveConfigUrl = 'https://cmsweb.cern.ch/couchdb/reqmgr_config_cache/fe55d6061c26d6e60fd635da09b033a0/configFile'
.steps.cmsRun1.application.configuration.pickledarguments = b'(dp0\nVglobalTag\np1\nV132X_mcRun3_2022_realistic_v3\np2\ns.'
.steps.cmsRun1.application.configuration.configCacheUrl = 'https://cmsweb.cern.ch/couchdb'
.steps.cmsRun1.application.configuration.section_('arguments')
.steps.cmsRun1.application.configuration.arguments.globalTag = '132X_mcRun3_2022_realistic_v3'
.steps.cmsRun1.application.configuration.configUrl = 'https://cmsweb.cern.ch/couchdb/reqmgr_config_cache/fe55d6061c26d6e60fd635da09b033a0'
.steps.cmsRun1.application.section_('setup')
.steps.cmsRun1.application.setup.scramCommand = 'scramv1'
.steps.cmsRun1.application.setup.cmsswVersion = 'CMSSW_13_2_6_patch2'
.steps.cmsRun1.application.setup.scramProject = 'CMSSW'
.steps.cmsRun1.application.setup.softwareEnvironment = ''
.steps.cmsRun1.application.setup.buildArch = None
.steps.cmsRun1.application.setup.scramArch = ['el8_amd64_gcc11']
.steps.cmsRun1.application.section_('gpu')
.steps.cmsRun1.application.gpu.gpuRequirements = None
.steps.cmsRun1.application.gpu.gpuRequired = 'forbidden'
.steps.cmsRun1.application.section_('command')
.steps.cmsRun1.application.command.configurationHash = None
.steps.cmsRun1.application.command.executable = 'cmsRun'
.steps.cmsRun1.application.command.arguments = ''
.steps.cmsRun1.application.command.configuration = 'PSet.py'
.steps.cmsRun1.application.command.psetTweak = None
.steps.cmsRun1.application.command.configurationPickle = 'PSet.pkl'

[anpicci]

@makortel FYI, Dario provided a reply for your question targeting WMCore

[anpicci]

By the way, any development from WMCore on this topic will likely materialize in the next quarter, since IMU we still lack a common consensus about where to implement the binding to the specific microarchitecture, and our effort is committed to finalizing Q3 dev issues and urgent operation activities. According to the discussion here, I would support @belforte's proposal, and I am not convinced WM is the right place to introduce this binding.

[belforte]

Thanks @smuzaffar for your summary. Let me rephrase from our (WM) POV to make sure I understood:

• SI will be in charge of running jobs on machines which match the microarch version specified (or a later one)
• WMA will need to request v3 for CMSSW_15 and later.
• CRAB will need to figure out (from SCRAM_MICRO_ARCHITECTURE) where the user application can run and ask for the proper microarch.
• We still do not know where those v2-only resources are and if there are full v2-only sites etc. so can't think how to deal with microarch-aware data placement or microarch-aware data access over WAN (a new twist to "overflow" ?)

[mmascher]

Hello all,

Just for my understanding, is this a use case that will happen very often, or is it something that, in general, is going to change rarely? I see WMAgent and CRAB are defining respecively:

CMSSW_Versions = "CMSSW_14_2_0_pre4"
CRAB_JobSW = "CMSSW_9_2_6"

It's very easy to add a job's requirement like (for example):

ifthenelse(regexp("$CMSSW_1[56789].*", CMSSW_Versions), Microarch=="x86_64-v3" || Microarch=="x86_64-v4", True)

We can possibly turn this into a "table" with CMSSW version and list of supported achitecture with some nested if.

I would still provision machines solely based on DESIRED_Sites, otherwise Factory ops will need to check all the worker nodes at all sites, and update the static factory xml configuration file with the information about sites.

[smuzaffar]

thanks @mmascher , your suggestion will work if we only allow to use the default micro-arch of a release. So yes for CMSSW_15_0 and above it can always request resources with x86_64-v3 (I think something like Microarch>="x86_64-v3")` should also work.

But we also need to allow running CMSSW_15_0 on Microarch>="x86_64-v2" if explicitly requested by user. As @belforte mentioned above , support for something like MY.DESIRED_MICROARCH=x86_64-v2 which then can be converted to Microarch>="x86_64-v2" should allow us to use v2 resources too

[belforte]

yes @mmascher , as @smuzaffar said CRAB situation is trickier.
Maybe we can have a CMSSW-version dependent default but if MY.DESIRED_MICROARCH is present it needs to be respected. It think that I can avoid to specify a list, but only indicate the minimum required value for microarch.

I hope too that we can avoid using the microarch in the provisioning, but if some jobs have a requirement for v2 and v2 is a scarce resource, it may not work. See GPU e.g. We need to have some understanding of where those v2's are to define policies that will work for us. I'd rather not try to handle this "a priori".
Does anybody have a clue about where those v2 machines are ? A small fraction here and there will not matter, whole sites are a different story.

[smuzaffar]

I can avoid to specify a list, but only indicate the minimum required value for microarch

@belforte , I was thinking the same and I think scram env variable can help you there. Instead of setting a comma separated list , scram can just set the min require micro-arch

[smuzaffar]

@mmascher , I see this in the Job.submit file

# These attributes help gWMS decide what platforms this job can run on; see https://twiki.cern.ch/twiki/bin/view/CMSPublic/CompOpsMatchArchitecture
+REQUIRED_ARCH = "X86_64"
+DESIRED_CMSDataset = undefined

So how hard it is to add support for +DESIRED_MICROARCH which can then be converted to Microarch>="${DESIRED_MICROARCH}" ?

We want to enable multi-arch with x86-64-v3 as default for 15.0.0.pre1 (which should be built on 12th Dec). If this can not be implemented before 15.0.0.pre1 then I am fine with the CMSSW version to microarch mapping

ifthenelse(regexp("^CMSSW_(2[0-9]+|1[5-9]+)_.*", CMSSW_Versions), Microarch>="x86_64-v3"", True)

At least this will allow us to validate 15.0.0.pre1 for x86-64-v3

[mmascher]

@belforte , we can do do something similar to ARCH, i.e., se in the job ad:

+REQUIRED_MICROARCH = "any" #default value if not set in the jobad

or

+REQUIRED_MICROARCH = "x86_64-v3,x86_64-v4" #job selects specific list of micro architectures

Then add the necessary matchmaking expressions in the machine start expression.

[smuzaffar]

thanks @mmascher , this will help.
By the way, do you really need comma separated list of microarch i.e. +REQUIRED_MICROARCH = "x86_64-v3,x86_64-v4" for matchmaking? This means for cmssw x86-64-v2 one has to set +REQUIRED_MICROARCH = "x86-64-v2,x86_64-v3,x86_64-v4" as v2 can run on any machines with v2 and above .

[belforte]

the problem with that list is not just that it is long and hard to read (well...once coded it will be consumed by machines, when debugging is not needed ☹️ ), but that as soon as a new, compatible microarch version is added, we do not match that hardware until we modify the code. @mmascher how difficult would it be to parse the Microarch machine classAd to extract the version number so that

• job indicates e.g. +REQUIRED_MINIMUM_MICROARCH_VERSION = 3
• execution slots are matched on Target.Microarch-version >= MY.REQUIRED_MINIMUM_MICROARCH_VERSION

?

[smuzaffar]

thanks @belforte , let me try script_exe

[makortel]

Thanks @smuzaffar for the detailed description in #12168 (comment). I'd suggest to document this behavior somewhere more permanently (e.g. twiki or https://cms-sw.github.io/), and make it more clear what action exactly makes a developer area on v3 node to switch from "multi microarch" behavior (that is used by production jobs) to "single microarch" (what a user building CMSSW packages will see), and what are the values of the two environment variables in each case.

[belforte]

@mmascher @amaltaro we need to finalize names :-)

I propose to use MY.REQUIRED_MINIMUM_MICROARCH as the new classAd in job submission
OK ? Or will you prefer to have something which starts with DESIRED as we do e.g. for sites ? (avoid using "require" and "request" works which are used in some standard HTC ads)

[amaltaro]

Looking into https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py, we already have a mix of REQUIRED and DESIRED job classads.
And given that we are already using My.REQUIRED_ARCH, I think MY.REQUIRED_MINIMUM_MICROARCH would make the most sense. Agree?

[makortel]

MY.REQUIRED_MINIMUM_MICROARCH

Is this name something that in the future could work for ARM or RISC-V microarchitectures, that may have different nomenclature (e.g. not necessarily "versions"), or is this thought to be specific for the x86 case we have at hand now?

[amaltaro]

Is this name something that in the future could work for ARM or RISC-V microarchitectures, that may have different nomenclature (e.g. not necessarily "versions"), or is this thought to be specific for the x86 case we have at hand now?

Ideally, yes. I think the only requirement is that we can use it with an integer, such that >, < comparisons are easily applied.
Anything more complex could potentially require further changes in glideinWMS (the requirements string), e.g. 1.2, etc.

[belforte]

@makortel you tell us ! In case you should also generalize SCRAM_MIN_SUPPORTED_MICROARCH.

If a version # is not guaranteed, I do no see alternatives to a list. Or a "method" which creates the list on the fly given the "MIN" name.
It is also a matter of which "strings" HTCondor will expose for us to match with. Current proposal is indeed based on the list which Marco put in initial posting.

Hard to decide on anything w/o concrete examples.

[makortel]

A challenge is that we don't know how we would want to deal with multiple ARM or RISC-V microarchitectures (and we don't have any real experience on either, so deciding anything now would be premature). I think this implies that if you want to have a general solution that wouldn't have be changed ever, we'd need to spend more time to think what that would be (and there would be a risk of overengineering).

If we proceed with that we know for x86(which is perfectly fine for me), I'd suggest it's made clear the behavior is specific to x86, and it is understood that if we ever want/need to deal with multiple ARM/RISC-V microarchitectures the machinery will be reassessed.

[belforte]

I fully agree. Let's stick with what is simplest now. And when we will now better, reassess. In optimistic spirit, let's avoid to put X86 in the name at this time.

[belforte]

answering my question about "which are v2-(almost-)only sites". I used condor_status -af GLIDEIN_CMSSite Microarch and a bit of python to get this table. Which suggests that T3_US_UMiss could be removed from the pool.

Site                 slots  v1%   v2%  v3%  v4%  
------------------------------------------------
T1_DE_KIT             4497   0%    0%  97%   2%  
T1_ES_PIC             1232   0%    0%  98%   1%  
T1_FR_CCIN2P3         2545   0%    0%  77%  22%  
T1_IT_CNAF            3249   0%    0%  71%  28%  
T1_RU_JINR            6416   0%    0%  86%  13%  
T1_UK_RAL             3610   0%    0%  76%  23%  
T1_US_FNAL           16839   0%    1%  63%  34%  
T2_AT_Vienna           221   0%    0%   0% 100%  
T2_BE_IIHE            4778   0%    0%  43%  56%  
T2_BE_UCL             1110   0%    0%  76%  23%  
T2_BR_SPRACE           993   0%   31%  68%   0%  
T2_BR_UERJ              99   0%    0% 100%   0%  
T2_CH_CSCS            3084   0%    0% 100%   0%  
T2_CN_Beijing          229   0%    0%   0% 100%  
T2_DE_DESY            4424   0%    0%  45%  54%  
T2_DE_RWTH            4859   0%   18%  63%  18%  
T2_EE_Estonia         2680   0%   37%  33%  29%  
T2_ES_CIEMAT          1544   0%    0%  19%  80%  
T2_ES_IFCA             930   0%   17%   0%  82%  
T2_FI_HIP              275   0%    0% 100%   0%  
T2_FR_GRIF            3070   0%    2%  77%  20%  
T2_FR_IPHC            1413   0%    6%  47%  45%  
T2_HU_Budapest        1423   0%    0% 100%   0%  
T2_IT_Bari            9728   0%    0%  96%   3%  
T2_IT_Legnaro         3042   0%    0%  88%  11%  
T2_IT_Pisa             342   0%    0%  12%  87%  
T2_IT_Rome             778   0%    0%  93%   6%  
T2_KR_KISTI            354   0%   90%   0%   9%  
T2_LB_HPC4L              7   0%  100%   0%   0%  
T2_PL_Cyfronet         752   0%    0% 100%   0%  
T2_PL_Swierk           151   0%    0% 100%   0%  
T2_PT_NCG_Lisbon       161   0%    0% 100%   0%  
T2_RU_IHEP             342   0%  100%   0%   0%  
T2_RU_INR               47   0%  100%   0%   0%  
T2_RU_ITEP              53   0%  100%   0%   0%  
T2_RU_JINR             734   0%   38%  41%  19%  
T2_TW_NCHC               9   0%    0% 100%   0%  
T2_UA_KIPT             186   0%   88%   5%   5%  
T2_UK_London_Brunel    725   0%   27%  70%   2%  
T2_UK_London_IC       5859   0%    8%  85%   5%  
T2_UK_SGrid_RALPP     2774   0%   14%  72%  13%  
T2_US_Caltech         2428   0%    3%  82%  14%  
T2_US_Florida         2186   0%    0% 100%   0%  
T2_US_MIT             8432   0%   12%  22%  65%  
T2_US_Nebraska        4756   0%   34%  37%  28%  
T2_US_Purdue          1350   0%    0%  71%  28%  
T2_US_UCSD            6258   0%    1%  96%   2%  
T2_US_Vanderbilt      6735   0%   55%  18%  26%  
T2_US_Wisconsin       8931   0%   16%  68%  15%  
T3_IT_Trieste         1185   0%   32%  59%   8%  
T3_KR_KNU              288   0%   52%  14%  33%  
T3_TW_TIDC               4   0%    0% 100%   0%  
T3_UK_London_QMUL       16   0%    0%   0% 100%  
T3_UK_London_RHUL      683   0%    0%   0% 100%  
T3_UK_SGrid_Oxford     495   0%    0%  97%   2%  
T3_UK_ScotGrid_GLA    5577   0%    0% 100%   0%  
T3_US_Baylor            37   0%    0% 100%   0%  
T3_US_Colorado         249   0%   50%  10%  39%  
T3_US_FNALLPC          606   0%  100%   0%   0%  
T3_US_OSG               38   0%   23%  76%   0%  
T3_US_PuertoRico        69   0%   14%  85%   0%  
T3_US_Rutgers           36   0%  100%   0%   0%  
T3_US_UMD                5   0%  100%   0%   0%  
T3_US_UMiss              8 100%    0%   0%   0%  

[stlammel]

Hallo Stefano,
yes, T3_US_UMiss was hit hard when we phased out v1.
They do have one machine that is not v1. (According to
the site admin.)
For the Tier-3s with 100% v2, this is likely due to the
condor_status being a snapshot in time and not all the
worker nodes there being used at the time.
Thanks,
cheers, Stephan

[belforte]

@makortel @smuzaffar about "generalization" this is what drives current CRAB implementation which I am working on:

• SCRAM_MIN_SUPPORTED_MICROARCH is a "free" string which carries enough information to use an algorithm to derive from it the correct requirements to be added in job submission to HTC
  • My initial algorithme will be:

  if min_supported_microarch.startswith('x86-64-v'): MY.REQUIRED_MINIMUM_MICROARCH=min_supported_microarch

• SCRAM_MIN_SUPPORTED_MICROARCH satisfies (r"^(?=.{0,255}$)[a-zA-Z0-9\-\_\.]+$") (i.e. max 255 chars allphanumeric plus dash, underscore and dot)

it will be easy for CRAB to change the algorithm, but I'd rather not change the above logic nor the string format. I.e. if you think that you may like to use :, let me know now.

@mmascher please confirm if MY.REQUIRED_MINIMUM_MICROARCH="'x86-64-v3" (e.g.) is OK. I will let you know when I can have jobs which defined it so that we can test matching.

[belforte]

Hi @stlammel , sure. It is indeed curious that FNALLPC is v2-only ! Anyhow the only conclusions which I derive from that table are:

• it is OK to continue to provision resources based only on site name and leave microarch selection to the job matchmaking (as Marco stated)
• it is OK to submit jobs to site which have the data, w/o worrying about microarch
• it is OK not to worry about microarch when placing datasets at sites, as long as we avoid KISTI

In other words, we keep doing things "as now", simply jobs which require v3 will see some sites as "smaller" and may be queued longer

In the global view of things, T3's are almost only used for user MC and sort of "bundled together"

[stlammel]

fully agree Stefano! - Stephan

[amaltaro]

As mentioned in this comment:
https://indico.cern.ch/event/1487904/

I was considering to use it as a single integer, such that comparison/evaluation in glideinWMS can be simple.

However, if @mmascher and others think that a construction like MY.REQUIRED_MINIMUM_MICROARCH="'x86-64-v3" is going to be more solid, I see no reason why WMCore should do something different than CRAB. So we will just use the same convention.

[belforte]

Very easy for me to set a integer too. Waiting for Marco to say what he wants!

[mmascher]

Ok, let's go with the integer solution. Here is the constraint, I gave it a test.

MICROARCH_CONSTRAINT = (REQUIRED_MINIMUM_MICROARCH <= int(substr(split(Microarch,"-")[1],1)))

It could be either added as a machine requirements in the glideinWMS frontend, or as a job requirement by WMagent when it writed the JDL.

Should I add it to the frontend?

[belforte]

thanks @mmascher . My preference is that match making requirements are set in the gWms, like done e.g. for DESIRED_Sites. Having reuquirements in multiple places is confusing.
But maybe I miss something ... I see that CRAB jobs specify Requirements = stringListMember(TARGET.Arch, REQUIRED_ARCH) to which HTC adds requirements on OpSys and other things. In other words: I am confused :-)

[belforte]

@mmascher does this mean that the default is not any anymore as you wrote earlier ?
what should it be ? 0 ?

[mmascher]

Hi @belforte ,

If we go with REQUIRED_MINIMUM_MICROARCH as an integer I would not set "any" as default. The final constraint I tested in ITB has a protection for REQUIRED_MINIMUM_MICROARCH being defined. So you can either let it undefined, or set it as 0 as a default behaviour. Either way all slots will be matched.

[amaltaro]

@mmascher one more question. Should we define this job classad only if the cpu architecture is x86-64? Or can we always have it defined, regardless of the architecture, and only update the minimum microarch according to the CMSSW version?

From what I can tell, production jobs could be requesting multiple architectures:
https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/BossAir/Plugins/SimpleCondorPlugin.py#L641

will the settings of microarchitecture keep this functionality intact? In other words, a job setting for example:

REQUIRED_MINIMUM_MICROARCH=2
REQUIRED_ARC="X86_64,aarch64"

will it still manage to match against either resource type?

[srimanob]

Hi @amaltaro
I assume if we use ARM (aarch64), we should keep "any" for the moment. We don't have much experiance on the microarch for ARM. Thanks.

[mmascher]

I can add a protection for the machine microarch as well. As far as I know it is not defined for aarch64, so in this case we can allow jobs to run regardless of their REQUIRED_MINIMUM_MICROARCH. Does this sound ok?

[amaltaro]

Yes Marco, I think this would be ideal for the moment. In other words, the desired behavior I have in mind is:

if REQUIRED_ARCH != X86_64
  disregard REQUIRED_MINIMUM_MICROARCH
else
  MICROARCH >=  REQUIRED_MINIMUM_MICROARCH  # assuming that default is REQUIRED_MINIMUM_MICROARCH=0

Does it sound good to you? @mmascher @belforte

[belforte]

that's implicit in the fact that we specify microarch with a number. Should at some point we need sam afunctionality for Arm, or whatever else, even if still a number, will likely be a different one.
I assumed that Marco is prepared to add the proper amount of if REQUIRED_ARCH at that point.
Of course we can always change on client side as well... as discussed already it is difficult to be future-proof against any possible future !

[belforte]

the agreed behavior has been implemented in CRAB, and deployed on production server, but it is only accessible using crab-dev for CRABClient. crab-prod (i.e. crab) still ignores the microarchitecture.
I'd rather not touch crab-prod client until after the holidays.