support cray EX compute chassis

[garlick]

The Cray Shasta Hardware Architecture includes a compute chassis we need to support with powerman + redfishpower.

As discussed in #126, it would be cleanest if we had one device specification for the entire chassis. To do this we can define "plugs" for each controllable target (8 compute nodes, 8 switches,1 chassis) and ask redfishower to substitute them into the URIs configured with setstatpath, setonpath, and setoffpath (#129).

Then the dev script can look something like this

specification "redfishpower-cray-olympus" {
        timeout 60

        plug name {
                "Blade0"
                "Blade1"
                "Blade2"
                "Blade3"
                "Blade4"
                "Blade5"
                "Blade6"
                "Blade7"
                "Perif0"
                "Perif1"
                "Perif2"
                "Perif3"
                "Perif4"
                "Perif5"
                "Perif6"
                "Perif7"
                "Enclosure"
        }

        script login {
                expect "redfishpower> "
                send "auth root:initial0\n"
                expect "redfishpower> "
                send "setheader Content-Type:application/json\n"
                expect "redfishpower> "
                send "setstatpath redfish/v1/Chassis/%s\n"
                expect "redfishpower> "
                send "setonpath redfish/v1/Chassis/%s/Actions/Chassis.Reset {\"ResetType\":\"On\"}\n"
                expect "redfishpower> "
                send "setoffpath redfish/v1/Chassis/%s/Actions/Chassis.Reset {\"ResetType\":\"ForceOff\"}\n"
                expect "redfishpower> "
        }
        script logout {
                send "quit\n"
        }
        script status_all {
                send "stat\n"
                foreachnode {
                        expect "([^\n:]+): ([^\n]+\n)"
                        setplugstate $1 $2 on="^on\n" off="^off\n"
                }
                expect "redfishpower> "
        }
        script on_ranged {
                send "on %s\n"
                expect "redfishpower> "
        }
        script off_ranged {
                send "off %s\n"
                expect "redfishpower> "
        }
        script cycle_ranged {
                send "on %s\n"
                expect "redfishpower> "
                send "off %s\n"
                expect "redfishpower> "
        }

Note: we observed on test hardware (hetchy-cmm2) that when the chassis is off

• querying a blade works (shows off)
• turning a blade on fails with 400 "bad request"

[chu11]

Just an addendum, the powerman.conf would subsequently look like:

device "chassis0" "redfishpower-cray-olympus" "redfishpower -h chassis0 |&"
device "chassis1" "redfishpower-cray-olympus" "redfishpower -h chassis1 |&"

node "node[0-7]"   "chassis0" "Blade[0-7]"
node "switch[0-7]" "chassis0" "Perif[0-7]"
node "chassis0"    "chassis0" "Enclosure"

node "node[8-15]"   "chassis1" "Blade[0-7]"                                                                                                        
node "switch[8-15]" "chassis1" "Perif[0-7]"                                                                                                        
node "chassis1"     "chassis1" "Enclosure"  
...

[garlick]

This could also work (assumes a particular plug order):

device "chassis0" "redfishpower-cray-olympus" "redfishpower -h chassis0 |&"
device "chassis1" "redfishpower-cray-olympus" "redfishpower -h chassis1 |&"

node "node[0-7],switch[0-7],chassis0"    "chassis0"
node "node[8-15],switch[8-15],chassis1"  "chassis1"                                                                                                 

[garlick]

New information from the sys admins:

• the compute blades have two nodes each
• each node has its own BMC with an IP address independent of the chassis
• those BMC IPs cannot be reached when the blade is turned off.

Summarizing an offlline discussion (correct me if I'm wrong @chu11), we need something like:

• add plugs to the chassis for each node, e.g. Node[0-15]
• add a way to tell redfishpower all the plug names in advance (to support stat with no arguments)
• add a way to express the hierarchy of Chassis -> Blade -> Node so that redfishpower can be smart about those relationships (see redfishpower: recognize hierarchies / pre-requisites #81)
• add a way to set different paths for different plugs
• add a way to associate a plug with a particular hostname

Idea: add the following "set" command:

# setplugs <pluglist> <hostindices> [<parentplug>]
# setpath <pluglist> <type> <path>

where

• <hostindices> is a list of indices into the hostlist provided on the command line with -h hosts, and associates a BMC address with each plug.
• <parentplug> is one plug that is the parent of the <pluglist> in terms of control hierarchy
• <type> is stat|on|off

Example (presumes Node0 ... Node15 were added to the plug name list)

                # setplugs <pluglist> <hostindices> [<parentplug>]
                send "setplugs Enclosure 0\n"
                expect "redfishpower> "
                send "setplugs Perif[0-7],Blade[0-7] 0 Enclosure\n"
                expect "redfishpower> "
                send "setplugs Node[0-1] [1-2] Blade0\n"
                expect "redfishpower> "
                send "setplugs Node[2-3] [3-4] Blade1\n"
                expect "redfishpower> "
                send "setplugs Node[4-5] [5-6] Blade2\n"
                expect "redfishpower> "
                send "setplugs Node[6-7] [7-8] Blade3\n"
                expect "redfishpower> "
                send "setplugs Node[8-9] [9-10] Blade4\n"
                expect "redfishpower> "
                send "setplugs Node[10-11] [11-12] Blade5\n"
                expect "redfishpower> "
                send "setplugs Node[12-13] [13-14] Blade6\n"
                expect "redfishpower> "
                send "setplugs Node[14-15] [15-16] Blade7\n"
                expect "redfishpower> "
                # setpath <pluglist> <type> <path>
                send "setpath Enclosure,Perif[0-7],Blade[0-7] stat redfish/v1/Chassis/%s\n"
                expect "redfishpower> "
                send "setpath Enclosure,Perif[0-7],Blade[0-7] on redfish/v1/Chassis/%s/Actions/Chassis.Reset {\"ResetType\":\"On\"}\n"
                expect "redfishpower> "
                send "setpath Enclosure,Perif[0-7],Blade[0-7] off redfish/v1/Chassis/%s/Actions/Chassis.Reset {\"ResetType\":\"ForceOff\"}\n"
                expect "redfishpower> "
                send "setpath Node[0-15] stat redfish/v1/Systems/Self\n"
                expect "redfishpower> "
                send "setpath Node[0-15] on redfish/v1/Systems/Self/Actions/ComputerSystem.Reset {\"ResetType\":\"On\"}\n"
                expect "redfishpower> "
                send "setpath Node[0-15] off redfish/v1/Systems/self/Actions/ComputerSystem.Reset {\"ResetType\":\"ForceOff\"}\n"
                expect "redfishpower> "

[chu11]

so I guess in the instructions for configuring this, we require the user to input hosts in a specific order, correct? And if they mess up, it just won't work?

I guess alternately we'd need some type of "map" option?

redfishpower -h pfoo[16-31] --map=pfoo[8-15]:Node[0-15] -h cmm2 --map=cmm2:Enclosure

[garlick]

Yeah, an example config would look like this:

include "/etc/devices/redfishpower-cray-olympus.dev"
device "cmm0" "redfishpower-cray-olympus" "/usr/sbin/redfishpower -h elcap-cmm0,pelcap[0-15] |&"
device "cmm1" "redfishpower-cray-olympus" "/usr/sbin/redfishpower -h elcap-cmm1,pelcap[16-31] |&"
...
device "cmm1023" "redfishpower-cray-olympus" "/usr/sbin/redfishpower -h elcap-cmm1023,pelcap[16368-16383] |&"

node "elcap-cmm0,elcap-perif[0-7],elcap-blade[0-7],elcap[0-15]" "cmm0"
node "elcap-cmm1,elcap-perif[8-15],elcap-blade[8-15],elcap[16-31]" "cmm1"
...
node "elcap-cmm1023,elcap-perif[8184-8191],elcap-blade[8184-8191],elcap[16368-16383]" "cmm1023"

It didn't seem too onerous to me to require the hostlist to track the internal plug order. These things will be generated by scripts for a big system anyway, and if it's wrong, it likely won't work at all.

Edit: well I sort of misspoke - it happens to track the plug order in this case but the mapping is determined by the second argument to setplugs so it would be device specific (and would require a note at the top of the dev file I guess):

# setplugs <pluglist> <hostindices> [<parentplug>]

[chu11]

Realized something while prototyping:

                send "setplugs Enclosure 0\n"
                expect "redfishpower> "
                send "setplugs Perif[0-7],Blade[0-7] 0 Enclosure\n"

makes sense, Enclosure and the blades both map to index 0 as the power control address.

                send "setpath Enclosure,Perif[0-7],Blade[0-7] stat redfish/v1/Chassis/%s\n"
                expect "redfishpower> "
                send "setpath Enclosure,Perif[0-7],Blade[0-7] on redfish/v1/Chassis/%s/Actions/Chassis.Reset {\"ResetType\":\"On\"}\n"
                expect "redfishpower> "
                send "setpath Enclosure,Perif[0-7],Blade[0-7] off redfish/v1/Chassis/%s/Actions/Chassis.Reset 

we actually do not need to set the paths for all the plugs Enclosure,Perif[0-7],Blade[0-7], we only need to set the path for one host ... index 0 host.

I'm contemplating if we could do it by index instead? Although I see why this config design was done, as it makes more sense. If we did

                send "setpath Enclosure stat redfish/v1/Chassis/%s\n"

it might confuse the average person b/c we're not configuring a bunch of plugs.

Just debating it a bit. We'll see how prototype code falls out.

[garlick]

I was trying to keep it open ended so any plug could have a custom path and/or host mapping.

Maybe adding the semantics to a man page would make it less opaque ?

[garlick]

@chu11 have you thought about how to handle unpopulated slots? Since the "plugs" are configured with hostlist indicies in this proposal, the hostlist should still be fully populated.

I guess you could do something like -h pfoo[16-17],unused[18-29],pfoo[20-31] since the unpopulated plugs would never be actually used...

[chu11]

honestly, I had forgotten about that possibility :-)

thinking over some alternatives, I think your suggestion might be the best one. The person doing the powerman.conf config would just have to be a little smart about it, i.e.

device "cmm2" "redfishpower" "/usr/sbin/redfishpower -h cmm2,pfoo[16-17],unused[18-19],pfoo[20-31] |&"
...
node "pfoo[16,17,20-31]" "cmm2" "Node[0-1,4-15]"

(You did "unused[18-29]" above, which I assumed you typoed and mean "unused[18-19]")

[garlick]

Yeah typo.

We have to be sure that status_all doesn't query the unpopulated nodes though. Maybe we should just implement status so we don't have to deal with that.

[chu11]

We have to be sure that status_all doesn't query the unpopulated nodes though. Maybe we should just implement status so we don't have to deal with that.

Ahh, I was going to suggest we skip any hosts with a special prefix, but implementing status seems better. Should we convert all the old device scripts to status? Redfishpower has always supported it being passed nodes (and in the future plugs).

[garlick]

Minor design point: those were always plugs from powerman's perspective. When the plug list is not declared in the device script, it just means that any plug you put on the node declaration will be used. That case is sort of weird because when you have multiple instances of a .dev script, the plugs are unique for each one.

But that was convenient abuse of the design not the design :-)

[chu11]

Minor design point: those were always plugs from powerman's perspective. When the plug list is not declared in the device script, it just means that any plug you put on the node declaration will be used. That case is sort of weird because when you have multiple instances of a .dev script, the plugs are unique for each one.

Ahhh ... hmmm. Maybe skipping any hosts with a special "unused" prefix would work out better for backwards compatibility?

Edit: Oh wait ... i guess we only have to develop a status script for chassis-like devices, where the idea of "unused" is actually possible. And we can put the plug list in there.

[garlick]

Right, the downside is status is one plug at a time, so status would be serialized across the device (chassis in this case).

Edit: actually I think powerman is smart and will use status_all when it wants all plugs, and status when it only wants some of them, and can also make do if one or the other is missing. Should confirm though.

[chu11]

Right, the downside is status is one plug at a time, so status would be serialized across the device (chassis in this case).

Or do we need a "status_ranged" equivalent script? I guess not implemented at the moment, but shouldn't be too hard to add ... I somehow added on/off/cycle_ranged :-)

[garlick]

I'm guessing we don't need it but we can keep that in mind.

[chu11]

I'm guessing we don't need it but we can keep that in mind.

Thinking about it a bit this weekend, with parent support we're doing a lot of extra messages. w/o a status_ranged we would have to do about 2.4X the number of http messages. Perhaps not a deal breaker, but could be a lot of excessive messages we send out.

pm -q cmmX,blade[0-7],switch[0-7],nodes[0-15]

if we have to query each target serially

query cmmX for its status (1 total)
query of cmmX * 16 + blade[0-7] + switch[0-7] for their status (32 total)
query of cmxxX * 16 + bladeo[0-7] * 2 + 16 nodes for nodes status (48 total)

81 queries total

if status_ranged is supported, and I implement redifshpower smartly, I think we get it down to

1 query of cmmX
8 queries of blades
8 queries of switches
16 queries of nodes

for a total of 33 queries.

but I first have to determine if I can program redfishpower to be smart.

[chu11]

As an aside, looked into whether status_ranged would be that hard to do. I think it shouldn't be too hard. Most of the heavy lifting is already handled by the other ranged script functions, gotta add the "STATUS_RANGED" equivalent stuffs.

The only trickery appears to be how to get foreachnode or foreachplug to handle the nodes/plugs that were sent to status_ranged.

There may be corner cases I don't know about, but I think in _process_foreach(), if we could do the following

diff --git a/src/powerman/device.c b/src/powerman/device.c
index 8e343c1..a473e2c 100644
--- a/src/powerman/device.c
+++ b/src/powerman/device.c
@@ -941,8 +941,13 @@ static bool _process_foreach(Device *dev, Action *act, ExecCtx *e)
     Plug *plug = NULL;
 
     /* we store a plug iterator in the ExecCtx */
-    if (e->plugitr == NULL)
-        e->plugitr = pluglist_iterator_create(dev->plugs);
+    if (e->plugitr == NULL) {
+        if (e->parent_command == PM_STATUS_PLUGS_RANGED
+            && e->plugs)
+            e->plugitr = pluglist_iterator_create(e->plugs);
+        else
+            e->plugitr = pluglist_iterator_create(dev->plugs);
+    }
 
     /* Each time the inner block is executed, its argument will be
      * a new plug name.  Pick that up here.

it should presumably work. This is pseudo-code above, b/c e->plugs is a List not a PlugList and e->parent_command does not exist. So some refactoring necessary to test.

[garlick]

Yes but if this is true, most of the time status_all would be used, e.g. when a chassis is fully populated and someone runs pm -q:

Edit: actually I think powerman is smart and will use status_all when it wants all plugs, and status when it only wants some of them, and can also make do if one or the other is missing. Should confirm though.

It feels like a premature optimization to go down that path before we have evidence that it's a problem.

[chu11]

Edit: actually I think powerman is smart and will use status_all when it wants all plugs, and status when it only wants some of them, and can also make do if one or the other is missing. Should confirm though.

Ahhh I didn't think about the fact we could implement both in a device file, I thought it was an either / or situation, since most scripts only have one of them.

[chu11]

Hmmmm, this isn't a deal breaker by any measure, but non-optimal

under most circumstances the number of plugs equals the number of host indices when configuring setplugs, e.g.

+               send "setplugs Node[0-1] [1-2] Blade0\n"

the exception of course is when we do plug substitution

+               send "setplugs Perif[0-7],Blade[0-7] 0 Enclosure\n"

BUT ... we don't define plug substitution in setpath until AFTER setplugs defines those plugs

+               send "setpath Enclosure,Perif[0-7],Blade[0-7] stat redfish/v1/Chassis/%%p\n"

so in setplugs i can't verify that the path has a substitution in it.

I could stick a flag in to say "hey this plug needs a double check later on .." but that's annoying to do for several reasons (most notably, when do I do the check? I don't know for sure when the login script is actually done ... unless I add a "check_my_config" option?)

Going back and forth on ideas ... it's non-optimal, but livable.

Edit : ... ehhh ... maybe it's not quite as annoying to add a check when a power control command is issued as I first thought, like

redfishpower> on Blade0
Blade0: missing substitution

[chu11]

Edit: actually I think powerman is smart and will use status_all when it wants all plugs, and status when it only wants some of
them, and can also make do if one or the other is missing. Should confirm though.

it appears from testing that if you define both script status and script status_all, it'll just use status_all. I think the reason is that in _enqueue_targeted_actions() we search for "all" scripts before "singleton" scripts.

Here's my testing example where blades[4-7] (and subsequently nodes[8-15]) and perif[4-7] are unpopulated.

	device "d0" "redfishpower-hpe-cray-supercomputing-ex-chassis" "$redfishdir/redfishpower -h cmm0,t[0-7],unused[0-7] --test-mode |&"
	node "cmm0,perif[0-3],blade[0-3],t[0-7]" "d0" "Enclosure,Perif[0-3],Blade[0-3],Node[0-7]"

result from stat

send(d0): 'stat\n'
recv(d0): 'Enclosure: off\n'
recv(d0): 'Perif0: off\n'
recv(d0): 'Perif1: off\n'
recv(d0): 'Perif2: off\n'
recv(d0): 'Perif3: off\n'
recv(d0): 'Perif4: off\n'
recv(d0): 'Perif5: off\n'
recv(d0): 'Perif6: off\n'
recv(d0): 'Perif7: off\n'
recv(d0): 'Blade0: off\n'
recv(d0): 'Blade1: off\n'
recv(d0): 'Blade2: off\n'
recv(d0): 'Blade3: off\n'
recv(d0): 'Blade4: off\n'
recv(d0): 'Blade5: off\n'
recv(d0): 'Blade6: off\n'
recv(d0): 'Blade7: off\n'
recv(d0): 'Node0: off\nNode1: off\nNode2: off\nNode3: off\nNode4: off\nNode5: off\nNode6: off\nNode7: off\nNode8: off\nNode9: off\nNode10: off\nNode11: off\nNode12: off\nNode13: off\nNode14: off\nNode15: off\nredfishpower> '
on:      
off:     blade[0-3],cmm0,perif[0-3]
unknown: t[0-7]

So it's expecting 17 plugs in the response. But because the redfishpower device file still configures "Blades[4-7]", "Perif[4-7]", and "Node[8-15]" plugs, way more than 17 plug status are output. My suspicion is after 17 plugs are output, all the remaining stdout is combined into that last line above. And since t0-7]/"Node[0-7]" were not amongst the first 17 output, they are listed as "unknown" status.

potential solutions

• go with status singleton script only, but I think we don't believe that's good as that's slowing down the common case
• implement status_ranged per discussion above
• have foreachplug ignore any plugs that it doesn't care about ... might be implemented similarly to status_ranged pseudo-code snippet above.
• other ideas?

Edit: another idea, select singleton script for all script if singleton and all defined? Not 100% how to do at the moment, but presumably is doable?

[garlick]

Edit: another idea, select singleton script for all script if singleton and all defined? Not 100% how to do at the moment, but presumably is doable?

Having a hard time parsing this sentence. Do you mean when status_all and status are both defined, select status_all only if the target plug list includes all plugs, otherwise use status?

Now that I think about it, why allow both status_all and status to be defined in a script if status_all is always used if defined? That would mean status is always dead code.

I like the idea of making powerman a little bit smarter as described above. We could avoid any impact on existing device files that we distribute by deleting status in all scripts that provide both, since that would mimic the current behavior. The impact on dev scripts out in the wild would just be performance so not the end of the world. (And who has serial-connected high latency devices nowadays anyway?)

[chu11]

implement status_ranged per discussion above

uh oh, didn't notice this line in the powerman code

    /* Try _all version of script.
     */
    if (all || _is_query_action(com)) {
        int ncom = _get_all_script(dev, com);

        if (ncom != -1) {
            act = _create_action(dev, ncom, NULL, complete_fun,
                                 vpf_fun, client_id, arglist);
            list_append(dev->acts, act);
            count++;
        }
    }

an all script is used if you are targetting all hosts OR doing a query.

[garlick]

What happens if we just drop the || _is_query_action() there? Would it do this:

when status_all and status are both defined, select status_all only if the target plug list includes all plugs, otherwise use status?

[chu11]

Having a hard time parsing this sentence. Do you mean when status_all and status are both defined, select status_all only if the target plug list includes all plugs, otherwise use status?

Sorry, yeah that's what I was thinking.

What happens if we just drop the || _is_query_action() there? Would it do this:

I think it would, but lots of test failures. I'm assuming b/c a lot of devices don't define a status script.

After going down and (thus far) failing the "ignore plugs we don't care about" path ... I'm seeing promising results with

diff --git a/src/powerman/device.c b/src/powerman/device.c
index 067d31b..cfc5be9 100644
--- a/src/powerman/device.c
+++ b/src/powerman/device.c
@@ -684,7 +684,7 @@ static int _enqueue_targeted_actions(Device * dev, int com, hostlist_t hl,
 
     /* Try _all version of script.
      */
-    if (all || _is_query_action(com)) {
+    if (all || (_is_query_action(com) && dev->scripts[com] == NULL)) {
         int ncom = _get_all_script(dev, com);

[garlick]

Ah I like it! A one line change!

[chu11]

A short chat with @watson6282 yesterday, it ends up that a common scenario is for the chassis to not be fully populated. This would lead to status being used over status_all in the common situation.

To get around this, we could write specific device specifications that re-defines what "all" means for those non-fully populated chassis.

But it makes me think that status_ranged might be a better long term solution.

[garlick]

Hmm, yeah maybe. But I'd hold off and see if it's actually going to be needed since it further complicates an already kind of awful interface.

[chu11]

Hmm, yeah maybe. But I'd hold off and see if it's actually going to be needed since it further complicates an already kind of awful interface.

Yeah, agreed. I was gonna try and hack up a test given previous comments, but I now see that it's a little hairier than i thought.

Notes for later: can't create execution context e->ranged_pluglist, b/c execution context could get destroyed?

Edit: i had an idea for a hack ... and somehow ipmipower tests (updated to use status_ranged) worked ....

@@ -943,6 +946,9 @@ static bool _process_foreach(Device *dev, Action *act, ExecCtx *e)
     /* we store a plug iterator in the ExecCtx */
     if (e->plugitr == NULL) {
         if (act->com == PM_STATUS_PLUGS_RANGED) {
+            if (!(e->pluglist = pluglist_create_from_plugs(e->plugs)))
+                fprintf(stderr, "boooooooooo\n");
+            e->plugitr = pluglist_iterator_create(e->pluglist);
         }
         else
             e->plugitr = pluglist_iterator_create(dev->plugs);