5.1rc10b

Bugfix

• Actually import the module that you want to use

• list pallet tag incorrect tag name

  List command reported all tags with correct values but a single
  incorrect name.

• Rewrite AUTO IP to improve performance

  The rewrite takes CLASS A network search time from 45s to 0.1s

• A few modules needed upgrading in test-framework

  Github started complaining about anisible and paramiko versions. I also got
  rid of the package hash checking, because it was a pain in my butt to
  maintain.

• Actually import the module that you want to use

Git

• starting 5.1rc10b

5.1rc10a

Bugfix

• Fix UPPERCASE hostnames (again)

  Stop manipulating the case on lookups and always store the original
  case (we always did this). When we do this lookups are correctly case
  insensitive but globbing is case incorrectly sensitive.

  Now do extra work to change only the globbing lookups and make the
  glob pattern case insensitive (Python cannot natively do this because
  it defaults to the OS behavior).

5.1rc10

Feature

• Better Git->ReleaseNotes

• Move hardcoding of commands that require sudo in Web Service

  FIXES - JIRA-739

  Currently, there are several commands in the Stacki command line, that,
  when invoked through the web-service, require sudo access. These commands
  are hardcoded into the web-service.

  This check-in adds the ability to dynamically update list of commands
  that require sudo access.

• add "list host switch" to sudo commands

• Auto sync when adding and removing sudo commands

  When adding or removing commands to the sudo list, default to
  automatically syncing the sudoers file. However give the user
  the option to turn syncing off. This way, you can add multiple
  commands, and then run sync once.

• Add ibfabric to the output of 'stack list switch expanded=true'

• Remove set host osaction and set host installaction

• Remove the bootflags related commands

  This commands are no longer used and were removed:

  • remove host bootflags
  • report host bootflags
  • set host bootflags

• Tags (sort of like Attributes but not)

  This is the stat of a Tags feature. Tags can be applied to various objects in the
  Stacki Database but do not roll up into XML entities when building host profiles.
  Tags are only visible through the CLI/API and for now only with the following
  commands:

  • stack add pallet tag
  • stack set pallet tag
  • stack list pallet tag
  • stack remove pallet tag

Bugfix

• Change how getBoxPallet returns information

  This branch started as a bugfix branch for "run pallet".
  "run pallet" did not work if there were 2 pallets of the same
  name on the frontend, but only one was enabled for the frontend box.

  This required a re-architecting of the "run pallet" command, which
  involved changing how getBoxPallet function returned information.

  With this change "getBoxPallets" function returns output in the
  same format as "getPallets" from the PalletArgument Processor.
  This makes it easy to compare the 2 data structures.

  This also means that, all commands using getBoxPallets need to change.
  As part of this cleanup, "report host repo" was also cleaned up.

• Specify command when erroring out.

• Fix uppercase hostnames (again)

• Have load json restore both the osaction and installaction

• set bootaction will now either create a new bootaction or override an existing one, if iRe
  already exists.

• Accept forwarded traffic by default.

  The Default Firewall policy on a machine installed
  by Stacki is to drop forwarded packets. This breaks
  installs with multiple networks, and hypervisor installs.

  To allow forwarded packets to pass-through by default, change
  the default policy on the FORWARD chain to ACCEPT.

  Fixes JIRA: STACKI-630

• Make sure the pallet matches the box OS before enabling it

Breaking Change

• Any commands that currently consume the output

  of stack {verb} api commands will need to be audited to
  ensure that the headers are being referenced correctly.

• schema addition

  $ mysql cluster
  CREATE TABLE tags (
  Scope           enum ('box', 'cart', 'host', 'network', 'pallet'),
  Tag             varchar(128) NOT NULL,
  Value           text,
  ScopeID         int(11),
  INDEX (Scope),
  INDEX (Tag),
  INDEX (ScopeID)
  );
  

Git

• starting 5.1rc10

5.1rc9

Feature

• Ability to add user-defined global configuration for the Dell 1052 switch.

  For example, to configure SNMP on the switch named ethernet-231-43, exeucte:

  stack add host attr ethernet-231-43 attr=switch_global_config value="snmp-server community \ public ro view Default"

• Attribute control to configure a host's switch port as a trunk.

  A trunk port allows all VLAN traffic to pass through it. You can set a host's switch port
  to a trunk by executing:

    `stack set host attr hostname attr=switch_port_mode value=true`
  

• Include man, and man-pages in install

  SLES does not include man, or man-pages as part
  of a Minimal install set. Include the packages.

• Support for globbing on uppercase hostnames

  Globbing on hostnames is now case insensitive.

• set host power updates the message queue

• [Switch] Manage ssh keys on Mellanox IB switches

  • Automatically add stacki's public key to the switch on a sync.
  • Wipe out all ssh authorized keys on nukeswitch=true

• Added nukeswitch=yes|no to the Dell 1052 ethernet switch configuration.

  When nukeswitch=yes, all VLANs will be removed from the switch and each port will be confi
  gured onto the default VLAN (i.e., VLAN 1).

Bugfix

• stack enable/disable pallet is now multiple arg friendly

• Make remove os actually remove the OS, even if it has attrs or routes attached

• [Switch] Correct membership settings for IB configs

• Duplicate default host interfaces won't be allowed in stack add host interface

Git

• starting stacki-5.1rc9

5.1rc8

Feature

• remove the deprecated status in list host

• Better tracking of 3rdparty packages

  • Update LICENSE.txt files for foundation packages
  • Generate a nosat.txt file for pip packages
  • Move more binaries out of git and into s3
  • Added top level LICENSE-3rdparty.txt

Bugfix

• Freeze python-daemon pip module to 2.1.2

Git

• starting 5.1rc8

5.1rc7

Feature

• When stack sync host network localhost is executed, cleanup the ifcfg-* files that

  don't have the stamp that indicates the ifcfg file was written by Stacki. The stamp is:

    # AUTHENTIC STACKI
  

• Add environment scope for routes and firewalls

  BREAKING CHANGE:
  CREATE TABLE environment_routes (
  Environment int(11) NOT NULL default '0',
  Network varchar(32) NOT NULL default '',
  Netmask varchar(32) NOT NULL default '',
  Gateway varchar(32) NOT NULL default '',
  Subnet int(11) default NULL references subnets,
  Interface varchar(32) default NULL
  );

  CREATE TABLE environment_firewall (
  Environment int(11) NOT NULL default '0',
  Tabletype enum('nat','filter','mangle','raw') NOT NULL
  default 'filter',
  Name varchar(256) default NULL,
  InSubnet int(11) default NULL references subnets,
  OutSubnet int(11) default NULL references subnets,
  Service varchar(256) default NULL,
  Protocol varchar(256) default NULL,
  Action varchar(256) default NULL,
  Chain varchar(256) default NULL,
  Flags varchar(256) default NULL,
  Comment varchar(256) default NULL
  );

• add subnet and interface columns to list route; clean up list route and add route

• Move 'stack report system' to its own package

• Checkin of baseline code to interact with Mellanox 7800 IB switch.

  This code provides the capability to:

  • add ssh keys
  • toggle and display subnet manager status
  • create/delete partitions based on the 'ibfabric' and 'ibpartition' parameters in the management host interface
  • add/remove members from partitions, based on the 'ibpartition' parameters in their ibN host interfaces
  • members are added at differing membership levels, determined automatically

• Remove insert-ethers

  The newer node discovery code replaces the older insert-ethers. This
  commit also removes a few pylib files no longer used.

  The backup-db script is moved into the admin package.

• dump and load system configuration information into and out of a json document

• Add a check in 'stack report system' for named

Bugfix

• fix sync host network cleanup for redhat

• Reverse the way stack sync host network localhost works.

  Keep all ifcfg files in /etc/sysconfig/network that don't have the Stacki stamp
  (AUTHENTIC STACKI) and remove all the ifcfg files that do have the Stacki stamp. Then later
  in the command, new ifcfg files will be written for each interface in the database for the
  frontend.

• Update stack list switch config to output info about general and trunk ports as

  well as output the coorelation with the hosts' interfaces.

• Suppress error message when removing routes from the live routing table.

• Expand firewall rules to multiple lines when needed in iptables.

  A firewall rule in the database can expand to multiple iptables rules in a couple cases:

    1) The database rule refers to a network that associated with multiple interfaces on
    the host.
  
    2) When 'protocol' equals 'all' and there is a 'service' specified.
  

  In case 1, we need a separate rule for each interface.

  In case 2, we need two separate rules, one with '-p tcp' and one with '-p udp'.

• [WebService] 'stack add pallet .iso' requires root for loopback mounting

• [WebService] remove the unneeded admin web interface as well as list_dir functionality

• [WebService] Fix string encoding bug in web service, and disable debug output from django

• Fix Stack report host bootfile so that it doesn't throw stack traces

• Fix stack sync host firewall restarts

  1. The current code uses the OS information of the frontend
     to decide which commands to run. This means for SLES 11
     backends installed from a SLES 12 frontend, the sync host firewall
     command will always be wrong, and error out.
  2. Cleanup firewall startup script for SLES 11 SP3.
     Add status, and restart commands to the service script

• Make argument processors in stack commands work consistently

  Continuation of a previous fix, some of the code went on walkabout but
  it is back now.

  Now fails when any of the following objects do not exist:

  • appliance
  • box
  • cart
  • environment
  • host
  • network
  • os
  • pallet

• In stack report switch, don't output a vlan stanza if there are no vlans

  configured for any backend hosts, and don't output a line that tells the switch to not
  DHCP for its management IP address (once you set the IP address, the switch automatically
  puts the management IP in static mode.

• Add test cases for add host and list attr

• stack add host won't accept empty string as rack and rank

• Modifying code to remove long name from appliances

  BREAKING CHANGE:
  ALTER TABLE appliances DROP COLUMN Longname;
  Replace places where longname is used to the appliance name.

• WS incorrectly handled params with = in value

  While we are here also

  • move logging to local1 (was local2)
  • log the complete command line called

• Log and ignore incorrect inconfigurations during reports

• in report system, only check named if it needs to be checked. Also added a check for sshd.

• Validation of host interface before entering them in etc/hosts file

Git

• starting 5.1rc7

5.1rc6

Feature

• Add a check in 'stack report system' for named

• Upgrade to Python 3.6.6

• turn status field into a real feature

  Enhanced the message queue status reporting to be hostid based (not
  hostname) and added a ttl to all messages. The first part fixes issues
  when hosts are renamed and the status gets lost, now everything is
  keyed off the ID from the nodes table (permanent for lifetime of
  host). The second allows us the age out messages at different rates.

  Instrumented multiple parts of the installer to send more fine grained
  status messages. The typical workflow of a system install will see the
  following messages:

  message	ttl
  install profile request	30
  install profile sent	30
  install profile	300
  install download	3600
  install stage=pre	3600
  install stage=pre-package	3600
  install download	3600
  install stage=post	3600
  install reboot	300
  install stage=boot-pre	3600
  install stage=boot-post	3600
  online	120

  The install download is seen both before the second stage image
  downloads and rpm tracker lookups. All install * messages should be
  thought of as the previous install and the extra information is for
  debugging and may change at any time.

  The online state was previously called up and just meant that the
  installer's first pass was done and the host was on the network. For
  SuSe it could be seens even though the second pass of the installer
  was active. To improve on this Stacki now tracks is sshd is running
  as a seperate state (piggy-back on the existing health message). The
  inspired a new command stack list host status which a state and
  ssh fields. The state field is the previous health message as
  described above. The ssh field is up IFF sshd is running, and None
  if it is not (TTL same as the state health).

  The new stack list host status command means:

  • stack list host status field is now deprecated and will be
    removed in the next release. Just still present to keep existing
    consumers from crashing but it will always says deprecated.

  • pluggins can be used to add move status (is service X runnning?)

  Other stuff:

      * update Redis to 4.0.11
    * change rmq to smq
    * health channel is now json payload (backwards compat with
        old text - but will deprecate)
    * stack-mq is now in the installer (SLES only)
    * stack-publish.py is gone replaced by updated smq-publish
    * fixed some leftover Python3 transition encode()/decode() bugs
    * remove old processors related to StackIQ Enterprise
    * changed `message` field in `Message` to `payload` (less confusing)
    * added psutil to foundation-python-packages
      * this requires refreshing version of all python code
      * hard code PyMySQL to 0.8.1 (newer version require newer pip)
  

  ToDo:

    * Instrument RedHat installer
    * Need a `install start` state (where to trigger it?)
    * fix test_enable_discover.py
  

• Added the ability to have 'auto' in the IP address field in a host configuration spreadsheet and 'set host interface' command.

  Host IP address will be automatically assigned based on the network address, mask.

  JIRA: STACKI-419

• remove the dump command

• Enhance stack add pallet to recognize and add SUSE Enterprise Storage media.

• Add 'load hostfile' test for duplicate interfaces

• add cart accepts url and credentials on the command line

  BREAKING CHANGE:
  ALTER TABLE carts ADD url TEXT;

  list pallet expanded=true displays the url
  generalize the download function of add pallet and place it in pylib
  from stack.download import fetch
  file = fetch(url, username="", password="")
  username and password are optional

• add ability to list supported switches

• Split switch library into multiple files

• Add vendor/make to output of 'stack list switch'

• Add tests for 'stack add/remove host alias'

  JIRA: STACKI-286

• add a url column to the rolls table

  BREAKING CHANGE:
  ALTER TABLE rolls ADD url TEXT;

  add an optional expanded parameter to list pallet to display the url
  accept username and password parameters for downloading pallets

• [add,list] storage [controller,partition] with 'os' scope

  changed sql commands to the new format to prevent injection

Bugfix

• Validation of host interface before entering them in etc/hosts file

• stack report ansible remove padChar ------ from output

  Also moved ansible into foundation-python-packages, and cleaned
  up the code to use correct stacki-style argument parsing

• when adding a new api group, if the group already exists, the command error that gets raised is malformed

• Ensure stack report host only produces unique entries.

• Update interface information with MAC addresses correctly

  When installing a backend node, all the interfaces are reported
  back by the installer. Any interfaces missing on the frontend,
  will be added to the database.

  The Bug is - when interfaces are specified in the database, with
  all the information except the MAC address, this call fails to
  update the MAC address.

  This commit fixes that.

  FIXES: JIRA STACKI-585

• Set the frontend root password during frontend-install.py

• minor tweak to make running set-up.sh more user-friendly

• Plugins no longer run on a remove of an empty a:backend

• Prevent syncing dhcpd with duplicate host interfaces

• Fix database backup cronjob script

• Major overhaul of the Dell x1052 ethernet switch configuration code.

  BREAKING CHANGE: Apply 'ALTER TABLE switchports DROP PRIMARY KEY;' to database.

• Prevent loading a hostfile with entries containing an IP but no network

  This fixes JIRA: STACKI-551

• In select(), we need to stringify the sql parameters to be able to join() them.

• Prevent set host boot from throwing exception when given a valid-but-empty host specification.

• add api user needs to return formatted output

  This fixes JIRA: STACKI-536

  When running stack add api user, the code would print
  the credentials of the user that were just created.
  This mean when this command was called through the ReST API,
  it would fail, since it would print the output on standard out,
  and not through stacki command line formatted output.

  This commit fixes the above problem.

  FEATURE: Calling "add api user" from the ReST API returns JSON

  The following JSON

  [
      {
          "hostname": "front.end.name",
          "key": "ran.dom.key",
          "username": "user.name"
      }
  ]

This can then be parsed by clients for future use.

• fix typo in remove storage partition and prevent call/command list.firewall from printing other appliance rules as their own

• 'sync host network' continues without backend's ssh available

• Added more required input and error handling for "remove storage partition"

  Now requires device="*" to remove all scope=global partitioning
  Warns when using deprecated host input without scope=host
  Warns when using a non-existent scope

• Removed extra "." on PQDN

Git

• starting release 5.1rc6

5.1rc5

Feature

• cleanup report attrfile

  print each entry on a single line
  ignore attrs with empty values; if loaded back these will erase attrs completely
  unload attrfile updated for python 3

• Add "chassis power" commands to control and list the power settings of hosts.

  To set the power on a host, execute:

  stack set host power {host ...} command="on"|"off"|"reset"

  To list the power for a host, execute:

  stack list host power {host ...}

Bugfix

• SLES nukedisks=false is now functional 💾

  JIRA: STACKI-307

Git

• starting release 5.1rc5

5.1rc4

Feature

• Add carts using URLs. Remove Unpack Carts

  Add carts using URL.
  Remove "unpack" cart command
  Clean up docstrings. Use authentication for downloading
  carts using URLS

  Fixes JIRA: STACKI-503

• Reject common hostnames ('frontend' and 'backend')

  This fixes JIRA: STACKI-350

• Add user settable host metadata

  This is similar to the AWS meta-data, the idea being a container for
  arbitrary data unique to a host. stack set host metadata is used
  to set this and the data itself is made available from a read-only
  host attribute called metadata.

  The same thing could have been accomplished using a standard attribute,
  but the intention is to have the semantics that this is unique to a host
  and not subject to the standard attribute inheritance rules.

  BREAKING CHANGE: schema addition

  $ mysql cluster
  MariaDB [cluster]> alter table nodes add column MetaData text default NULL;
  

• Have the test-framework VMs use less RAM

  Change all the VMs to be created with 1GB of RAM instead of 2GB. That
  should still be enough for them to get their jobs done.

• Generate Release Notes Automatically

  stack-releasenotes is installed on all nodes

  This is generated from the Git Log and assumes the new nice
  format for commits. Ugly old git logs are also included (for now)

Bugfix

• Cart permissions need to be recursively applied.

  Cart permissions were only being applied at the top level, and
  to the files in the directories. None of the directory permissions
  changed. This commit fixes it so that directory permissions change
  as well

• Give backends 2GB each during tests

  It turns out that the Centos 7 backends don't like to install into
  anything with less than 2GB of RAM.

• Don't install the stacki-releasenotes package on SLES11

• stack list os

  list os needs to trick the endOutput code to report only one column, all
  other commands report multiple columns. Recent changes in endOutput broke
  the trick. For output-format=json the was not broken, only for text output.

• stack list host sorts hosts by rack/rank numerically then alphabetically

  Hosts will be sorted by rack numerically first, then alphabetically. That is, all hosts with
  numerical rack values will be listed first, then all hosts with non-numerical rack values will
  be listed last.

  Hosts with the same rack value, will be sorted numerically first, then alphabetically. Below
  is an example output of stack list host:

  HOST	RACK	RANK	APPLIANCE
  stacki-2-1	0	0	frontend
  backend-0-0	1	station-11	backend
  ethernet-2-1	2	1	switch
  stacki-2-10	2	10	backend
  stacki-2-11	2	11	backend
  stacki-2-12	2	12	backend
  infiniband-2-18	2	18	switch
  infiniband-2-20	2	20	switch
  ethernet-2-43	2	43	switch
  backend-0-1	sector-42	2	backend
  backend-0-2	sector-42	3	backend
  backend-0-3	sector-42	4	backend
  backend-0-4	sector-42	station-8	backend

• Ignore drop database warning

  This fixes JIRA: STACKI-493

• SLES11 doesn't have the stack-releasenotes package (currently)

• Better handling for old pip wheel files

  pip2src crashed on a package update because it of lazy parsing
  of the METADATA file.

• stack-releasenotes missing from manifest

Git

• starting release 5.1rc4

5.1rc3

Feature

• Separate 'stack remove storage partition' into commands based on scope.

  Below commands have been introduced:
  stack remove appliance storage partition
  stack remove host storage partition
  stack remove os storage partition

  JIRA: STACKI-246

• Add non-ethernet interfaces to database

  • Send back ipmi interfaces, and InfiniBand interfaces to frontend, during install.
  • On brand new machines, root user is not created or enabled in the BMC.
    This explicity creates, and enables the root user.

• Add new exception ArgNotFound for invalid entities

  Don't return CommandError for bad lookups in *ArgumentProcessor, instead return ArgNotFound

• Add test-framework to Stacki codebase

  This moves the stacki-test-framework project into the stacki
  codebase so that the tests can be easily kept in sync with
  changes to the code.

• More boot argument for Console Installs

  • Add i40e driver support for all SLES 11 SP3 installs. This will load
    the Intel i40e driver into the kernel. If we happen to use Intel Purley
    nodes, then the driver gets used. Otherwise, it gets ignored.

  • Add nomodeset and textmode for all console installs
    We do not require X drivers for console installs. On Intel reference
    nodes, installs will stall if it tries to load the X VGA driver. So
    force it to not load the drivers.

• Calculate, report and list the MD5 hashs for hosts.

  Calculate, report and list the MD5 hashes of a host's:
  - pallets
  - carts
  - profile (e.g., kickstart file, autoyast file, etc.)

  This will be reported by a host via the message queue.

  One can check if a host is 'synced' or 'notsynced' by executing:

    stack list host <hostname> hash=y
  

  The above command adds a 'HASH' column with the status.

• add ability to specify 'channel' as a parameter for 'add host interface'

• Make table output format more consistent.

  stack list commands with NULL output in the last column will now have -'s printed
  up to the width of that column's header as in other columns.

  This allows cell filling in the 'channel' column of 'stack list host interface',
  without cluttering the output with lots of -'s in 'stack list host' if 'comment' is long.

• Add jmespath module

  This module is needed by Ansible's json_query filter.

Bugfix

• stack load storage partition failed for host partitions

  This was a new bug since stacki-5.1rc2

• Run udevsettle to wait for controller config to finish

  When configuring the controller, the command can exit, but the controller
  still hasn't completed configuration.
  udevsettle waits until the controller is done, before generating
  partitioning info

• Remove 'stack run host test' since it requires salt

  This was old code left over from StackIQ Enterprise. It's great stuff
  but beyond the mission of ping and prompt. It may come back again
  one day.

  Removed this since it was barfing on the code coverage tests due to
  missing python modules.

• WS flush cache and don't return stack trace on CommandError

  WS runs multiple threads with each long lived thread managing its own
  cache. A thread cannot invalidate another thread's cache, so always
  flush right before running the stack command (subcommands still use
  the cache).

  Previously we sent the stack trace on CommandErrors, now behave the
  same as stack.py and send the nice error message.

  Code cleanup based on flake8.

• Allow correct python db api parameters

  Allow our sql methods (execute() and select()) to accept either our current
  style, which is vulnerable to sql injection, or the correct style which allows
  the database driver to correctly handle escaping parameters. This will let us
  migrate code to the safer syntax gradually.

  See github issue #198 for more details

• Check for duplicate hosts correctly

  When using self.db.select function, make sure to not
  have "select" as the first word in the sql statement.
  This is implicit, and adding it will cause the code to
  fail.

• make new appliances 'managed' in some situations

  Appliances are already set to kickstartable==True if node==backend
  This piggy backs on that behavior to make them also managed==True 📎

  Remove redundant appliance attributes.

• Loading hostfile csv throws error on frontend renames 🔏

  CommandError(self, 'Renaming frontend is not supported!')
  JIRA: STACKI-272

• Run pallet fails if frontend has more than one pallet of the same name

  If there are 2 pallets of the same name on the frontend, even if they happen to
  be different versions, or releases, and they belong to different boxes, "run pallet"
  will still fail on an SQL subquery.

  Make the SQL query cleaner, and context specific.

• blank 'run host command=' no longer hangs

  🏃
  JIRA: STACKI-325

• stack list host graph

  Bring "stack list host graph" to the 21st century

• Refer to the correct information

  When printing nukecontroller information, refer
  to the list that contains nukecontroller information
  and not the nukedisks information.

• cleanup entity names in example csv files

• Make api response return JSON

  The api response was mistakingly running json.dumps against the text
  version of the command response, causing the output to be double
  encoded.

  This fixes JIRA: STACKI-432

Git

• starting release 5.1rc3

• Update README.md

• Update README.md