[e2e] [FEATURE] VM Import/Migration #522

harvesterhci-io-github-bot · 2022-09-20T01:22:42Z

What's the test to develop? Please describe

A clear and concise description of what the test you want to develop.

Prerequisite and dependency of test

Any prerequisite environment and pre-condition required for this test.
Provide test case dependency here if any.

Describe the items of the test development (DoD, definition of done) you'd like

Please use a task list for items on a separate line with a clickable checkbox https://docs.github.com/en/issues/tracking-your-work-with-issues/about-task-lists

Needs: [DevOps] Provision OpenStack on Sacrificial Harvester Cluster w/ Jenkins #1171

Additional context

Add any other context or screenshots about the test request here.

related issue: harvester/harvester#2274

irishgordo · 2023-05-05T23:04:13Z

As an update - so it's roughly figured out now how to provision openstack (via microstack) entirely automated at it's latest LTS release through the glory of Cloud-Init (so vastly better than juggling a mess of ansible scripts).

It is entirely dependent on a "Jammy" based Ubuntu image, Focal will not work with Microstack.

So it would be retrofitting something like:

@pytest.fixture(scope='class')
def user_data_with_guest_agent_and_openstack(keypair):
    # set to root user password to 'linux' to test password login in
    # addition to SSH login
    yaml_data = """#cloud-config
chpasswd:
  list: |
    root:linux
  expire: false
ssh_pwauth: true
users:
  - name: root
    ssh_authorized_keys:
      - %s
package_update: true
packages:
  - qemu-guest-agent
snap:
  commands:
    - snap install microstack --devmode --beta
runcmd:
  - - systemctl
    - enable
    - '--now'
    - qemu-ga
    - microstack init --auto --control --setup-loop-based-cinder-lvm-backend --loop-device-file-size 50
""" % (keypair['spec']['publicKey'])
    return yaml_data.replace('\n', '\\n')

With the cloud config of:

#cloud-config
package_update: true
packages:
  - qemu-guest-agent
snap:
  commands:
    - snap install microstack --devmode --beta
    - snap alias microstack.openstack openstack
    - snap set microstack config.credentials.keystone-password=testtesttest
runcmd:
  - - systemctl
    - enable
    - --now
    - qemu-guest-agent.service
  - echo fs.inotify.max_queued_events=1048576 | tee -a /etc/sysctl.conf
  - echo fs.inotify.max_user_instances=1048576 | tee -a /etc/sysctl.conf
  - echo fs.inotify.max_user_watches=1048576 | tee -a /etc/sysctl.conf
  - echo vm.max_map_count=262144 | tee -a /etc/sysctl.conf
  - echo vm.swappiness=1 | tee -a /etc/sysctl.conf
  - sysctl -p
  - microstack init --auto --control --setup-loop-based-cinder-lvm-backend
    --loop-device-file-size 50
  - snap restart microstack.cinder-{uwsgi,scheduler,volume}
ssh_authorized_keys:
  - ssh-ed25519
    AAAAC3NzaC1lZDI1NTE5AAAAIBzZT+yXkr28BJzki4WdisefgyR1hKMXWlJCd9KfajEm
    [email protected]

We'll need to "long-poll" until OpenStack is up, as it has nginx that has to provision ~20/25 min depending...
But running it on Harvester is successful.

( which is exactly what we need - then we can acquire the openstack.rc programatically and then leverage openstack's python library to provision resources we need to stand up a VM on their then utilize that VM for importing later on )

All in all MicroStack seems to be the "easier" option in comparision to needing to juggle Docker/ContainerD on the VM for DevStack.

* introduce vm-import test cases with OpenStack LTS Resolves harvester#522

irishgordo · 2023-07-27T19:52:39Z

Ran into a microstack configuration error apparently the nginx conf doesn't enable a client body size of anything - so image uploads were entirely just failing.

fixed with:

sudo vi /var/snap/microstack/common/etc/nginx/snap/nginx.conf

and modifying:

client_max_body_size 0;

to:

client_max_body_size 4G;

(to allow for max of 4G image uploads)

then fire off a restart of the nginx service in openstack/microstack via the snap package:

sudo snap restart microstack.nginx

All of that can probably be done in cloud-init, on editing the microstack nginx conf ...we just need to do like some sort of sed on it or something...

We'd need to upload an image that has the qemu-guest-tools so that when the vm gets imported into harvester, the ipv4 address will populate in the index page of the virtual machines.

Then an image can be uploaded, we would of course, in testing, leverage the openstack python sdk with the given info from openstack to do something similar to:

╭─mike at suse-workstation-team-harvester in ~/Documents/openstack
╰─○ openstack --insecure image create --disk-format vmdk --container-format bare --public --file ./jammy-server-cloudimg-amd64.vmdk jammy-image
+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| Field            | Value                                                                                                                                           |
+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| container_format | bare                                                                                                                                            |
| created_at       | 2023-07-27T19:37:27Z                                                                                                                            |
| disk_format      | vmdk                                                                                                                                            |
| file             | /v2/images/dbf9ebaf-9dda-40bc-af7b-a6f2d205ee08/file                                                                                            |
| id               | dbf9ebaf-9dda-40bc-af7b-a6f2d205ee08                                                                                                            |
| min_disk         | 0                                                                                                                                               |
| min_ram          | 0                                                                                                                                               |
| name             | jammy-image                                                                                                                                     |
| owner            | 0e8c69478f6e44cdba78426069859718                                                                                                                |
| properties       | os_hidden='False', owner_specified.openstack.md5='', owner_specified.openstack.object='images/jammy-image', owner_specified.openstack.sha256='' |
| protected        | False                                                                                                                                           |
| schema           | /v2/schemas/image                                                                                                                               |
| status           | queued                                                                                                                                          |
| tags             |                                                                                                                                                 |
| updated_at       | 2023-07-27T19:37:27Z                                                                                                                            |
| visibility       | public                                                                                                                                          |
+------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
╭─mike at suse-workstation-team-harvester in ~/Documents/openstack

irishgordo · 2023-07-27T20:47:38Z

Additionally needing to reconfigure, automatic snapshots causing issues as trying to purge & re-install:

snap set system snapshots.automatic.retention=no

Setting it all to no, would be useful

Additionally bumping the swap size from 200M to higher:

snap set system swap.size=4096M

irishgordo · 2023-07-27T21:56:39Z

After some debugging it seems that 22.04 Ubuntu is not a stable release for OpenStack - so needing to install with an older release of Ubuntu...
dang :/ :crying_cat_face:

irishgordo · 2023-07-27T22:06:34Z

tenantive new cloud-config:

#cloud-config
package_update: true
packages:
  - qemu-guest-agent
  - linux-modules-extra-5.8.0-63-generic
  - build-essential
snap:
  commands:
    - snap install microstack --devmode --beta
    - snap alias microstack.openstack openstack
    - snap set microstack config.credentials.keystone-password=testtesttest
    - snap set system swap.size=4096M
    - snap set system snapshots.automatic.retention=no
runcmd:
  - - systemctl
    - enable
    - --now
    - qemu-guest-agent.service
  - echo fs.inotify.max_queued_events=1048576 | tee -a /etc/sysctl.conf
  - echo fs.inotify.max_user_instances=1048576 | tee -a /etc/sysctl.conf
  - echo fs.inotify.max_user_watches=1048576 | tee -a /etc/sysctl.conf
  - echo vm.max_map_count=262144 | tee -a /etc/sysctl.conf
  - echo vm.swappiness=1 | tee -a /etc/sysctl.conf
  - sysctl -p
  - microstack init --auto --control --setup-loop-based-cinder-lvm-backend
    --loop-device-file-size 100
  - snap restart microstack.cinder-{uwsgi,scheduler,volume}
  - sed -i 's/client_max_body_size 0/client_max_body_size 4G/g' /var/snap/microstack/common/etc/nginx/snap/nginx.conf
  - snap restart microstack.nginx
ssh_authorized_keys:
  - ssh-ed25519
    AAAAC3NzaC1lZDI1NTE5AAAAIBzZT+yXkr28BJzki4WdisefgyR1hKMXWlJCd9KfajEm
    [email protected]

Needed to update, if on 20.04 - or probably jammy jellyfish would work too we need to:

enable kernel module of thin provisioning
download gcc build essential and the linux modules extra that provide that to us

TBD: the system might need to be reboot - running modprobe dm_thin_pool then restarting all the cinder-* services.

irishgordo · 2023-07-31T00:45:57Z

Will need ca cert for system?

ubuntu@vm-for-openstack:~$ glance_ca_certificates_file = /var/snap/microstack/common/etc/ssl/certs/cacert.pem^C
ubuntu@vm-for-openstack:~$ suod cat /var/snap/microstack/common/etc/ssl/certs/cacert.pem 
-bash: suod: command not found
ubuntu@vm-for-openstack:~$ sudo cat /var/snap/microstack/common/etc/ssl/certs/cacert.pem 
-----BEGIN CERTIFICATE-----

ibrokethecloud · 2023-07-31T00:50:01Z

may be worth checking the openstack source secret:

default behaviour is if no ca_cert key is provided the controller will switch to skipping cert verification

snippet from controller code:

	customCA, ok := secret.Data["ca_cert"]
	if ok {
		caCertPool := x509.NewCertPool()
		caCertPool.AppendCertsFromPEM(customCA)
		config.RootCAs = caCertPool
	} else {
		config.InsecureSkipVerify = true
	}

khushboo-rancher · 2024-03-21T21:20:58Z

Testing is tracked #1171 also. @irishgordo could you check and duplicated close one of these?

khushboo-rancher · 2024-07-06T15:47:06Z

@irishgordo Please close this if this is running fine on the staging stage. Create a new ticket to move that to prod Jenkins to track that part.

irishgordo · 2024-07-09T16:28:59Z

Based on:

I'll go ahead and close this out to track the testing implementation & the movement to prod of the Jenkins pipeline

harvesterhci-io-github-bot mentioned this issue Sep 20, 2022

[FEATURE] VM Import/Migration harvester/harvester#2274

Closed

3 tasks

khushboo-rancher added the priority/0 Must be fixed in this release label Nov 27, 2022

khushboo-rancher assigned irishgordo May 3, 2023

khushboo-rancher added the area/backend-automation Implement with pytest label May 3, 2023

irishgordo added a commit to irishgordo/tests that referenced this issue May 9, 2023

feat: VM Import Test Cases

aa98cca

* introduce vm-import test cases with OpenStack LTS Resolves harvester#522

irishgordo added a commit to irishgordo/tests that referenced this issue May 9, 2023

feat: VM Import Test Cases

9a38a0e

* introduce vm-import test cases with OpenStack LTS Resolves harvester#522

irishgordo added a commit to irishgordo/tests that referenced this issue May 9, 2023

feat: VM Import Test Cases

9c47bfd

* introduce vm-import test cases with OpenStack LTS Resolves harvester#522

irishgordo mentioned this issue May 9, 2023

feat: VM Import Test Cases #795

Draft

irishgordo mentioned this issue Mar 19, 2024

[DevOps] Provision OpenStack on Sacrificial Harvester Cluster w/ Jenkins #1171

Closed

irishgordo mentioned this issue Apr 2, 2024

[DevOps] Fix/Improve OpenStack Provisioning EPIC #1204

Closed

This was referenced Jul 9, 2024

[DevOps] Rollout OpenStack Jenkins Pipeline to Prod #1372

Open

[e2e] Implement Tests For VM OpenStack Import #1373

Open

irishgordo closed this as completed Jul 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[e2e] [FEATURE] VM Import/Migration #522

[e2e] [FEATURE] VM Import/Migration #522

harvesterhci-io-github-bot commented Sep 20, 2022 •

edited by irishgordo

Loading

irishgordo commented May 5, 2023 •

edited

Loading

irishgordo commented Jul 27, 2023 •

edited

Loading

irishgordo commented Jul 27, 2023 •

edited

Loading

irishgordo commented Jul 27, 2023

irishgordo commented Jul 27, 2023 •

edited

Loading

irishgordo commented Jul 31, 2023

ibrokethecloud commented Jul 31, 2023

khushboo-rancher commented Mar 21, 2024

khushboo-rancher commented Jul 6, 2024

irishgordo commented Jul 9, 2024

[e2e] [FEATURE] VM Import/Migration #522

[e2e] [FEATURE] VM Import/Migration #522

Comments

harvesterhci-io-github-bot commented Sep 20, 2022 • edited by irishgordo Loading

What's the test to develop? Please describe

Prerequisite and dependency of test

Describe the items of the test development (DoD, definition of done) you'd like

Additional context

irishgordo commented May 5, 2023 • edited Loading

irishgordo commented Jul 27, 2023 • edited Loading

irishgordo commented Jul 27, 2023 • edited Loading

irishgordo commented Jul 27, 2023

irishgordo commented Jul 27, 2023 • edited Loading

irishgordo commented Jul 31, 2023

ibrokethecloud commented Jul 31, 2023

khushboo-rancher commented Mar 21, 2024

khushboo-rancher commented Jul 6, 2024

irishgordo commented Jul 9, 2024

harvesterhci-io-github-bot commented Sep 20, 2022 •

edited by irishgordo

Loading

irishgordo commented May 5, 2023 •

edited

Loading

irishgordo commented Jul 27, 2023 •

edited

Loading

irishgordo commented Jul 27, 2023 •

edited

Loading

irishgordo commented Jul 27, 2023 •

edited

Loading