add test mode to update.sh

[spacklePatch]

NetBox version

3.7.0

Feature type

New functionality

Proposed functionality

During the upgrade from a previous version, when upgrade.sh encounters unexpected data in the database, especially with old version data (we started with 2.x and upgraded to 3.3.7, now going to 3.7.0), an error it immediately stops. This is truly painful trying to revert and fix the errors in the data and try again, when we have no idea how many errors in data will be encountered.

Old data error example 1:
django.db.utils.IntegrityError: could not create unique index "dcim_device_unique_name_site"
DETAIL: Key (lower(name::text), site_id)=(cable mgt, 2955) is duplicated.

The above data used to allow case sensitive names of devices, but have the same characters. The is now longer the case, since the above Key contains (lower) method. this happened about 50 times, and would only fail on the 1st time.

Old data error example 2:
raise DataError(
django.db.utils.DataError: Unable to proceed with deleting asn field from Provider model: Found 1 providers with legacy ASN data. Please ensure all legacy provider ASN data has been migrated to ASN objects before proceeding. Or, set the NETBOX_DELETE_LEGACY_DATA environment variable to bypass this safeguard and delete all legacy provider ASN data.

This is just annoying and could be caught before hand with a Test mode.

Use case

upgrade.sh -Test would allow the upgraded to run in pretend mode and we can fix all errors encountered at once. Then run the upgrade in real mode 1 time with no errors,

Database changes

No response

External dependencies

No response

[jeremystretch]

This is just annoying and could be caught before hand with a Test mode.

You haven't included any detail about your proposed implementation. What specifically would a "test mode" do?

[spacklePatch]

upgrade.sh -Test would allow the upgraded to run in pretend mode and we can fix all errors encountered at once. Then run the upgrade in real mode 1 time with no errors,

[jeremystretch]

upgrade.sh is just a shell script that calls individual commands. What is the specific set of commands you're proposing it runs to effect "pretend mode?"

[spacklePatch]

Honestly, I don't know exactly what is required to make this happen. It may not be part of upgrade.sh. I'm running Netbox and using it, but I am not programming it. This is a feature I'm requesting be added to the upgrade process.

[jeremystretch]

Ok, I'm going to convert this to a discussion. Maybe someone else can weigh in to help you work out the specifics.

[candlerb]

I don't think this can work in general. upgrade.sh applies a sequence of changes, and later changes are dependent on earlier ones (e.g. data changes are dependent on the schema changes having been applied), so you can't simply skip applying the previous changes.

"Test mode" would have to have custom code written to search for all types of anomaly in the old database, which might impact migration to the new database.

For the example problem around dcim_device_unique_name_site, someone would have to write a SELECT statement which identifies all cases where group by lower(name) has count >= 2, and reports on the issues found. The automatically detected IntegrityError (from Postgres) only reports on the first one found.