Recalculate checksums #2
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Taking inspiration from Flyway, this commits adds the undo subcommand to the application, with the `--step <n>` flag. Undo scripts will have the `U` prefix, with the same name and version as their corresponding versioned script. It will attempt to undo <n> amount of versioned scripts until it finishes, or no corresponding undo script is found. Under the hood, the application is removing the `V` script from the change history table, so when the `deploy` command is ran, the versioned script can be applied again. Algorithm: * Fetch all applied `V` migrations * Fetch all `U` migrations * Get `U` migration of last `V` migration. * IF `V` migration has no `U` migration, process stops * Apply `U` script * Delete `V` script from the change_history_table related to `U` This commit closes issue Snowflake-Labs#19
riiwo
force-pushed
the
recalculate-checksums
branch
4 times, most recently
from
da59b37
to
a7ee0d0
Compare
riiwo
force-pushed
the
recalculate-checksums
branch
6 times, most recently
from
90395c8
to
5c4b02c
Compare
[Swithing to Debian 12](salemove/base-images#46) in python base image caused [OpenSSL switch](salemove/base-images#46 (comment)) from 1.1.1 to 3.0. Current version of `snowflake-connector-python` 2.8.0 which depends on `pyopenssl` 22.1.0 fails because of OpenSSL switch https://ci.at.samo.io/job/salemove/job/snowflake-ingestion/job/PR-589/8/consoleFull After the update of `snowflake-connector-python` to 3.7.0 `pyopenssl` becomes `23.3.0` and doesn't fail anymore in this image. Tested `snowflake-ingestion` in [this PR](salemove/snowflake-ingestion#600). [Pipeline](https://ci.at.samo.io/job/salemove/job/snowflake-ingestion/job/PR-600/6/consoleFull) doesn't fail anymore, and does migration/rollback successfully. CBRO-1908
riiwo
force-pushed
the
recalculate-checksums
branch
2 times, most recently
from
ec0d14e
to
62264a1
Compare
alcidesteixeira
approved these changes
Feb 9, 2024
| @@ -31,7 +35,7 @@ All notable changes to this project will be documented in this file. | |||
| - Cleaned up argument passing and other repetitive code using dictionary and set comparisons for easy maintenance. (Converted variable names to a consistent snake_case from a mix of kebab-case and snake_case) | |||
| - Fixed change history table processing to allow mixed case names when '"' are used in the name. | |||
| - Moved most error, log and warning messages and query strings to global or class variables. | |||
| - Updated readme to cover new authentication methods | |||
| - Updated readme to cover new authentication methods | |||
There was a problem hiding this comment.
Please rebase from CBRO-1218-undo-migrations and merge this PR into it. To later merge that branch into master.
The top commit from the branch already removes trailing whitespaces and tabs 94deeb5.
It will be easier to review this PR if it has your changes only.
Also feels unnecessary to write a bunch of logic in recalculate_checksum_command only to split it in the next commit, why not to split it initially?
There was a problem hiding this comment.
why not to split it initially
One was feature, other was refactoring
There was a problem hiding this comment.
anyways, it is rebased. I'll still point to master. You can skip first 3 commits
riiwo
force-pushed
the
recalculate-checksums
branch
2 times, most recently
from
ed73f27
to
ce5a9c2
Compare
In case you clone a database, you would want to recaclulate checksums for repeatable scripts as they already exist but checksum is wrong as database name has changed. Both deploy_command and recalculate_checksum use very similar body. Only differences are: 1) Deploy deals with V, R, A migrations, while recalculation deals only with R migrations. 2) Deploy also applies the script, while recalculation on updates schemachangetables.
riiwo
force-pushed
the
recalculate-checksums
branch
from
ce5a9c2
to
5fd538d
Compare
BondarenkoStas
approved these changes
Feb 9, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We have diverged from the original repo and it does not make sense anymore for us to try keeping up to date with the original.
Why? Because we rely on
undofunctionality that was not accepted to the schemachange: Snowflake-Labs#158