Skip to content

[AN-257] Update mysql version to 5.7 #4804

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
LizBaldo merged 14 commits into from
Dec 4, 2024
Merged

[AN-257] Update mysql version to 5.7 #4804

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
14a3b85
add allowpublickeyretrival to url, modify liquibase timestamp default…
LizBaldo Nov 26, 2024
ada3248
use mysql image and update to 8.4
LizBaldo Nov 26, 2024
f1e3eab
Merge branch 'develop' into AN-257-mysql-upgrade-to-8.0
LizBaldo Nov 26, 2024
5c33cea
apply unique constraint before creation of the foreign key
LizBaldo Nov 26, 2024
2f75cc9
this is silly but worth trying
LizBaldo Nov 26, 2024
0569098
try to set the --restrict-fk-on-non-standard-key to OFF
LizBaldo Nov 26, 2024
956aa48
try again
LizBaldo Nov 26, 2024
b4e5638
last try
LizBaldo Nov 26, 2024
739544d
add validchecsum for the label table as well
LizBaldo Dec 3, 2024
2db4a05
test backward compatibility with 5.6
LizBaldo Dec 4, 2024
7fb7b59
back to 8.4
LizBaldo Dec 4, 2024
071192a
only update to 5.7 for now
LizBaldo Dec 4, 2024
7f3e7c4
specify default timestamps for persistent disk table
LizBaldo Dec 4, 2024
7607506
add validchecksum fo persistent disk table
LizBaldo Dec 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/unit_test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:

services:
mysql:
image: mysql/mysql-server:5.6
image: mysql:5.7
env:
MYSQL_ROOT_PASSWORD: leonardo-test
MYSQL_USER: leonardo-test
Expand Down
6 changes: 3 additions & 3 deletions docker/run-mysql.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/usr/bin/env bash

# The CloudSQL console simply states "MySQL 5.6" so we may not match the minor version number
MYSQL_VERSION=5.6
# The CloudSQL console simply states "MySQL 5.7" so we may not match the minor version number
MYSQL_VERSION=5.7
start() {

echo "attempting to remove old $CONTAINER container..."
Expand All @@ -17,7 +17,7 @@ start() {
-e MYSQL_DATABASE=leotestdb \
-d \
-p 3311:3306 \
mysql/mysql-server:$MYSQL_VERSION
mysql:$MYSQL_VERSION

# validate mysql
echo "running mysql validation..."
Expand Down
7 changes: 6 additions & 1 deletion ...ute/dsde/workbench/leonardo/liquibase/changesets/20170901_cluster-createdDate-default.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,12 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<databaseChangeLog logicalFilePath="leonardo" xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.4.xsd">
<changeSet logicalFilePath="leonardo" author="kumra" id="cluster-createdDate-default">
<validCheckSum>8:c1023fe139609a1cab82cda7caededc0</validCheckSum>
<comment>
Mysql 5.7+ cannot add a not null constraint with a default value simultaneously. This changeSet has been modified to reflect that; the
validCheckSum is the checksum from when the table was not altered, but with addNotNullConstraint and addDefaultValue.
</comment>
<sql dbms="mysql">ALTER TABLE CLUSTER
CHANGE createdDate createdDate TIMESTAMP(6) NOT NULL DEFAULT 0</sql>
CHANGE createdDate createdDate TIMESTAMP(6) NOT NULL DEFAULT "1970-01-01 00:00:01.000000"</sql>
</changeSet>
</databaseChangeLog>
16 changes: 8 additions & 8 deletions .../dsde/workbench/leonardo/liquibase/changesets/20171011_cluster_destroyedDate_not_null.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<databaseChangeLog logicalFilePath="leonardo" xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.4.xsd">
<changeSet logicalFilePath="leonardo" author="vkumra" id="cluster_destroyeddate_not_null">
<addNotNullConstraint columnDataType="TIMESTAMP(6)"
columnName="destroyedDate"
defaultNullValue="1970-01-01 00:00:01.000000"
tableName="CLUSTER"/>
<addDefaultValue columnDataType="TIMESTAMP(6)"
columnName="destroyedDate"
defaultValue="1970-01-01 00:00:01.000000"
tableName="CLUSTER"/>
<validCheckSum>8:29280ede1de44e4c3f2966c6922e9865</validCheckSum>
<comment>
Mysql 5.7+ changed default time stamps. This changeSet has been modified to reflect that; the
validCheckSum is the checksum from when the default was 0.
</comment>
<sql>
ALTER TABLE CLUSTER MODIFY COLUMN destroyedDate TIMESTAMP(6) NOT NULL DEFAULT '1970-01-01 00:00:01.000000';
</sql>
</changeSet>
</databaseChangeLog>
9 changes: 6 additions & 3 deletions ...dinstitute/dsde/workbench/leonardo/liquibase/changesets/20180515_cluster_dateaccessed.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,11 @@
</changeSet>

<changeSet id="make dateAccessed not null" author="vkumra">
<addNotNullConstraint columnDataType="TIMESTAMP(6)"
columnName="dateAccessed"
tableName="CLUSTER"/>
<validCheckSum>8:5017000ff29d7873815c1bdb69895225</validCheckSum>
<comment>
Mysql 5.7+ cannot add a not null constraint with a default value simultaneously. This changeSet has been modified to reflect that; the
validCheckSum is the checksum from when the table was not altered, but with addNotNullConstraint and addDefaultValue.
</comment>
<sql>ALTER TABLE CLUSTER MODIFY COLUMN dateAccessed TIMESTAMP(6) NOT NULL DEFAULT '1970-01-01 00:00:01.000000'</sql>
</changeSet>
</databaseChangeLog>
11 changes: 8 additions & 3 deletions ...te/dsde/workbench/leonardo/liquibase/changesets/20200427_create_persistent_disk_table.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<databaseChangeLog logicalFilePath="leonardo" xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.4.xsd">
<changeSet logicalFilePath="leonardo" author="rtitle" id="persistent-disk-table">
<validCheckSum>8:6e2afa6af8c562abad1e253e74d79817</validCheckSum>
<comment>
Mysql 5.7+ changed default time stamps. This changeSet has been modified to reflect that; the
validCheckSum is the checksum from when the default was not set.
</comment>
<createTable tableName="PERSISTENT_DISK">
<column name="id" type="BIGINT" autoIncrement="true">
<constraints primaryKey="true"/>
Expand All @@ -26,13 +31,13 @@
<column name="creator" type="VARCHAR(255)">
<constraints nullable="false"/>
</column>
<column name="createdDate" type="TIMESTAMP(6)">
<column name="createdDate" type="TIMESTAMP(6)" defaultValue="1970-01-01 00:00:01.000000">
<constraints nullable="false"/>
</column>
<column name="destroyedDate" type="TIMESTAMP(6)">
<column name="destroyedDate" type="TIMESTAMP(6)" defaultValue="1970-01-01 00:00:01.000000">
<constraints nullable="false"/>
</column>
<column name="dateAccessed" type="TIMESTAMP(6)">
<column name="dateAccessed" type="TIMESTAMP(6)" defaultValue="1970-01-01 00:00:01.000000">
<constraints nullable="false"/>
</column>
<column name="sizeGb" type="INT">
Expand Down
18 changes: 14 additions & 4 deletions ...roadinstitute/dsde/workbench/leonardo/liquibase/changesets/20200501_kubernetes_tables.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@
<databaseChangeLog logicalFilePath="leonardo" xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.4.xsd">

<changeSet logicalFilePath="leonardo" author="jcanas" id="kubernetes_cluster_table">
<validCheckSum>8:599d13c602bc3a23e12dc577a848456f</validCheckSum>
<comment>
Mysql 5.7+ changed default time stamps. This changeSet has been modified to reflect that; the
validCheckSum is the checksum from when the default was 0000-00-00 00:00:00.000000.
</comment>
<createTable tableName="KUBERNETES_CLUSTER">
<column name="id" type="BIGINT" autoIncrement="true">
<constraints primaryKey="true"/>
Expand All @@ -27,13 +32,13 @@
<column name="creator" type="VARCHAR(254)">
<constraints nullable="false"/>
</column>
<column name="createdDate" type="TIMESTAMP(6)" defaultValue="0000-00-00 00:00:00.000000">
<column name="createdDate" type="TIMESTAMP(6)" defaultValue="1970-01-01 00:00:01.000000">
<constraints nullable="false"/>
</column>
<column name="destroyedDate" type="TIMESTAMP(6)" defaultValue="1970-01-01 00:00:01.000000">
<constraints nullable="false"/>
</column>
<column name="dateAccessed" type="TIMESTAMP(6)" defaultValue="0000-00-00 00:00:00.000000">
<column name="dateAccessed" type="TIMESTAMP(6)" defaultValue="1970-01-01 00:00:01.000000">
<constraints nullable="false"/>
</column>
<column name="apiServerIp" type="VARCHAR(254)"/>
Expand All @@ -45,6 +50,11 @@
</changeSet>

<changeSet logicalFilePath="leonardo" id="nodepool_table" author="jcanas">
<validCheckSum>8:854fccdae58bc9906a94269397f04ae3</validCheckSum>
<comment>
Mysql 5.7+ changed default time stamps. This changeSet has been modified to reflect that; the
validCheckSum is the checksum from when the default was 0000-00-00 00:00:00.000000.
</comment>
<createTable tableName="NODEPOOL">
<column name="id" type="BIGINT" autoIncrement="true">
<constraints primaryKey="true"/>
Expand All @@ -61,13 +71,13 @@
<column name="creator" type="VARCHAR(254)">
<constraints nullable="false"/>
</column>
<column name="createdDate" type="TIMESTAMP(6)" defaultValue="0000-00-00 00:00:00.000000">
<column name="createdDate" type="TIMESTAMP(6)" defaultValue="1970-01-01 00:00:01.000000">
<constraints nullable="false"/>
</column>
<column name="destroyedDate" type="TIMESTAMP(6)" defaultValue="1970-01-01 00:00:01.000000">
<constraints nullable="false"/>
</column>
<column name="dateAccessed" type="TIMESTAMP(6)" defaultValue="0000-00-00 00:00:00.000000">
<column name="dateAccessed" type="TIMESTAMP(6)" defaultValue="1970-01-01 00:00:01.000000">
<constraints nullable="false"/>
</column>
<column name="machineType" type="VARCHAR(254)">
Expand Down
9 changes: 7 additions & 2 deletions ...adinstitute/dsde/workbench/leonardo/liquibase/changesets/20200520_kubernetes_refactor.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,11 @@
</changeSet>

<changeSet logicalFilePath="leonardo" id="kubernetes_add_app_tables" author="jcanas">
<validCheckSum>8:a2cec6057088736af10dcdc3c2dd3091</validCheckSum>
<comment>
Mysql 5.7+ changed default time stamps. This changeSet has been modified to reflect that; the
validCheckSum is the checksum from when the default was 0000-00-00 00:00:00.000000.
</comment>
<createTable tableName="APP">
<column name="id" type="BIGINT" autoIncrement="true">
<constraints primaryKey="true"/>
Expand All @@ -41,13 +46,13 @@
<column name="creator" type="VARCHAR(254)">
<constraints nullable="false"/>
</column>
<column name="createdDate" type="TIMESTAMP(6)" defaultValue="0000-00-00 00:00:00.000000">
<column name="createdDate" type="TIMESTAMP(6)" defaultValue="1970-01-01 00:00:01.000000">
<constraints nullable="false"/>
</column>
<column name="destroyedDate" type="TIMESTAMP(6)" defaultValue="1970-01-01 00:00:01.000000">
<constraints nullable="false"/>
</column>
<column name="dateAccessed" type="TIMESTAMP(6)" defaultValue="0000-00-00 00:00:00.000000">
<column name="dateAccessed" type="TIMESTAMP(6)" defaultValue="1970-01-01 00:00:01.000000">
<constraints nullable="false"/>
</column>
<column name="namespaceId" type="BIGINT">
Expand Down
7 changes: 6 additions & 1 deletion ...nstitute/dsde/workbench/leonardo/liquibase/changesets/20230911_create_app_usage_table.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,19 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<databaseChangeLog logicalFilePath="leonardo" xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.4.xsd">
<changeSet logicalFilePath="leonardo" author="qi" id="qi_create_app_usage_table">
<validCheckSum>8:10d891c4c811215803753f8b060621be</validCheckSum>
<comment>
Mysql 5.7+ changed default time stamps. This changeSet has been modified to reflect that; the
validCheckSum is the checksum from when the default was 0000-00-00 00:00:00.000000.
</comment>
<createTable tableName="APP_USAGE">
<column name="id" type="BIGINT" autoIncrement="true">
<constraints primaryKey="true"/>
</column>
<column name="appId" type="BIGINT(20)">
<constraints nullable="false"/>
</column>
<column name="startTime" type="TIMESTAMP(6)" defaultValue="0000-00-00 00:00:00.000000">
<column name="startTime" type="TIMESTAMP(6)" defaultValue="1970-01-01 00:00:01.000000">
<constraints nullable="false"/>
</column>
<column name="stopTime" type="TIMESTAMP(6)" defaultValue="1970-01-01 00:00:01.000000">
Expand Down
2 changes: 1 addition & 1 deletion http/src/test/resources/reference.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ mysql {
port = 3311
db {
driver = "com.mysql.cj.jdbc.Driver"
url = "jdbc:mysql://"${mysql.host}":"${mysql.port}"/leotestdb?createDatabaseIfNotExist=true&useSSL=false&rewriteBatchedStatements=true&nullNamePatternMatchesAll=true"
url = "jdbc:mysql://"${mysql.host}":"${mysql.port}"/leotestdb?createDatabaseIfNotExist=true&allowPublicKeyRetrieval=true&useSSL=false&rewriteBatchedStatements=true&nullNamePatternMatchesAll=true"
Copy link
Contributor

@aednichols aednichols Nov 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would love to enable SSL everywhere and disable allowPublicKeyRetrieval but that seems like a project for another PR.

Copy link
Collaborator Author

@LizBaldo LizBaldo Nov 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I checked that this is only used for our unit testing at least

user = "leonardo-test"
password = "leonardo-test"
connectionTimeout = "5 seconds" // default of 1 sec sometimes too short for docker local mysql
Expand Down
2 changes: 1 addition & 1 deletion leonardo-example.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ dataproc {
# Database connection information
mysql {
db {
url = "jdbc:mysql://YOUR_DB_HOST/leonardo?requireSSL=true&useSSL=true&rewriteBatchedStatements=true&nullNamePatternMatchesAll=true"
url = "jdbc:mysql://YOUR_DB_HOST/leonardo?allowPublicKeyRetrieval=true&useSSL=false&rewriteBatchedStatements=true&nullNamePatternMatchesAll=true"
user = "USER_NAME"
password = "PASSWORD"
}
Expand Down
Loading