Migrations 1.10.3 - add rollout deleted flag failing with AWS Aurora MySQLv2

[japel]

Using hawkbit-update-server:latest-mysql and AWS Aurora MySQLv2 migrations fail.

The aurora cluster's engine is Aurora MySQL 3.04.1 (compatible with MySQL 8.0.28) - default for major version 8.0
I created the an empty database hawkbit with charset latin1 and collate latin1_bin

These are the logs I get:

Eclipse hawkBit Update Server (v0.4.1)
using Spring Boot (v2.7.18)
Go to https://www.eclipse.org/hawkbit for more information.

2024-01-30 20:04:57.902 INFO 1 --- [ main] org.eclipse.hawkbit.app.Start : Starting Start v0.4.1 using Java 17.0.9 on ip-10-0-1-238.ec2.internal with PID 1 (/BOOT-INF/classes started by hawkbit in /)
"2024-01-30 20:04:57.979 INFO 1 --- [ main] org.eclipse.hawkbit.app.Start : The following 1 profile is active: ""mysql"""
2024-01-30 20:05:08.079 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2024-01-30 20:05:09.395 INFO 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 1295 ms. Found 20 JPA repository interfaces.
"2024-01-30 20:05:12.289 INFO 1 --- [ main] faultConfiguringBeanFactoryPostProcessor : No bean named 'errorChannel' has been explicitly defined. Therefore
"2024-01-30 20:05:12.393 INFO 1 --- [ main] faultConfiguringBeanFactoryPostProcessor : No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore
2024-01-30 20:05:12.520 INFO 1 --- [ main] o.s.c.a.ConfigurationClassPostProcessor : Cannot enhance @configuration bean definition 'com.vaadin.spring.VaadinConfiguration' since its singleton instance has been created too early. The typical cause is a non-static @bean method with a BeanDefinitionRegistryPostProcessor return type: Consider declaring such methods as 'static'.
2024-01-30 20:05:13.904 INFO 1 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=5a9bc832-7c1b-3103-930b-e3c8a813b559
2024-01-30 20:05:15.176 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jdbc.DataSourceConfiguration$Hikari' of type [org.springframework.boot.autoconfigure.jdbc.DataSourceConfiguration$Hikari] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-01-30 20:05:15.192 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.commons.config.CommonsConfigAutoConfiguration' of type [org.springframework.cloud.commons.config.CommonsConfigAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-01-30 20:05:15.198 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.client.loadbalancer.LoadBalancerDefaultMappingsProviderAutoConfiguration' of type [org.springframework.cloud.client.loadbalancer.LoadBalancerDefaultMappingsProviderAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-01-30 20:05:15.277 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'loadBalancerClientsDefaultsMappingsProvider' of type [org.springframework.cloud.client.loadbalancer.LoadBalancerDefaultMappingsProviderAutoConfiguration$$Lambda$604/0x00007f677a4b6748] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-01-30 20:05:15.283 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'defaultsBindHandlerAdvisor' of type [org.springframework.cloud.commons.config.DefaultsBindHandlerAdvisor] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-01-30 20:05:15.295 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.flyway.FlywayAutoConfiguration' of type [org.springframework.boot.autoconfigure.flyway.FlywayAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-01-30 20:05:15.297 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'stringOrNumberMigrationVersionConverter' of type [org.springframework.boot.autoconfigure.flyway.FlywayAutoConfiguration$StringOrNumberToMigrationVersionConverter] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-01-30 20:05:15.386 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'spring.datasource-org.springframework.boot.autoconfigure.jdbc.DataSourceProperties' of type [org.springframework.boot.autoconfigure.jdbc.DataSourceProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-01-30 20:05:15.490 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'dataSource' of type [com.zaxxer.hikari.HikariDataSource] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-01-30 20:05:15.500 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'spring.jpa-org.springframework.boot.autoconfigure.orm.jpa.JpaProperties' of type [org.springframework.boot.autoconfigure.orm.jpa.JpaProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-01-30 20:05:15.785 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.eclipse.hawkbit.repository.jpa.RepositoryApplicationConfiguration' of type [org.eclipse.hawkbit.repository.jpa.RepositoryApplicationConfiguration$$EnhancerBySpringCGLIB$$a3819a03] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-01-30 20:05:17.186 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'UISecurityConfigurationAdapter.UIMethodSecurity' of type [org.eclipse.hawkbit.ui.autoconfigure.UISecurityConfigurationAdapter$UIMethodSecurity$$EnhancerBySpringCGLIB$$8cfb0f75] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-01-30 20:05:18.001 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.vaadin.spring.events.config.EventBusConfiguration' of type [org.vaadin.spring.events.config.EventBusConfiguration$$EnhancerBySpringCGLIB$$9ef10ea4] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-01-30 20:05:18.083 INFO 1 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.vaadin.spring.security.config.VaadinSharedSecurityConfiguration' of type [org.vaadin.spring.security.config.VaadinSharedSecurityConfiguration$$EnhancerBySpringCGLIB$$6ab0042b] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-01-30 20:05:21.191 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2024-01-30 20:05:21.203 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2024-01-30 20:05:21.276 INFO 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.83]
2024-01-30 20:05:21.798 INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2024-01-30 20:05:21.877 INFO 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 23376 ms
2024-01-30 20:05:23.488 INFO 1 --- [ main] o.f.c.internal.license.VersionPrinter : Flyway Community Edition 8.5.13 by Redgate
2024-01-30 20:05:23.489 INFO 1 --- [ main] o.f.c.internal.license.VersionPrinter : See what's new here: https://flywaydb.org/documentation/learnmore/releaseNotes#8.5.13
2024-01-30 20:05:23.489 INFO 1 --- [ main] o.f.c.internal.license.VersionPrinter :
2024-01-30 20:05:23.594 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2024-01-30 20:05:24.609 INFO 1 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
2024-01-30 20:05:24.782 INFO 1 --- [ main] o.f.c.i.database.base.BaseDatabaseType : Database: jdbc:mariadb://aurora-cluster.xyz.us-east-1.rds.amazonaws.com/hawkbit (MySQL 8.0)
2024-01-30 20:05:25.196 INFO 1 --- [ main] o.f.core.internal.command.DbValidate : Successfully validated 47 migrations (execution time 00:00.189s)
2024-01-30 20:05:25.305 INFO 1 --- [ main] o.f.c.i.s.JdbcTableSchemaHistory : Creating Schema History table hawkbit.schema_version ...
2024-01-30 20:05:25.524 INFO 1 --- [ main] o.f.core.internal.command.DbMigrate : Current version of schema hawkbit: << Empty Schema >>
"2024-01-30 20:05:25.885 INFO 1 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema hawkbit to version ""1.0.1 - init """
"2024-01-30 20:05:37.052 INFO 1 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema hawkbit to version ""1.2.0 - update target info for message """
"2024-01-30 20:05:37.475 INFO 1 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema hawkbit to version ""1.4.0 - cascade delete """
"2024-01-30 20:05:40.598 INFO 1 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema hawkbit to version ""1.4.1 - cascade delete """
"2024-01-30 20:05:40.888 INFO 1 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema hawkbit to version ""1.5.0 - target filter query """
"2024-01-30 20:05:41.189 INFO 1 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema hawkbit to version ""1.6.0 - rollout management """
"2024-01-30 20:05:44.764 INFO 1 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema hawkbit to version ""1.7.0 - swmType maxAssignment greater 0 """
"2024-01-30 20:05:44.862 INFO 1 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema hawkbit to version ""1.7.1 - reduce length enums """
"2024-01-30 20:05:46.263 INFO 1 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema hawkbit to version ""1.8.0 - auto assign ds filter """
"2024-01-30 20:05:46.591 INFO 1 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema hawkbit to version ""1.8.1 - cascade delete """
"2024-01-30 20:05:47.584 INFO 1 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema hawkbit to version ""1.8.2 - remove external artifact """
"2024-01-30 20:05:47.805 INFO 1 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema hawkbit to version ""1.9.0 - add rollout groups created """
"2024-01-30 20:05:47.903 INFO 1 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema hawkbit to version ""1.10.0 - advanced rolloutgroup """
"2024-01-30 20:05:48.164 INFO 1 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema hawkbit to version ""1.10.1 - consolidate artifact sha1 """
"2024-01-30 20:05:48.770 INFO 1 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema hawkbit to version ""1.10.2 - rollout auto start """
"2024-01-30 20:05:48.862 INFO 1 --- [ main] o.f.core.internal.command.DbMigrate : Migrating schema hawkbit to version ""1.10.3 - add rollout deleted flag """
2024-01-30 20:05:48.915 WARN 1 --- [ main] o.m.jdbc.message.server.ErrorPacket : Error: 1832-HY000: Cannot change column 'target': used in a foreign key constraint 'fk_targ_act_hist_targ'
"2024-01-30 20:05:48.929 ERROR 1 --- [ main] o.f.core.internal.command.DbMigrate : Migration of schema hawkbit to version ""1.10.3 - add rollout deleted flag "" failed! Please restore backups and roll back database and code!"
2024-01-30 20:05:48.977 ERROR 1 --- [ main] o.s.b.web.embedded.tomcat.TomcatStarter : Error starting Tomcat context. Exception: org.springframework.beans.factory.UnsatisfiedDependencyException. Message: Error creating bean with name 'org.eclipse.hawkbit.autoconfigure.security.SecurityManagedConfiguration$ControllerDownloadSecurityConfigurationAdapter': Unsatisfied dependency expressed through constructor parameter 0
'"-------------------------------------------------------------"
SQL State : HY000
Error Code : 1832
Message : (conn=13208) Cannot change column 'target': used in a foreign key constraint 'fk_targ_act_hist_targ'
Location : db/migration/MYSQL/V1_10_3__add_rollout_deleted_flag__MYSQL.sql (/file:/BOOT-INF/lib/hawkbit-repository-jpa-0.4.1.jar!/db/migration/MYSQL/V1_10_3__add_rollout_deleted_flag__MYSQL.sql)
Line : 5
Statement : ALTER TABLE sp_action MODIFY target BIGINT NOT NULL
2024-01-30 20:05:49.091 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
...

Is there any specific configuration needed for hawkbit to run with AWS aurora mysql compatible?

Thanks in advance for any clue.

[japel]

I thought this could be due to unsupported serverless aurora mysql compatible, but this is also happening on a normal mysql RDS instance.

Anyone any idea how to resolve this?

[japel]

anyone any clue?

[japel]

awkward enough, dumping a migrated database (locally) to a sql file and executing it on the RDS instance works perfectly, without any constraint violations.

[cybrox]

I just tried setting up HawkBit from zero again and run into the same issue, so it's not just you.

To test, I did this locally:

docker run --rm\
      -p 3306:3306 \
      -e MARIADB_ROOT_PASSWORD=password \
      -e MARIADB_USER=hawkbit \
      -e MARIADB_PASSWORD=hawkbit \
      -e MARIADB_DATABASE=hawkbit \
      --name hawkbit-mysql \
      mariadb

docker run --rm \
      -p 8080:8080 \
      -v "$(pwd)/hawkbittest/application.properties:/opt/hawkbit/application.properties:ro" \
      -e "SPRING_DATASOURCE_URL=jdbc:mariadb://192.168.1.xx:3306/hawkbit" \
      -e "SPRING_DATASOURCE_USERNAME=hawkbit" \
      -e "SPRING_DATASOURCE_PASSWORD=hawkbit" \
      --name hawkbit-update-server \
      hawkbit/hawkbit-update-server:0.3.0-mysql

This works fine.

Then I created a database and user in AWS RDS like so:

CREATE DATABASE hawkbit;
CREATE USER 'hawkbit'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON hawkbit.* to 'hawkbit';

and start HawkBit. This results in the same issue as you documented.

[swanmatch]

@cybrox I am in the same situation.
Users and permissions are set up correctly and migration seems to have been completed halfway through.
I would appreciate your continued investigation.

[avgustinmm]

Hi,
you could try with a docker image with mariadb driver 2.7.11 (instead of the 3.1.4 currently used in MySQL docker image) - because as mentioned in the link below - the newest versions of mariadb driver doesn't support amazon aurora.
Also, you could try with SPRING_DATASOURCE_URL jdbc:mariadb:aurora://mysql:3306/hawkbit.
In https://aws.amazon.com/blogs/database/using-the-mariadb-jdbc-driver-with-amazon-aurora-with-mysql-compatibility there are some recommendations regarding aurora db.