Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add collation support in profiles.yml / Fix incorrect collation for utf8mb4 #173

Merged

Conversation

kzajaczkowski
Copy link
Contributor

@kzajaczkowski kzajaczkowski commented Apr 25, 2024

resolves #172

Description

  1. Bumped mysql-connector-python to 8.1, because previous version was by default using incorrect collation for charset utf8mb4 which was utf8mb4_general_ci. The correct default one is utf8mb4_0900_ci_ai and 8.1 version uses it by default.
  2. Added support for overriding charset and collation via profiles.yml which is useful if someone wants to use different charsets and collations by default
  3. Adjusted readme with examples of new settings
  4. I've run all the tests and they passed

Checklist

  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md with information about my change

@kzajaczkowski kzajaczkowski marked this pull request as ready for review April 25, 2024 11:40
@maver1ck
Copy link

@kzajaczkowski
Could you please squash your commits ?

@kzajaczkowski
Copy link
Contributor Author

@kzajaczkowski Could you please squash your commits ?

Done, can you please recheck?

@mwallace582
Copy link
Collaborator

Thank you for the pull request! I just have a few small comments, but otherwise looks good. I like the idea of exposing this in the adapters configuration file.

setup.py Outdated Show resolved Hide resolved
README.md Show resolved Hide resolved
@mwallace582 mwallace582 added the ok to test This pull request can run integration tests label Apr 25, 2024
@kzajaczkowski
Copy link
Contributor Author

@mwallace582, thank you for your comments! I've accepted one suggestion, commented on the other one, and also pushed support for mariadb and mysql5 which were missing from the initial PR.

Pipelines are failing though. I've tried to recreate it on my local env and they fail for the main branch as well. So far I didn't manage to find the source of the issue. Any recommendations?

@mwallace582
Copy link
Collaborator

Thank you for making the fixes. I'll look into the test failures, at first glance they look unrelated to your changes.

@mwallace582
Copy link
Collaborator

I've fixed tests in #174. Would you mind re-targeting this pull request at 1.7.latest and merging with that branch? That should fix the tests.

@kzajaczkowski kzajaczkowski changed the base branch from main to 1.7.latest April 26, 2024 15:41
@mwallace582 mwallace582 merged commit 149157d into dbeatty10:1.7.latest Apr 26, 2024
10 checks passed
mwallace582 added a commit that referenced this pull request Apr 26, 2024
* Revert "Upgraded to dbt-core 1.4. (#146)"

This reverts commit 3db05eb.

* Fix version bumping

* Migration bump2version (unmaintained) to bump-my-version
* Import `bumpversion.cfg` from `dbt-snowflake`
* Convert `bumpversion.cfg` to `bumpversion.toml`
* Fix a few incorrect version labels

* Remove macos and windows as integration targets

* Fix requirements conflict between dbt 1.2 and tox

* Bump version to 1.3

* Upgraded to dbt-core 1.4. (#146)

* Upgraded to dbt-core 1.4.

* Updated CHANGELOG.

* Fixed policy fields definitions for mariadb and mysql5.

* Replaced deprecated dbt.exceptions.raise_compiler_error() with dbt.exceptions.CompilationError.

* Now using dbt.exceptions.DbtDatabaseError insead of dbt.exceptions.DatabaseException.

* Update version

* Update changelog

---------

Co-authored-by: Doug Beatty <[email protected]>

* Bump version to 1.5

* Update dbt-core to 1.5 and implement support for model contracts (#163)

Update dbt-core to 1.5.9 and implement support for constraints
* Update changelog
* Fix unit test error
     - The error was: `AttributeError: 'Namespace' object has no attribute 'MACRO_DEBUGGING'`
* Allow Unix socket connection rather than just TCP (#165)

* Support Black & MyPy pre-commit hooks (#167)

* Add black and mypy as pre-commit hooks

* Run black formatter on all files

* Add MyPy configuration & make tweaks and ignore errors to make MyPy pass

* Add .git-blame-ignore-revs to ignore `black` changes in git blame

* Update changelog

* Run black formatter on all files

* Add black commit to .git-blame-ignore-revs

* Run black formatter on all files

* Add black commit to .git-blame-ignore-revs

* Fix MyPy & black errors

* Make the `database` config field nullable again

* Fix mypy

* Bump version to 1.6

* Bump version to 1.7

* Fix import error

* Fix unicode decode error (#169)

* Update setup.py

Explicitly defining utf-8 encoding to avoid UnicodeDecodeError on setup.py in Windows environment

* Update CHANGELOG.md

---------

Co-authored-by: Scott Gunn <[email protected]>

* Pin jsonschema version to fix tests (#174)

* Add collation support in profiles.yml / Fix incorrect collation for utf8mb4 (#173)

* Add support for charset and collation in profile.yml

* Change mysql-connect-python version

* Update readme with new profile options

* Update setup.py with less restricting mysql-connector-python version after code review

Co-authored-by: Matthew Wallace <[email protected]>

* Add collation and charset support for mariadb and mysql5

---------

Co-authored-by: Matthew Wallace <[email protected]>

* Pin jsonschema version to fix tests

* Update black to resolve security alert

https://github.com/dbeatty10/dbt-mysql/security/dependabot/5

---------

Co-authored-by: Luke <[email protected]>
Co-authored-by: Doug Beatty <[email protected]>
Co-authored-by: Scott Gunn <[email protected]>
Co-authored-by: Krzysztof Zajączkowski <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ok to test This pull request can run integration tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Incorrect collation used for charset utf8mb4
3 participants