Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Would it be possible to not specify the location so it inherits whatever is configured in the glue database location? #712

Open
1 task done
maduxi opened this issue Sep 4, 2024 · 3 comments
Labels
enhancement New feature or request

Comments

@maduxi
Copy link

maduxi commented Sep 4, 2024

Is this your first time submitting a feature request?

  • I have searched the existing issues, and I could not find an existing issue for this feature

Describe the feature

Currently there is a way to configure the table location based on several variables and strategies, but I couldn't find a way of not specifying it, so the Glue catalog would end up using the database location.

Describe alternatives you've considered

I'm using a workgroup without enforced output to control the location on dbt, but I couldn't find a way to not specify it.

Who will this benefit?

This would be interesting to decouple the location configuration from my dbt project. A benefit would be to point different schemas to different buckets.

Are you interested in contributing this feature?

It would be interesting, yes.

Anything else?

No response

@maduxi maduxi added the enhancement New feature or request label Sep 4, 2024
@nicor88
Copy link
Contributor

nicor88 commented Sep 4, 2024

@maduxi At the moment there is a possibility to not specify the s3 location, and it's using an athena workgroup with enforced parameters. Did you try that?

@maduxi
Copy link
Author

maduxi commented Sep 5, 2024

Thank you for the reply, but that would require me to use different workgroups if I want schemas to go to different locations, right?

@nicor88
Copy link
Contributor

nicor88 commented Sep 5, 2024

@maduxi yes correct.
also, the feature that you requested might lead to substantial change, because for each model we need to pass the glue database where the table is going to be written, if the database has a not empty location, then we return the s3 location from the db, ideally we can control this behavior via a flag.

Long story short, databases must be a parameter of generate_s3_location, and there is definitely some code to touch - overall your feature could make sense, but if you want to do it, I suggest to add another flag to configure such behavior: s3_data_dir_from_glue_db: true.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants