[BitSail] support load hive conf from specified location #390
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
garyli1019
commented
Feb 13, 2023
| @@ -117,6 +117,7 @@ private void addSinkConf(BitSailConfiguration jobConf) { | |||
| return; | |||
| } | |||
| if (HIVE_FORMAT_TYPE_VALUE.equalsIgnoreCase(formatType)) { | |||
| //TODO: whether to load from hive conf location? | |||
| Preconditions.checkState(jobConf.fieldExists(FileSystemSinkOptions.HIVE_METASTORE_PROPERTIES)); | |||
There was a problem hiding this comment.
@hk-lrzy not quite sure about this check, is this config a must have option here?
hk-lrzy
reviewed
Feb 13, 2023
| HiveConf hiveConf = HiveMetaClientUtil.getHiveConf(hiveProperties); | ||
| HiveConf hiveConf; | ||
| if (hiveConfLocation != null) { | ||
| hiveConf = HiveMetaClientUtil.getHiveConf(hiveConfLocation); |
There was a problem hiding this comment.
I think we can't get hive site file in here when we run job in yarn or k8s environment.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by:
Pre-Checklist
Note: Please complete ALL items in the following checklist.
Purpose
support load hive configuration file from a user defined location
Approaches
pass the location from
BitSailConfigurationto hive meta client builderRelated Issues
Close #193
New Behavior (screenshots if needed)
N/A