Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Changes and bug fixes to support shared clusters in DBR 14.2 #248

Merged
merged 64 commits into from
Feb 22, 2024
Merged

Changes and bug fixes to support shared clusters in DBR 14.2 #248

merged 64 commits into from
Feb 22, 2024

Conversation

ronanstokes-db
Copy link
Contributor

Proposed changes

When using a shared cluster with UC shared mode in DBR 14.2, referencing the sparkContext produces an attribute error.
To fix this, if an error is thrown when accessing the sparkContext, we use a default parallelism of 200 unless an alternative
value was explicitly specified.

This PR also uses the Spark SQL function element_at rather than direct array indexing due to incompatibilities with some spark versions.

Types of changes

What types of changes does your code introduce to dbldatagen?
Put an x in the boxes that apply

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Change to tutorials, tests or examples
  • Non code change (readme, images or other non-code assets)
  • Documentation Update (if none of the other choices apply)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR.
If you're unsure about any of them, don't hesitate to ask. We're here to help!
This is simply a reminder of what we are going to look for before merging your code.

  • Lint and unit tests pass locally with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)
  • Any dependent changes have been merged and published in downstream modules
  • Submission does not reduce code coverage numbers
  • Submission does not increase alerts or messages from prospector / lint

Further comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you
did and what alternatives you considered, etc...

@ronanstokes-db
wip
aebbb06
@ronanstokes-db
merge from origin
5f0ffc0
@ronanstokes-db
wip
1eda552
@ronanstokes-db
Merge branch 'master' of https://github.com/databrickslabs/dbldatagen
7de014c 
merged from remote
@ronanstokes-db
Merge branch 'master' of https://github.com/databrickslabs/dbldatagen
c859475 
merged of changes from remote
@ronanstokes-db
Merge branch 'master' of https://github.com/databrickslabs/dbldatagen
3094e96 
merged in upstream changes
@ronanstokes-db
Merge branch 'master' of https://github.com/databrickslabs/dbldatagen
3bf6e9b 
merge from remote
@ronanstokes-db
Merge branch 'master' of https://github.com/databrickslabs/dbldatagen
caaff18 
merge from remote
@ronanstokes-db
Merge branch 'master' of https://github.com/databrickslabs/dbldatagen
87d5c50 
merge from remote
@ronanstokes-db
Merge branch 'master' of https://github.com/databrickslabs/dbldatagen
4536794 
merge from upstream
@ronanstokes-db
Merge branch 'master' of https://github.com/databrickslabs/dbldatagen
eba6193 
merge from remote
@ronanstokes-db
wip
c4fdc3b
@ronanstokes-db
Merge branch 'master' of https://github.com/databrickslabs/dbldatagen
8734b19 
merge from remote commit
@ronanstokes-db
Merge branch 'master' of https://github.com/databrickslabs/dbldatagen
f063235 
merge from master
@ronanstokes-db
Merge branch 'master' of https://github.com/databrickslabs/dbldatagen
b9fb552 
merge from remote
@ronanstokes-db
Merge branch 'master' of https://github.com/databrickslabs/dbldatagen
3eb15f4 
merge from upstream
@ronanstokes-db
Merge branch 'master' of https://github.com/databrickslabs/dbldatagen
c85f915 
merge from remote
@ronanstokes-db
changes for release
e53f8fe
@ronanstokes-db
Merge branch 'master' of https://github.com/databrickslabs/dbldatagen
4259cac 
update from remote
@ronanstokes-db
example notebook
fef24fa
@ronanstokes-db
Merge branch 'master' of https://github.com/databrickslabs/dbldatagen
35e2de4 
merge from remote
@ronanstokes-db
Merge branch 'master' of https://github.com/databrickslabs/dbldatagen
ce1a12d 
merge from upstream
@ronanstokes-db
Merge branch 'master' of https://github.com/databrickslabs/dbldatagen
63c8e70 
merge from upstream
@ronanstokes-db
updates to handle shared spark session restrictions
3ac33c0
@ronanstokes-db
updates to handle shared sparkSession
c5036a6
@ronanstokes-db
updates to handle shared sparkSession
e735643
@ronanstokes-db
updates to handle shared sparkSession
ce83fe8
@ronanstokes-db
updates to handle shared sparkSession
81310b5
@ronanstokes-db ronanstokes-db requested a review from a team as a code owner December 18, 2023 08:46
@ronanstokes-db ronanstokes-db removed the request for review from a team December 18, 2023 08:46
ronanstokes-db and others added 26 commits February 16, 2024 17:08
@ronanstokes-db
Feature v34 (#201)
0eec26a 
* wip

* prep for release

* prep for release

* wip
@ronanstokes-db
Feature generate from existing data (#163)
240ef41 
* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip
@MarvinSchenkel @ronanstokes-db
Remove calls to root logger. (#205)
383da23
@ronanstokes-db
Release v34post1 (#206)
c95a995 
* wip

* hotfix release for logger fixes

* fix for root logger configuration
@ronanstokes-db
Fix doc typos and minor clarification (#207)
9b48095 
* wip

* wip

* wip
@ronanstokes-db
Feature issue 209 (#210)
4ce97cd 
* wip

* wip

* wip
@ronanstokes-db
Release 0v34post2 (#211)
571e4f2 
* wip

* wip
@ronanstokes-db
Feature html formatting (#208)
318fcfc 
* wip

* wip

* wip

* wip

* wip
@ronanstokes-db
wip
f935c63
@ronanstokes-db
Build fixes (#213)
c01baa3 
* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* updated build version

* updated build version
@ronanstokes-db
Feature doc change generating text (#218)
bc2bd57 
* wip

* wip

* wip

* updated running of prospector
@ronanstokes-db
Feature build update (#220)
b2e09ca 
* wip

* wip

* updates to handle changes in upstream pipenv package

* updated build version
@ronanstokes-db
Feature struct changes (#219)
9b0847b 
* wip

* wip

* added support for inferred types

* added support for inferred types

* wip

* wip

* wip

* updates and fixes to unit tests

* wip

* updated pipfile due to upstream changes in pipenv

* additional tests

* wip

* wip

* wip

* wip

* removed old code
@ronanstokes-db
Feature additional docs (#222)
1dc6606 
* wip

* wip

* doc updates

* function doc changes only

* function doc changes only
@ronanstokes-db
changes for release
0de8665
@ronanstokes-db
example notebook
7dca740
@ronanstokes-db
Feature readme updates - updates readme to note compatible Unity Cata…
fe6865e 
…log releases (#236)

* wip

* wip

* changes for release

* example notebook

* updated readme to note compatible unity catalog runtimes

* updated readme to note compatible unity catalog runtimes

* updated readme to note compatible unity catalog runtimes

* updated readme to note compatible unity catalog runtimes
@ronanstokes-db
Feature add codeowners (#238)
23d5db9
@nfx @ronanstokes-db
Test assign (#239)
aa772f0
@ronanstokes-db
updates to handle shared spark session restrictions
c830d20
@nfx @ronanstokes-db
Update LICENSE (#246)
a482861
@ronanstokes-db
updates to handle shared sparkSession
ed989ed
@ronanstokes-db
changes per code review
a59df0e
@ronanstokes-db
Merge branch 'feature_shared_support' of https://github.com/databrick…
3c55687 
…slabs/dbldatagen into feature_shared_support

Merged from upstream
@ronanstokes-db
Merge branch 'master' of https://github.com/databrickslabs/dbldatagen
64ed5de
@ronanstokes-db
Merge branch 'master' into feature_shared_support
22925fd 
merge changes from master
@GeekSheikh
Copy link

It's working for me. We're testing it a little deeper but looks good. Please don't forget to bump the version.

GeekSheikh
GeekSheikh approved these changes Feb 19, 2024
View reviewed changes
Copy link

@GeekSheikh GeekSheikh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please don't forget to bump the version but looks to be working on 14.3 LTS.

@ronanstokes-db ronanstokes-db merged commit b0d23e9 into master Feb 22, 2024
5 checks passed
@ronanstokes-db ronanstokes-db deleted the feature_shared_support branch May 24, 2024 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@GeekSheikh GeekSheikh GeekSheikh approved these changes

@howardwu-db howardwu-db Awaiting requested review from howardwu-db

Assignees

@ronanstokes-db ronanstokes-db

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@ronanstokes-db @GeekSheikh @howardwu-db @MarvinSchenkel @nfx