Upgrade this lib to be compatible with Spark Connect / DB Connect

[MrPowers]

Expected Behavior

This library works the same with Spark Connect.

Current Behavior

This library uses sparkSession.sparkContext which doesn't work with Spark Connect, here is an example:

dbldatagen/dbldatagen/data_generator.py

Line 251 in debb29f

if sparkSession.sparkContext is not None:

. This actually might work cause the exception would be caught, but you get the idea.

Steps to Reproduce (for bugs)

Run the test suite with Spark Connect enabled and fix all issues.

[ronanstokes-db]

We recently released an update to deal with situations where the spark context is not available to query things like default parallelism. This should address this

In general, the way to safeguard against this is to explicitly specify the number of partitions requested when generating the specification for your dataset. This will avoid the query against the sparkContext.

While we have not tested against Spark Connect, we have tested against other environments where there is no sparkContext available