SparkSQLCompare does not work on Databricks serverless

[scardella]

Got this error when running on serverless:

[NOT_SUPPORTED_WITH_SERVERLESS] PERSIST TABLE is not supported on serverless compute. SQLSTATE: 0A000
File , line 1
----> 1 compare = SparkSQLCompare(
2 spark,
3 old,
4 new,
5 join_columns=['CurrencyExchangeRateDate','SourceCurrencyCode', 'TargetCurrencyCode'], # You can also specify a list of columns
6 df1_name='old', # Optional, defaults to 'df1'
7 df2_name='new' # Optional, defaults to 'df2'
8 )
10 # This method prints out a human-readable report summarizing and sampling differences
11 print(compare.report())

[fdosani]

@scardella Thanks for the issue. I believe this was a known issue with databricks and its serverless option: https://docs.databricks.com/aws/en/compute/serverless/limitations

Given it doesn't support the full suite of options I'm not sure this is something we can support or tweak. Maybe there is a case for a special databricks comparer? I don't have databricks access so I can't really test this out.

[scardella]

Given the issue, all you really need is to eliminate some cache() or persist() type statements. OR give the end user the capacity to toggle that.

[fdosani]

Given the issue, all you really need is to eliminate some cache() or persist() type statements. OR give the end user the capacity to toggle that.

If you are able to test this out on your end and submit a PR I'd be able to consider/review that. Like I mentioned I don't have databricks serverless access.

[Cheffress]

Thanks for DataCompy - a great tool.
As of September 2025 the free version of Databricks uses serverless compute, so you could probably test it there.