[SPARK-53441][PS] Bools |/&/^ None should fail under ANSI

[xinrong-meng]

What changes were proposed in this pull request?

Bools |/&/^ None should fail under ANSI, following native pandas

For example,

>>> pd.Series([True, False]) | None
Traceback (most recent call last):
...
TypeError: unsupported operand type(s) for |: 'bool' and 'NoneType'

but under ANSI

>>> ps.Series([True, False]) | None
0    False                                                                      
1    False
dtype: bool

Why are the changes needed?

Part of https://issues.apache.org/jira/browse/SPARK-53389

Does this PR introduce any user-facing change?

No, the feature hasn't been released yet.

Now bools |/&/^ None fails under ANSI, e.g.

>>> ps.Series([True, False]) | None
Traceback (most recent call last):
...
TypeError: OR can not be applied to given types.

How was this patch tested?

Unit tests

Commands below passed:

 1027  SPARK_ANSI_SQL_MODE=true ./python/run-tests --python-executables=python3.11 --testnames "pyspark.pandas.tests.data_type_ops.test_boolean_ops BooleanOpsTests"
 1028  SPARK_ANSI_SQL_MODE=false ./python/run-tests --python-executables=python3.11 --testnames "pyspark.pandas.tests.data_type_ops.test_boolean_ops BooleanOpsTests"

Was this patch authored or co-authored using generative AI tooling?

No

[xinrong-meng]

We need to check self.dtype == bool because

>>> pd.Series([True, False, np.nan]) | None
0    False
1    False
2    False

>>> ps.Series([True, False, np.nan]).dtype
dtype('O')
>>> ps.Series([True, False, np.nan]).spark.data_type
BooleanType()

BooleanType() will direct the code path here, but we want to support (instead of TypeError) this case, so we check dtype explicitly

[xinrong-meng]

Tested here https://github.com/apache/spark/blob/master/python/pyspark/pandas/tests/series/test_series.py#L262

[xinrong-meng]

@HyukjinKwon @ueshin @zhengruifeng may I get a review please?

[xinrong-meng]

Merged to master, thank you!