BUG: fix convert_dtypes to preserve timezone from tz-aware pyarrow timestamp dtype

[road-1]

Removed if statement blocks that were causing timezone to be removed, added a test to confirm that bug is fixed, and added BUG fix to 3.0.0 documentation.
Fixes issue in #60237

[jorisvandenbossche]

Thanks for looking into this! There are some test failures though (it seems that for older pyarrow versions, this now looses the datetime64 unit?)

[jorisvandenbossche]

By removing this, it will go through np.dtype(self.pyarrow_dtype.to_pandas_dtype()), which I think will raise a type error? (because to_pandas_dtype returns a DatetimeTZDtype, and np.dtype(..) does not recognize that) And so this will start to return object dtype instead of datetime64?

[jamesdow21]

For a bit of context from my linked issue, in pandas 2.0, an ArrowDtype for a pyarrow timestamp with a non-null timezone did return a numpy object dtype

The if statements that are removed in this pull request were added in pandas 2.1.0rc0 (#51800) to fix the other issue you pointed out with the datetime unit being lost in older pyarrow versions (fix for that was added in pyarrow 14 apache/arrow#35656). It wasn't noted in the pull request that added them that it was intended to change the semantics for tz-aware types as well, so I think it was just an unintentional side effect that they started returning a numpy dtype of datetime64 instead of object.

I noted in the issue that the pyarrow table in the pandas arrays, scalars, and data types section of the API docs seems to indicate that a pyarrow timestamp should map to a pandas DatetimeTZDtype and a numpy datetime64 dtype (which does match how pyarrow itself handles converting a tz-aware array to pandas and numpy respectively)

Quoting from my comment on the issue

I would definitely defer to someone else's judgement on whether that is correct, or if there should be a distinction in that table linked between a pa.timestamp() type with and without timezone

You are pretty much exactly the person I had in mind of who would be best suited to make that judgement call. As far as I can tell, a tz-aware pyarrow timestamp is the only instance in that table that can lose information when it maps to the shown numpy dtype

[road-1]

pre-commit.ci autofix

[road-1]

Hi all, apologies for so many commits to this PR, this is my first issue and I'm not that familiar with GitHub so I was learning as I went. I believe I've fixed the timezone preservation issue and my test case passes for the minimum version unit test as well now. Let me know if you see any issues that need to be addressed though! Thank you for your help!

Fix for #60237