CI: use matplotlib stubs

[twoertwein]

I hope there is a nicer solution: this makes the MS stubs a submodule and then symlinks the matplotlib folder. Pyright and mypy seem to pick up on it as "." is the stub folder.

xref #45

[Dr-Irv]

The symbolic link won't work cross-platform.

I like the idea of linking to the submodule, but I think the way to do it is to modify pyproject.toml to have both pyright and mypy look at those stubs via the directory they would be considering.

[twoertwein]

I think the way to do it is to modify pyproject.toml to have both pyright and mypy look at those stubs via the directory they would be considering.

Mypy's mypy_path and pyright's stubPath can only point to one location, but I think with the current setup we would need two locations: 1) tests needs to use ./pandas-stubs instead of using the pandas code but 2) the pandas-stubs should use ./microsoft-python-type-stubs/matplotlib instead of using the matplotlib code.

The symbolic link won't work cross-platform.

The CI seems to be fine with it :) but I don't know how much friction it would cause contributors working on windows.

The much simpler&better approach is to wait for microsoft/python-type-stubs#211

[breno-jesus-fernandes]

I'm not sure but i think mypy and pyright allows to overrride the config file with a cli command

[twoertwein]

As far as I know, the current tests do not assert any matplotlib type: using the matplotlib-stubs will have no/little impact at the moment. It is probably better to wait until there is a ms-stubs-matplotlib package.

About installing matplotlib as part of pandas-stubs: I'm in favor of removing the dependency on matplotlib. If a user doesn't have matplotlib installed, they probably also do not interact with any parts of pandas that return a matplotlib object. Unless there is a technical reason, I also wouldn't mind removing the dependecy on pandas. matplotlib and pandas could simply be moved to the development dependecies.

[Dr-Irv]

Create an environment with pandas-stubs installed, but not pandas or matplotlib. You might have to hand-delete pandas and matplotlib from site-packages. Then see if pyright and mypy will type check the test code. If so, then I agree with respect to the dependencies being removed as a requirement for installation. I think we may still need to have them installed due to various imports.

[twoertwein]

I think we should definitely remove pandas (and matplotlib) and also disable useLibraryCodeForTypes! It seems there are many types, e.g. CategoricalIndex that are referenced by pandas-stubs but from pandas.core.indexes.category import CategoricalIndex as CategoricalIndex does not exist at all! I assume mypy just treats it is Any and pyright uses the pandas code (useLibraryCodeForTypes) to infer the missing type.

$ grep -R "class CategoricalIndex" pandas-stubs/
<nothing>

[twoertwein]

After fixing the missing types, we would need to disable reportMissingImports and reportMissingModuleSource.

Here is a list of the errors:

pandas-stubs/__init__.pyi:13:5 - error: "CategoricalIndex" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:36:5 - error: "PeriodIndex" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:37:5 - error: "RangeIndex" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:48:5 - error: "array" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:57:5 - error: "period_range" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:58:5 - error: "set_eng_float_format" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:68:35 - error: "eval" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:72:5 - error: "cut" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:73:5 - error: "get_dummies" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:81:5 - error: "qcut" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:87:5 - error: "ExcelFile" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:88:5 - error: "ExcelWriter" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:89:5 - error: "HDFStore" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:90:5 - error: "read_clipboard" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:92:5 - error: "read_excel" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:93:5 - error: "read_feather" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:95:5 - error: "read_gbq" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:96:5 - error: "read_hdf" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:97:5 - error: "read_html" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:99:5 - error: "read_orc" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:100:5 - error: "read_parquet" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:101:5 - error: "read_pickle" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:102:5 - error: "read_sas" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:103:5 - error: "read_spss" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:104:5 - error: "read_sql" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:105:5 - error: "read_sql_query" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:106:5 - error: "read_sql_table" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/__init__.pyi:109:5 - error: "to_pickle" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/core/api.pyi
pandas-stubs/core/api.pyi:47:5 - error: "CategoricalIndex" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/core/api.pyi:54:5 - error: "PeriodIndex" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/core/api.pyi:55:5 - error: "RangeIndex" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/core/arrays/sparse/__init__.pyi
pandas-stubs/core/arrays/sparse/__init__.pyi:6:5 - error: "BlockIndex" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/core/arrays/sparse/__init__.pyi:7:5 - error: "IntIndex" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/core/indexes/datetimes.pyi
pandas-stubs/core/indexes/datetimes.pyi:4:5 - error: "PeriodIndex" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/io/api.pyi
pandas-stubs/io/api.pyi:3:5 - error: "ExcelFile" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/io/api.pyi:4:5 - error: "ExcelWriter" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/io/api.pyi:5:5 - error: "read_excel" is unknown import symbol (reportGeneralTypeIssues)
pandas-stubs/io/api.pyi:20:27 - error: "read_sas" is unknown import symbol (reportGeneralTypeIssues)

[twoertwein]

Some classes such as ExcelFile are defined but the api/init files just import them from the wrong files.

pandas-stubs/io/api.pyi: from pandas.io.excel import ExcelFile as ExcelFile but pandas-stubs/io/excel/__init__.pyi does not exist!

[Dr-Irv]

I think we should definitely remove pandas (and matplotlib) and also disable useLibraryCodeForTypes! It seems there are many types, e.g. CategoricalIndex that are referenced by pandas-stubs but from pandas.core.indexes.category import CategoricalIndex as CategoricalIndex does not exist at all! I assume mypy just treats it is Any and pyright uses the pandas code (useLibraryCodeForTypes) to infer the missing type.

I discovered that we were missing these things the other day. I spent 2 hours fixing these, but ended up losing the fixes due to the issue mentioned in #36, which was very frustrating! So I have to find the time to make the same set of fixes (probably this coming weekend), then we can change the settings for pyright as you mention.

[Dr-Irv]

Some classes such as ExcelFile are defined but the api/init files just import them from the wrong files.

pandas-stubs/io/api.pyi: from pandas.io.excel import ExcelFile as ExcelFile but pandas-stubs/io/excel/__init__.pyi does not exist!

There is a fix in #33, but I'm awaiting a test for that.

[twoertwein]

Closing this: probably makes more sense to wait for microsoft/python-type-stubs#211