Integrating with a ColumnTransformer

[dcbrien]

I thought I would explain how I implemented the skfda tools into my pipeline. For a classification based task I wanted to mix fda outputs with numerical data to create a feature vector. This isn't straight forward in sklearn. What I ended up doing was:

1. Created a pandas DataFrame that mixed FDataGrid objects with standard numerical types. This splits the functional data though into rows of FDataGrids with one functional data each.
2. Used a ColumnTransformer to selectively direct the functional data to a custom pipeline.
3. However, when using a cross-validation with this, the data will come out as an array of FDataGrids for each fold.
4. I implemented a FunctionTransformer to use skfda.concatentate to reconstitute the full FDataGrid on the subset of data from the current fold.
5. The subset of data in the FDataGrid for each fold can then use the skfda tools and output some features, which are ultimately concatenated with the numerical features for use in some kind of scorer / estimator.

That was the simplest way I found, but nicely avoids data leakage in the fda data. I have multiple functional data as well, so each one can go to a custom pipeline with custom basis / smoothing.

I wonder if something like multivariate fPCA could be considered for future versions to add to this?

[vnmabus]

I don't get 1. Currently all FData objects inherit from Pandas' ExtensionArray, so they should be usable as Pandas columns directly (without the need of creating a list of FData objects). We want to expand that functionality in the future, though.

[dcbrien]

Perhaps there is a better way, but if I create a DataFrame such as:

X = pd.DataFrame( { "fd_data": Some_FData, "feature1", Some_Numeric } )

Then the structure rows are Data set: [[[...]]] of each functional data. This is fine.

When using the a function such as cross_val_score on this X, the data is sent to the Transformers as a list of FData, not one, so I needed to concatenate them.

[vnmabus]

I think they are just represented that way (by iterating over each element). We should definitely improve the representation of FData inside a DataFrame, as it is currently inefficient and unreadable. If you do X["fd_data"] you can see that the dtype is correctly identified as FDataGrid or FDataBasis, and internally it is stored as just one FData with the total number of samples.

I have checked that some functions such as train_test_split seem to work fine with the data. I have not tried cross_val_score in particular. Do you have a small example that I can use to look for a possible solution?

[dcbrien]

Indeed it does indicate that dtype, but I cannot pass it to any functions, such as FData.mean or FPCA

[vnmabus]

No, currently most functions don't accept a Pandas DataFrame or Series. We are working to improve that. You can recover the internal FData when you need it using X["fd_data"].values.

[dcbrien]

Okay that makes sense, so either way a FunctionTransformer appears needed if you're mixing types like I am and using cross_val_score.