Time x Space tutorial with basis representation and FPCA ?

[mat-ej]

Hi Scikit-Fda team,

First, let me start by saying that this is one of the best packages I have ever stumbled upon. Your tutorials are top notch introduction into functional data analysis. I love the way it is similar and works beautifully with original sci kit learn.

What I am missing is a tutorial or perhaps just a short guide on how to work with Time x (Latitude, Longitude) data. I have found the tutorial for dealing with time and I have found tutorial for dealing with surfaces.

https://fda.readthedocs.io/en/latest/auto_examples/plot_composition.html

The tutorials are very good, but I am having trouble understanding how to construct basis representation for
(Latitude x Longitude x Time) data, how can I perform FPCA in that case etc.

Is there such a tutorial or perhaps a slack community where I might get some pointers? : )

Thanks in advance for your help and such an awesome package.

Have a great week.

[vnmabus]

First, let me start by saying that this is one of the best packages I have ever stumbled upon. Your tutorials are top notch introduction into functional data analysis. I love the way it is similar and works beautifully with original scikit learn.

Thank you! Making our work compatible with scikit-learn is one of our core principles, as it allows familiarity and code reuse, and simplifies some workflows in which both functional and multivariate data appear.

What I am missing is a tutorial or perhaps just a short guide on how to work with Time x (Latitude, Longitude) data. I have found the tutorial for dealing with time and I have found tutorial for dealing with surfaces.

https://fda.readthedocs.io/en/latest/auto_examples/plot_composition.html

As functional data analysis is a large topic, we have focused first in the 1D (curves) case, in both the code and documentation. I agree that we should attempt to improve that situation. The state of the package (develop branch) in the ND case is the following:

• Functions of several variables discretized in a complete grid can be represented as a FDataGrid.
• Some bases, such as TensorBasis and FiniteElementBasis can be employed to create a basis for functions of several variables.
• The conversion from FDataGrid to FDatabasis should work (but may be slow or use much memory).
• Many of the methods that work with FDataGrid do not attempt to work with functions of several variables. A notable exception are methods based on distances, which will work if the distance is defined for such functions (e.g. the default L2 distance).
• As a design principle, we try to make the methods work in general if it is not too difficult. However, if it slows development too much we focus in the 1D case. Methods that do not support several variables should raise an exception when that type of data is provided (but as we do not focus for now in that case, it is probable that some failed to do so and produce bogus results instead. This should be treated as a bug).
• Methods that use FDataBasis tend to work also with basis for functions of several variables, as they are usually based on the inner product, which is implemented for that case. However, they MAY be slow or use much memory. An example of these methods would be FPCA for FDataBasis.
• Unfortunately, we have not yet implemented LinearDifferentialOperator for the ND case, so smoothing based on the derivatives will not work.

The tutorials are very good, but I am having trouble understanding how to construct basis representation for (Latitude x Longitude x Time) data, how can I perform FPCA in that case etc.

• If your data has been measured in a complete grid Latitude x Longitude x Time you can represent it as a FDataGrid.
• Then, you can create a TensorBasis passing to it the individual desired 1D bases for the Latitude, Longitude and Time.
• You can convert your original discretized data to FDataBasis using either the to_basis method or the BasisSmoother class.
• You can then use the FPCA class to perform FPCA in basis form.

Is there such a tutorial or perhaps a slack community where I might get some pointers? : )

As I said, we should improve the tutorial including topics such as registration, classification or regression, and probably also a section on functions of several variables.
We do not have a Slack (nor Discord nor Matrix) community. I agree that it would be an amazing resource to have, but we do not have currently the time to manage one. We have however the Github discussions open, so that users of the package (or functional data analysis researchers/practitioners, even if they do not use it) can ask and answer questions or comment in related topics.

Thanks in advance for your help and such an awesome package.

Thank you again! I hope that my answers helped you.

[mat-ej]

Perhaps I need to read more about the functional view on data it seems.

My objective is a regression of y with X, (Lat, Lon) map of weather data as the feature vector. The points on map are highly spatially correlated and hence I hoped for spatial representation to be able to "compress" this data before the regression.

My data is perfectly equispaced across space and also time, so regular structure I assume. Yes it can be 1 curve for each station, hence the data has dimension.

X = (Time, Latitude, Longitude)

X = (92472, 50, 50), y = (92490)

But there is strong time autocorrelation in time for each of measurements and also the target, (and other structures w.r.t. to time such as seasonality)

So, I might reshape the data into "repetition" of days, (or years) for each of the stations to get Time X Lat x Lon data ?
i.e. X would have the shape

X = (Day Count, Daily Hour, Lat, Lon)

X = (3853, 24, 50, 50), y = 3853

but then I assume my target y would have to be daily as well.

In any case, so far the spatial representation does improve my model where I simply represent the equispaced (Latitude, Longitude) weather data in a functional form:

weather_basis = TensorBasis([
    BSplineBasis(n_basis=n_basis),
    BSplineBasis(n_basis=n_basis)
])

This allows me to simply FPCA various weather measurements, (each separately fcourse) into very few features and use them for regression.

I was only wondering about the time dimension If I would then be able to calculate derivatives of "spatial" weather w.r.t to time and use it, but I am still trying to understand if it would be any different from simply diffing each stations measurements with its previous observation etc.

I found the FDAs ability to deal with missing data both in space and time very interesting and possibly useful.

Thanks for your patience and help : ). This is somewhat new view on data for me so I am still trying to wrap my head around the concepts and see when it is beneficial and when should FDA be used.

Have a great day.

[vnmabus]

I think that the way to frame this problem depends on which data do you have at prediction time. Consider the following examples:

Example 1:
At prediction time I have a functional variable (e.g. annual curve of temperatures of the station). I want to predict some scalar value (e.g. mean annual precipitation). This is a problem of scalar-on-function regression (or functional regression with scalar response). You can apply here linear regression, k-NN regression, kernel regression or FPCA regression (which is FPCA + Linear regression in the FPCA basis).

Example 2:
Same as above, but I have additional scalar variables (such as latitude or longitude) or functional variables (such as annual curve of wind speed). This is mixed data (in the first case) or multivariate functional data (in the second case), with scalar response. You can still apply linear regression, but in order to apply other regression methods based on distances, such as k-NN, you would need to define your own distances, as we currently do not provide distances for mixed or multivariate functional data.

Example 3:
At prediction time I have just scalar variables (e.g. latitude and longitude). I want to predict a whole curve (e.g. the annual curve of temperatures). This is function-on-scalar regression. You can apply the distance-based methods such as k-NN regression or kernel regression. The implementation of linear regression for this model is currently being done in #531, if you want to try it.

Example 4:
At prediction time I have scalar variables (e.g. latitude, longitude and time). The response is also scalar (e.g. temperature at that time and position). This is standard multivariate regression. This particular case can be seen as an alternative way of expressing Example 3. Note however that in Example 3 we could force certain smoothness on the predicted functions, either by choosing a basis or via regularization, which is not done here.

Example 5:
We have at prediction time a functional variable (e.g. annual curve of temperatures of the station) and another functional variable as the response (e.g. annual curve of precipitations). This is function-on-function regression. Again you can apply distance-based methods. Concurrent linear regression (in which the point t of the response depends only on the point t of the covariate) is implemented in #531, but not yet merged. A historical linear regression model (in which the response can depend on points s ≤ t) is already available as a separate class. The general linear case is not yet implemented.

Example 6:
Same as above, but with additional scalar or functional covariates. We still do not have implemented a linear regression model for this case. Distance-based methods would require your own definition of the distance.

You should consider what is your case, to see which model applies.

[mat-ej]

Great great answer, thanks so much.

[mat-ej]

So from the error messages I seem to have understood that regularization is not possible for 2D bases, is that correct ?

# Combine them into a TensorBasis
basis_b = TensorBasis([lon_basis_b, lat_basis_b])

smoother = skfda.preprocessing.smoothing.BasisSmoother(
    basis=basis_b,
    regularization=regularization,
    return_basis=True,
)

fd_basis = smoother.fit_transform(fd)

NotImplementedError: basis does not support the construction of a basis of the derivatives.

[vnmabus]

You are right, it is not currently implemented.

[mat-ej]

Thanks for answer, I ll move my question to the discussions

[vnmabus]

Moved to discussions.

[mat-ej]

So from the error messages I seem to have understood that regularization is not possible for 2D bases,

# Combine them into a TensorBasis
basis_b = TensorBasis([lon_basis_b, lat_basis_b])

smoother = skfda.preprocessing.smoothing.BasisSmoother(
    basis=basis_b,
    regularization=regularization,
    return_basis=True,
)

fd_basis = smoother.fit_transform(fd)

NotImplementedError: basis does not support the construction of a basis of the derivatives.