Multivariate (bivariate) FPCA

[SJBiomech]

Hi,

I am aware you can create multivariate functional data and store this as an FDataGrid type. I have done this and have some bivariate functional data with domain dimension = 1 and codomain dimension = 2 (similar to that of the weather data where the codomains are temperature and precipitation).

I was wondering if it is possible to apply FPCA to such bivariate functional data using scikit-fda as I am unable to do this and I have browsed through the github and haven't seen anything saying that it is possible?

Many thanks

[vnmabus]

Yes, for now FPCA does not work for discretized functions with domain dimension greater than one. If I recall correctly it should work for functions in a basis expansion (using the standard $L^2$ inner product,).

Increasing support for high dimensional functions is something that needs to be done, but as we usually deal with unidimensional datasets we don't usually allocate time for it. PRs are welcome though!

@Ddelval You worked on FPCA recently. Do you think that expanding it to higher dimensional codomains is easy to do? I think Ramsay does that by using the standard $L^2$ inner product, which is similar to just concatenating the functions (at least for equally spaced grid points). However, if the different dimensions have different units or scales, probably we should allow the use of a different inner product.

[Ddelval]

At first glance, it does not seem too complicated. Indeed, Ramsay defines an inner product equivalent to the standard $L^2$ inner product of the concatenation of all the codomains. This does seem like a natural extension and could work well as long as the scale of the different codomains is similar. If that is not the case, I think we might be able to do some kind of normalisation to the input and solve the issue that way. Maybe it could even be interesting to consider the possibility of assigning different weights to each dimension in the inner product.

However, I am not entirely sold on this inner product. Mainly, I am concerned about the orthogonality that it defines. Moreover, without looking more into it, I think maximising the square of the sum of the integrals (one integral for each dimension) might lead to cases where the variance on two dimensions is high but they have opposite signs. In that case, they would cancel out, and that "direction" would not be considered a good one. This would not be equivalent to unidimensional FPCA, where, since the variance is squared, the sign is irrelevant.
Regardless, I think it might be worth it to discuss this.

In the meantime, @SJBiomech, if you want to try to get some results, you could try to do FPCA with an FDataGrid obtained by concatenating the two dimensions of the one you have. That is to say, if your functional observations are $f_i= (f_{i,1},f_{i,2})$ evaluated on the grid $X=\{x_1,\dots,x_m\}$, consider the FDataGrid where the data matrix for the $i$-th observation is $\{f_{i,1}(x_1),\dots, f_{i,1}(x_m), f_{i,2}(x_1),\dots,f_{i,2}(x_m)\}$ and the grid is $X'=\{x_1,\dots,x_m,x_1,\dots,x_m\}$.

Ramsay does explain this approach better than I think I just did, so if you want to have a look at their explanation, check page 167 of Ramsay, J. O. & Silverman, B. W. (2005), Functional Data Analysis, Springer .

[SJBiomech]

Thanks both of you for your replies.

I have seen Ramsay's explanation in their book but will have to give it a closer look to try and understand it some more.

I have also seen similar in the following paper where they describe their implementation of bivariate FPCA to angle-angle data where the units are the same for both codomains.
https://doi.org/10.1080/14763140701323042

I have however also seen an implementation of bivariate FPCA where the domain is the same but the two codomains have different units. They do appear to acknowledge that the units are different but seem to go on to describe that they also used the inner product. (This is the kind of application that I would be more interested in implementing.)
https://doi.org/10.1080/14763141.2017.1384050

Thanks for your suggested implementation, @Ddelval. I will try to figure out how to go about doing this (and then ideally try to visualise the bivariate FPCs as done by Ramsay and in the papers I have mentioned) and see how far I can get.

[ego-thales]

Hi!
I'm also interested in this lately and would like to share some thoughts.

Inner product concern

@Ddelval on Feb 9
[...] I think maximising the square of the sum of the integrals (one integral for each dimension) might lead to cases where the variance on two dimensions is high but they have opposite signs. In that case, they would cancel out, and that "direction" would not be considered a good one. This would not be equivalent to unidimensional FPCA, where, since the variance is squared, the sign is irrelevant. Regardless, I think it might be worth it to discuss this.

I am confused by your confusion. Ramsay naturally defines an inner product as the sum of two integrals,
$$\langle f, g\rangle := \int f^{(1)}g^{(1)} + \int f^{(2)}g^{(2)},\quad\quad(f, g:\mathbb{R}\mapsto\mathbb{R}^2)$$
which is just equivalent to "adding more coordinates" to the original $1$-D FPCA, which in turn, is equivalent to the classical PCA where you stack the dimensions along $t$. When you do usual PCA in $\mathbb{R}^k$, you also try to maximize (with $\Vert \xi\Vert_2=1)$
$$\sum_i\langle x_i, \xi\rangle^2=\sum_i\left(\sum_{1\leqslant j\leqslant k}x_i^{(j)}\xi^{(j)}\right)^2,$$
where cancellations occur in the inner sum. So I don't personally think that there is a conceptual flaw here.

Weights in the inner product

@vnmabus on Feb 9
However, if the different dimensions have different units or scales, probably we should allow the use of a different inner product.

@Ddelval on Feb 9
I think we might be able to do some kind of normalisation to the input and solve the issue that way. Maybe it could even be interesting to consider the possibility of assigning different weights to each dimension in the inner product.

When doing PCA, one "compares" dimensions against one another on the same scale, as if they were homogeneously comparable, which is sometimes the case, sometimes not. In classical $1$-D FPCA, this is by construction always satisfied, as the value of a functional sample at instant $t$ lives "in the same space" as the value of this parameter at instant $t+1$.

When introducing multivariate FPCA as described above, one starts assuming that those multivariates are comparable, in the same way that it is assumed in classical PCA. Thus, as for PCA, it might indeed be a good idea to normalize the data across every dimension before running multivariate FPCA. This is, in my opinion, neither more, nor less, satisfying than doing PCA, and thus seems a viable solution.

Simply concatenate?

@vnmabus on Feb 9
I think Ramsay does that by using the standard
inner product, which is similar to just concatenating the functions (at least for equally spaced grid points)

@Ddelval on Feb 9
In the meantime, @SJBiomech, if you want to try to get some results, you could try to do FPCA with an FDataGrid obtained by concatenating the two dimensions of the one you have.

To understand if it is indeed, or not, equivalent to what we want, we should focus on what FPCA brings that PCA doesn't. From my perspective, the only mechanism that makes FPCA different from PCA is regularization (or smoothness, interpolation, ..., call it what you will) and the computational tricks that come with it. The main point is that there should be some form of continuity from one timestep to the other.

As such, simply concatenating generates a small problem: it forces the PCs to be continuous at the transition point from one coordinate to another, which could be a major drawback in some cases I guess. This should not be too hard to fix though.

When additional structure is known

(Not a mature POV)
Imagine you are working with $f=(f^{(1)}, f^{(2)})$ where coordinates are naturally in the same space -- for example $(x, y)$ coordinates in a plane. Then I think in some cases, it might be sensible to consider looking for complex PC scores instead of real PC scores. This would mean that variability would be described not as a multiple of a component in a fixed direction, but also with a rotation of the component. This might be another way to extend FPCA, which could be more meaningful in certain use cases I guess, but would also require a bit more work to properly extend.