Pearson correlation without z-normalization

[tomweb153]

Hello everyone,

I would like to ask, if it's possible to compute the pearson correlation without z-normalization using stumpy?

I have read something about the pearson correlation here: #484

However, due to many flat sequences z-normalization becomes counterproductive for me.

Thanks!

[seanlaw]

@tomweb153 Thank you for your question and welcome to the STUMPY community.

It isn't so clear to me what you mean when you say "Pearson correlation without z-normalization". At the end of the day, Pearson correlation has a very clear definition (and does not necessarily depend on z-norm). In general, if your time series contains many constant subsequences then, by definition, the Pearson correlation will be 1.0 by simply following its definition above. Now, if you know that you are disinterested in those constant subsequences then you can change those constant sections to np.nan and then they will NEVER become a 1-nearest-neighbor to any subsequence.

Hopefully that helps. Otherwise, please feel free to provide a small working code example to demonstrate your point.

[tomweb153]

@seanlaw Thank you for your response.

I tried to provide a minimal example. I have selected data from 6 sensors from 2 different systems. Measurements are taken every 5 minutes and the data spans over 1 week. Subsequence length is set to 12 (==1hour of data).

The code I provided calculates the (left) matrix profile for every sensor independently. As you can see the data is mostly steady with some outliers/anomalies present in data2.csv. I wish to detect these outliers/anomalies with the matrix profile (I provided a very easy case with some global point outliers).
When tweaking the normalize flag to False you can see high distances for the outliers/anomalies (e.g. around index 700 for arr2). However one cannot see any high mp-values when normalize is set to True. For the sake of completeness I also added some data without any outliers/anomalies (see data1.csv).

I would love to set normalization to true, but I don't seem to be able to get my data into the right shape. I was hoping I could use a bounded output (e.g. range -1...1) so a threshold estimation could be done. The threshold is supposed to determine whether a anomaly may have manifested itself on the data or not.

Unfortunately I cannot paste my Jupyter-Notebook as a file, so I have to write the code here. You should be able to run this code in a Jupyter-Notebook with Python 3.10.0 installed.

from stumpy import stumpi
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

df1 = pd.read_csv('data1.csv', delimiter=',', index_col='timestamp')
df2 = pd.read_csv('data2.csv', delimiter=',', index_col='timestamp')

arr1 = df1.to_numpy()
arr2 = df2.to_numpy()

fig, axs = plt.subplots(6, sharex=True, figsize=(30, 15))

# change arr for different plot
for i in range(6):
    axs[i].plot(arr2[:,i])
    plt.show()
    
# calc lmp for each dimension independently
# change arr1 to arr2 if needed
dimensions = arr1.shape[1]
all_matrix_profiles = []
for dim in range(dimensions):
        
    n_time_series = arr1[:,dim]
    # change normalize to see "problems"
    n_matrix_profile = stumpi(n_time_series, m=12, normalize=False);
    n_lmp = n_matrix_profile.left_P_.astype(np.float64)
    
    # exclude warmup-phase 
    # -> exclude inf and -1 values
    n_lmp = n_lmp[12:]

    all_matrix_profiles.append(n_lmp)
    
    # transpose to get dimensions as columns and rows as index
all_matrix_profiles = np.asarray(all_matrix_profiles).T

# plot lmps
fig, axs = plt.subplots(6, sharex=True, figsize=(30, 15))

# change arr for different plot
for i in range(6):
    axs[i].plot(all_matrix_profiles[:,i])
    plt.show()

data2.csv
data1.csv

[seanlaw]

So, one thing to understand is that matrix profiles with z-norm are good for finding subsequences with similar shape (regardless of their overall magnitude). However, they are horrible for finding anomalous (single point) spikes in your data. As you've identified, setting normalize=False is the only reasonable way to identify those spikes.

I was asking, if there was any way one could bound the non-z-normalized Euclidean distance to any range.

No, unfortunately there is no way to bound the non-normalized Euclidean distance .

[NimaSarajpoor]

@tomweb153
Since you probably got most of your answers, I am going to just add two short notes.

When tweaking the normalize flag to False you can see high distances for the outliers/anomalies (e.g. around index 700 for arr2). However one cannot see any high mp-values when normalize is set to True. For the sake of completeness I also added some data without any outliers/anomalies (see data1.csv).

I think you first need to understand what normalize=False means, and how the distance calculation works (e.g. see this video).

Let's say your time series is T, and length of subsequence is m. Also, let's say the minimum value in T is MIN, and the maximum value is MAX. It can be shown that the distance between two subsequences cannot be outside of the range $[0, (MAX-MIN)\sqrt{m}]$ (when normalize is False).

When normalize=True, the distance between two subsequences is the distance between their z-normalized version (i.e. EACH of the two subsequences is Z-NORMALIZED first, and THEN their distance is being calculated). It can be shown that this distance is always in the range $[0, 2\sqrt{m}]$. For each DISTANCE value (dissimilarity), you can find its CORRESPONDING SIMILARITY SCORE, which is just the "correlation" between the two subsequence. See this doc for further info.

Note: What is the lesson here? Sometimes it might be good to know, at some level, the underlying concept / logic. This can help you better understand the result. Otherwise, you will be forced to justify!

I was asking, if there was any way one could bound the non-z-normalized Euclidean distance to any range.

I do not know the "why" behind this. Maybe you have some reasons, and I am not sure if I understand your message correctly or not. So, I am going to give answer from my perspective based on my own understanding. I assume you just want to change the upper-bound $(MAX-MIN)\sqrt{m}$ (provided earlier above) to something else WHILE keeping the relative distance between subsequences untouched. In this case, you can consider the factor $\frac{1}{(MAX-MIN)\sqrt{m}}$ and scale your whole time series by that, or maybe the general scaler $\frac{A}{B}$. I will leave it to you to think and see how that can affect that distances, and the range (and you, the user, should think if that makes sense for your problem or not).

[seanlaw]

Let's say your time series is T, and length of subsequence is m. Also, let's say the minimum value in T is MIN, and the maximum value is MAX. It can be shown that the distance between two subsequences cannot be outside of the range
(when normalize is False).

Note that while this may be true for a static time series, this assumption will break if you have a new data point that is larger than MAX and/or smaller than MIN (and normalize=False). The point is that when normalize=False, this implies that new/additional data points in your time series is unbounded. This is similar to the problem of normalization in machine learning problems where you may cap your min/max range according to your training data in order to limit the range between 0 and 1 but your test data may contain values that are much larger than your MAX and therefore it will be normalized to something larger than 1 (or smaller than MIN and therefore will be normalized to something smaller than 0). Accordingly, you will run into the same issue if you use one time series to decide your MIN/MAX and then you find that a second time series has values outside of that range (i.e., either below the MIN OR above the MAX) then you'll get into trouble (again for normalize=False). So please proceed with caution.

[tomweb153]

@seanlaw @NimaSarajpoor thank you both for your insights.

I will try to take the things you have said into account!