How do you use different masks in PyGOD datasets.

[prabhant]

Hi,
When loading a dataset in pygod eg:

from pygod.utils import load_data
data = load_data('inj_cora')
print(data)
Data(x=[2708, 1433], edge_index=[2, 11060], y=[2708], train_mask=[2708], val_mask=[2708], test_mask=[2708])

I wanted to know more about he usage of train_mask, val_mask and test_mask here?
Do you split the dataset for these tasks?
right now I try to compute the validation loss like this

auc_score = eval_roc_auc(data.val_mask.numpy(), outlier_scores)

Does this make sense?

[kayzliu]

Thanks for your questions! Actually the masks are inherited from PyG built-in Cora dataset. In current PyGOD unsupervised setting, there is no need to use the masks. Computing the loss and evaluating on all nodes are acceptable. Of course, if you only want to compute the loss for part of the nodes (e.g., the nodes in the validation mask), you can definitely do so.

[kayzliu]

"But is that still valid for a model selection/hyperparameter optimization setting? does that not lead to overfitting on that particular bit of data?":
By far, we are still under unsupervised setting. As we discussed in Section 4.1 Hyperparameter Grid in our benchmark paper, we claim that using ground truth labels to tune hyperparameter under unsupervised setting is not appropriate.

"does the validation mask and train mask have even outlier splits for the selected datasets?":
Our outlier injection is independent from the splits, so the outlier distribution should be approximately even. For example, in 'inj_cora', the outlier ratio is 0.050, 0.054, and 0.046 in training set, validation set, and testing set, respectively.

[prabhant]

Thank you for your reply. Is there any documentation to find out the outlier ratio in different masks for other datasets as well? I. was not able to find it in the docs.

[prabhant]

I have another small confusion here, if I decide to use just data.x[data.train_mask] to train then I cannot pass this "data.x[data.train_mask]" to any pygod estimator as they assume I am parsing entire PyG dataset.
so right now what I am trying to do is to evaluate the performance. please correct me if I am wrong here

model.fit(data)
scores = mode.decision_function(data)
eval_roc_auc(data.y.numpy()[data.train_mask], scores[data.train_mask])

Is this score a valid approach for this setting? I assume we do not have access to ground truth label of other masks while computing these scores, so it should be alright.

[kayzliu]

Thank you for your reply. Is there any documentation to find out the outlier ratio in different masks for other datasets as well? I. was not able to find it in the docs.

We typically don't use the masks, so we do not provide the docs about it. But you can validate by

y = data['y'][mask]
ratio = sum(y.bool())/len(y)

[kayzliu]

I have another small confusion here, if I decide to use just data.x[data.train_mask] to train then I cannot pass this "data.x[data.train_mask]" to any pygod estimator as they assume I am parsing entire PyG dataset. so right now what I am trying to do is to evaluate the performance. please correct me if I am wrong here

model.fit(data)
scores = mode.decision_function(data)
eval_roc_auc(data.y.numpy()[data.train_mask], scores[data.train_mask])

Is this score a valid approach for this setting? I assume we do not have access to ground truth label of other masks while computing these scores, so it should be alright.

In the fit method, we already train on the entire graph. The loss is calculated on the entire graph without any ground-truth labels. You may not be able to train on part on the graph if you only modify the graph feature but not the edge list. It is acceptable, but not necessary I think, to evaluate on the nodes only in train_mask.