How do I specify multiple data folders with in catalog.yml file?

[praneeth5222]

Hello,

I wanted to know if there is a way to specify multiple data folders in the catalog.yml file
At the moment my data storage has a root folder which has one folder corresponding to each date like - 2022-09-30, 2022-10-01... etc. and internally each date folder has many parquet files.
I was able to load the multiple parquet files corresponding to one date folder in this way using the PartitionedDataSet-

type: PartitionedDataSet
path: "/Users/praneeth/Desktop/data/2022-09-30"
dataset: pandas.ParquetDataSet
load_args:
  engine: pyarrow
  use_nullable_dtypes: True

Can anyone suggest me if there is a way to load the specified folders instead of a single folder? (for e.g -> to load all the folders between 2022-09-26 till 2022-10-03)
Ideally I wanted to load the last 7 day folders based on the current date

Thank you!!

[datajoely]

So there isn't a native way of doing this. The idea of a partitioned, partitioned dataset has come up before so it's always good to see this sort of request come up as it can influence our next set of priorities.

This does somewhat break one of Kedro's core tenants of reproducibility, but it may be possible to do by subclassing partitioned dataset and introducing some custom logic that handles the date window parts you need.

[praneeth5222]

Hi @datajoely

Thank you for your response. Do you mind explaining in detail about subclassing partitioned dataset and also how to implement the custom logic that you're talking about?

[andrewcstewart]

@praneeth5222 -> I think @datajoely is just referring to creating a 'Custom Dataset' class per https://kedro.readthedocs.io/en/stable/extend_kedro/custom_datasets.html