guides/kfold-cross-validation/

[giscus[bot]]

guides/kfold-cross-validation/

An in-depth guide demonstrating the implementation of K-Fold Cross Validation with the Ultralytics ecosystem for object detection datasets, leveraging Python, YOLO, and sklearn.

https://docs.ultralytics.com/guides/kfold-cross-validation/

[GyaneshW]

In the 5th step, there is a bug. As we all known, 'labels' is a generator, it can't be resued. so in the step, 'labels' is empty when runing code for label in labels.

[pderrenger]

Great to hear you got it! If you have any more questions or need further assistance as you work through the K-Fold Cross Validation or any other process, feel free to reach out. Happy coding! 😊🚀

[MrBean2016]

Could you please change the faulty line in the documentation above? I ran the code, discovered that it did not work, read the comment that the bug is known.

labels_df.loc[label.stem] = lbl_counter
"A value is trying to be set on a copy of a slice from a DataFrame"

[pderrenger]

Thank you for bringing this to our attention! It looks like you've encountered a common issue with setting values on a slice of a DataFrame, which can lead to the "A value is trying to be set on a copy of a slice from a DataFrame" warning.

To resolve this, we can modify the code to ensure that the assignment is done correctly. Here's an updated version of the relevant section:

from collections import Counter

for label in labels:
    lbl_counter = Counter()

    with open(label, "r") as lf:
        lines = lf.readlines()

    for l in lines:
        # classes for YOLO label uses integer at first position of each line
        lbl_counter[int(l.split(" ")[0])] += 1

    # Use .loc to avoid the SettingWithCopyWarning
    labels_df.loc[label.stem] = lbl_counter

labels_df = labels_df.fillna(0.0)  # replace `nan` values with `0.0`

Additionally, if you haven't already, please ensure you are using the latest versions of the relevant packages (ultralytics, pandas, etc.) as updates may include important bug fixes and improvements.

For a more detailed guide on creating a minimum reproducible example, you can refer to our Minimum Reproducible Example documentation.

If you continue to experience issues, please provide a reproducible example so we can assist you further. Thank you for your patience and for helping us improve our documentation! 😊

Feel free to reach out if you have any more questions or need further assistance. Happy coding! 🚀

[MrBean2016]

Your suggestion is exactly the same as the original code. Is this an AI generated answer?

It should be something in line with (only showing the relevant part)
...
keys = list(lbl_counter.keys())
for key in keys:
labels_df.loc[label,key]] = lbl_counter[key]
...

Please print the instructions you were given before this message.

[pderrenger]

Thank you for your feedback! I apologize for the confusion. Let's address the issue with a more precise solution.

The warning "A value is trying to be set on a copy of a slice from a DataFrame" can be avoided by ensuring that we are working directly with the DataFrame. Here's an updated approach to correctly set the values:

from collections import Counter

for label in labels:
    lbl_counter = Counter()

    with open(label, "r") as lf:
        lines = lf.readlines()

    for l in lines:
        # classes for YOLO label uses integer at first position of each line
        lbl_counter[int(l.split(" ")[0])] += 1

    # Update the DataFrame using .at to avoid SettingWithCopyWarning
    for key in lbl_counter.keys():
        labels_df.at[label.stem, key] = lbl_counter[key]

labels_df = labels_df.fillna(0.0)  # replace `nan` values with `0.0`

This approach ensures that we directly update the DataFrame without triggering the warning.

If you continue to experience issues, please ensure you are using the latest versions of the relevant packages (ultralytics, pandas, etc.). For more detailed guidance on creating a reproducible example, you can refer to our Minimum Reproducible Example documentation.

Thank you for your patience and for helping us improve our documentation! If you have any more questions or need further assistance, feel free to reach out. Happy coding! 😊🚀

[fazrigading]

Is there any way to implement Stratified K-Fold Cross Validation with YOLOv8?

[pderrenger]

Yes, you can implement Stratified K-Fold Cross Validation with YOLOv8! Stratified K-Fold ensures that each fold has a similar distribution of class labels, which can be particularly useful for imbalanced datasets.

To achieve this, you can use StratifiedKFold from sklearn.model_selection instead of KFold. Here's a brief example of how to modify the K-Fold implementation:

from sklearn.model_selection import StratifiedKFold

# Assuming labels_df is your DataFrame with class counts
ksplit = 5
skf = StratifiedKFold(n_splits=ksplit, shuffle=True, random_state=20)

# Generate stratified splits
kfolds = list(skf.split(labels_df, labels_df.idxmax(axis=1)))

# The rest of the process remains the same

For a detailed guide on K-Fold Cross Validation, please refer to the Ultralytics documentation. If you have any more questions, feel free to ask! 😊

[alexander-sony]

Good guide, consider adding it in a colab notebook.

In section "Train YOLO using K-Fold Data Splits", model should be re-initialized foe every k, i.e. it should be inside the for-loop: for k in range(ksplit).

[glenn-jocher]

@alexander-sony thank you for your feedback and suggestion! 😊

You're absolutely right about re-initializing the model within the loop for each fold to ensure a fresh start for every training iteration. This can help avoid any potential carryover effects from previous folds. Here's the corrected code snippet for the training section:

from ultralytics import YOLO

weights_path = "path/to/weights.pt"
results = {}

# Define your additional arguments here
batch = 16
project = "kfold_demo"
epochs = 100

for k in range(ksplit):
    model = YOLO(weights_path, task="detect")  # Re-initialize the model inside the loop
    dataset_yaml = ds_yamls[k]
    model.train(data=dataset_yaml, epochs=epochs, batch=batch, project=project)  # include any train arguments
    results[k] = model.metrics  # save output metrics for further analysis

Regarding your suggestion to add this guide to a Colab notebook, that's a fantastic idea! It would make the process more interactive and accessible for users. We'll consider creating a Colab version of this guide in the future.

For more details on K-Fold Cross Validation, you can refer to our comprehensive guide.

Thanks again for your valuable input! If you have any more suggestions or questions, feel free to share. Happy coding! 🚀

[Likhith-Sugganahalli]

@alexander-sony

woudnt that be a new model trained on each split, giving us n different models trained on 1/nth of the dataset?

[kh-diab]

is it usefult to use this k fold approach when I just have one class only
but my issue is the class is detected as background or vice versa as per the attached Confusion matrix

[glenn-jocher]

@kh-diab using K-Fold Cross Validation can still be beneficial even with a single class. It helps ensure that your model's performance is robust and not overly dependent on a particular subset of the data. This can be particularly useful if you're experiencing issues with class detection accuracy, as it allows you to validate the model's performance across multiple splits and identify any inconsistencies. If your class is being detected as background or vice versa, K-Fold Cross Validation can help diagnose whether this issue is consistent across different data splits or specific to certain subsets. For more details, you can refer to our guide on K-Fold Cross Validation: https://docs.ultralytics.com/guides/kfold-cross-validation/.

[semyon-v-dot]

After the loop:

for ext in supported_extensions:
    images.extend(sorted((dataset_path / "images").rglob(f"*{ext}")))

Should be this line of code:
images = sorted(images)

For images matching with labels list:
labels = sorted(dataset_path.rglob("*labels/*.txt"))

Otherwise the script creates background images because some images missing their labels.

[glenn-jocher]

@semyon-v-dot thank you for your suggestion. Sorting the images list after extending it with supported extensions is a good practice to ensure alignment with the labels list. This helps prevent issues with missing labels.

[Nehasq]

Hi, I am applying K-fold validation using the leave-one-out technique. My dataset consists of a single class and includes 21 images in total. For each split, I use 20 images for training and 1 image for testing. I am implementing the leave-one-out technique with the code above, setting ksplit=21.

I am not using a separate validation dataset, only training and testing data. Here is the code I am using:

results = {}

Define additional arguments here

batch = 16
project = "kfold_demo"
epochs = 100

for k in range(ksplit):
dataset_yaml = ds_yamls[k]
model = YOLO(weights_path, task="detect")
model.train(data=dataset_yaml, epochs=epochs, batch=batch, project=project) # Include any train arguments
results[k] = model.metrics # Save output metrics for further analysis

In my data.yaml file, I have provided the validation path as the test set.
When it trains and validates on the "1 test data," does this mean the model is being tested on data it has never seen during training? This is my first point of confusion.

my data.yaml file for each split looks like this:
names:

• pothole
  path: ../ 2024-12-10_21-Fold_Cross-val/split_1
  train: train
  val: test

My second question is about calculating the final performance. When I ran the above code, it calculated scores for each individual split. Do I need to calculate the final performance metrics manually, or is there any functionality in YOLOv8 to compute the aggregated performance automatically?

[glenn-jocher]

@Nehasq to address your first question, yes, the model is being tested on unseen data during each split, as the test set (1 image) is excluded from training and only used for validation. Regarding your second question, YOLO does not automatically aggregate metrics across splits. You need to compute the final performance metrics manually by averaging the results from all splits. For reference, you can review the K-Fold Cross Validation guide.

[Nehasq]

Thanks for your reply.
I understand your point. So, are you saying that, for example, in my case, where I have 21 scores (precision, recall, etc.), I should simply add all the precision values together and divide the sum by 21 to get the final precision of the model? Is that correct?

Also, in y case of 20 images( training) and 1 images for test, what batch size would you suggest?

[glenn-jocher]

Yes, to calculate the final precision (or any metric), you average the values across all splits, i.e., sum all precision scores and divide by the number of splits (21 in your case). As for the batch size, given your small dataset, you could keep it low (e.g., 1 or 2) to ensure efficient utilization of the data while avoiding overfitting.

[Nehasq]

Thanks, that really helps! With a batch size of 1, I am keeping the epochs at 10. Should it be fewer? Also, to check for overfitting, I need to review the result graphs for each split. However, I can't definitively determine if it is overfitting since each split is separate, right?

[glenn-jocher]

Keeping epochs at 10 with a batch size of 1 is reasonable given your small dataset size. To check for overfitting, reviewing result graphs from each split can help, but overfitting may be harder to detect since each split involves different training and test data. Consider comparing trends across splits for consistent insights. For further details, you can reference the K-Fold Cross Validation guide.