Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

how long will the training process takes? #62

Open
guchengxi1994 opened this issue Mar 23, 2021 · 2 comments
Open

how long will the training process takes? #62

guchengxi1994 opened this issue Mar 23, 2021 · 2 comments

Comments

@guchengxi1994
Copy link

i have trained the model on my own dataset for 27000 rounds, but the d_oss is still between 1.5 to 4.0 and the g_loss is between 9.0 to 40, how many iterations will it takes ?

@knicholes
Copy link

I calculated how many it would take for me to train 1M epocs, and it ended up to be ~7 days with my RTX 3090. If your model isn't converging, maybe you should add more data or look through your dataset to make sure there isn't anything in there that is wrong and screwing up the model's attempt to converge.

@jdunkelheit
Copy link

For me who's training on the CPU, 1 million iterations would take 254 days, but the model is already showing some results after 12,000 iterations.
As far as I know, the time it takes to converge depends on how much data you have and the complexity of the problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants