feat: Advertools - Check website pages status code

[FlorentLvr]

This PR resolves #2099

[FlorentLvr]

@eliasdabbas, I have created a notebook that crawls a website and checks the status code of all URLs using your notebook templates. It has helped us identify 100 failed URLs on our website page. :)
Could you please take a look at it and let me know if there is anything you would suggest improving?🙏

[eliasdabbas]

@FlorentLvr
Great to see this, and happy to know that it helped discover pages with issues!

Please note that the crawl function already checks the status codes, and you can get them through the status column.
The main differences:

• crawl_headers: You need to know the URLs that you want to check status codes. It is also much faster, because you don't download the page, it just sends a HEAD request, which includes the status code and a few other things.
• crawl: It discovers URLs by starting from the home page, and discovering and following links in the website.

So, if the website is frequently changing, adding/removing new pages, crawl would be a better choice. If you have a static list of URLs that you want to check on a periodic basis, you can use crawl_headers.

So, based on this, how do you prefer to with this notebook, split it, or something else?
Happy to help on this.

[FlorentLvr]

@FlorentLvr Great to see this, and happy to know that it helped discover pages with issues!

Please note that the crawl function already checks the status codes, and you can get them through the status column. The main differences:

• crawl_headers: You need to know the URLs that you want to check status codes. It is also much faster, because you don't download the page, it just sends a HEAD request, which includes the status code and a few other things.
• crawl: It discovers URLs by starting from the home page, and discovering and following links in the website.

So, if the website is frequently changing, adding/removing new pages, crawl would be a better choice. If you have a static list of URLs that you want to check on a periodic basis, you can use crawl_headers.

So, based on this, how do you prefer to with this notebook, split it, or something else? Happy to help on this.

@eliasdabbas, Thank you for the review and providing valuable feedback! I appreciate your observation regarding the status on the crawldf and it has been very helpful. I have now removed the section with crawl headers, and the code is working perfectly now. 🙏

[github-actions]

The template is now available on the master branch on this link:
https://github.com/jupyter-naas/awesome-notebooks/blob/master/Advertools/Advertools_Check_website_pages_status_code.ipynb

[github-actions]

Thank you for your contribution @FlorentLvr, your PR has been merged into the master branch of awesome-notebook.
Here is the contribution certificate you can share on social media so everybody knows how awesome you are 🤙🌎.
Spread the #opensource love 💚