Add to links active/inactive flag

[chaoSefat]

• Check if get_text(link) == None and link in seed_urls:
• add flag inactive.
• otherwise add active.

[chaoSefat]

1. remove self links from links to scrape i.e. all_links and get a dictionary of scraped link. add active '1' and also save the engines.
2. convert all_data to a format where it is a dictionary with 'link' as keys for fast look up.
   Then use it to check
   if 'common-crawl' in engines :
   save 'cc-text'
   save engines
   link for all entries are active or not.
   if in active then active:1 else active: 0
   else:
   use trafilatura scrape and filter

Then get info of that. check if it's category is commoncrawl and has snippet or not.

[chaoSefat]

def check_urls(url_list):
    result = []
    
    for url in url_list:
        try:
            response = requests.get(url, timeout=5)
            if response.status_code == 200:
                result.append({"link": url, "active": 1})
            else:
                result.append({"link": url, "active": 0})
        except requests.RequestException:
            result.append({"link": url, "active": 0})
    
    return result