How to bring in Candidate URLs from the test server

Bringing in select collections

Log into the test server (bastion)
From the bastion, RDP into the following server 10.51.14.135
On the server, open up Anaconda Powershell Prompt (miniconda3) and navigate to E:\sde-indexing-helper\config_generation
Find python file called export_collections.py and, in the list called COLLECTIONS_TO_UPLOAD, add a list of collection config folders.
Run python export_collections.py
This will upload zipped json files to the S3 bucket here.
Go to the collections admin page here.
Select collection ONE OR TWO AT A TIME (otherwise celery might throw an error because each process tries to write to postgres at the same time). Then from the dropdown menu click on Import candidate URLs

Follow steps, 1, 2, 3 from above.

Run python export_whole_index.py
This will upload one large zipped json file to the S3 bucket here.
Log into the sde_indexing_helper server and navigate to the folder sde_indexing_helper. Run drun /bin/bash. Then inside the docker run python manage.py load_urls_from_s3

If you spot something in the pipeline or this documentation that can be optimized, be the change you wish to see in the world. :D