diff --git a/README.md b/README.md index d712d2a49..e24ab3be7 100644 --- a/README.md +++ b/README.md @@ -22,11 +22,10 @@ This project uses Docker compose to setup and run all the necessary components. cd data-hub-api ``` -2. Create `.env` files from `sample.env` +2. Create a `.env` file ```shell - cp sample.env .env - cp config/settings/sample.env config/settings/.env + cp sample-docker-dev.env .env ``` If you're working with data-hub-frontend and mock-sso, `DJANGO_SUPERUSER_SSO_EMAIL_USER_ID` should be the same as @@ -87,11 +86,10 @@ There is now a `make` command to bring up the three environments on a single doc cd data-hub-api ``` -2. Create `.env` files from `sample.env` +2. Create a `.env` file ```shell - cp sample.env .env - cp config/settings/sample.env config/settings/.env + cp sample-docker-dev.env .env ``` Ensure `DJANGO_SUPERUSER_SSO_EMAIL_USER_ID` is the same as `MOCK_SSO_EMAIL_USER_ID` @@ -142,7 +140,7 @@ There is now a `make` command to bring up the three environments on a single doc Dependencies: - Python 3.10.x -- PostgreSQL 12 +- PostgreSQL 16.x - redis 6.x - OpenSearch 1.x @@ -171,7 +169,7 @@ Dependencies: brew install libpq ``` -4. Install _postgres_, if not done already, as this is required by **psycopg2** in the requirements below +4. Install Postgres On Ubuntu: @@ -183,52 +181,85 @@ Dependencies: ```shell brew install postgresql + brew services start postgresql ``` -5. Create and activate the virtualenv: +5. Setup the postgres user and database + + On macOS: + ```shell + createdb + ``` + + ```shell + psql + CREATE DATABASE datahub; + CREATE DATABASE test_datahub; + CREATE USER datahubuser WITH SUPERUSER PASSWORD 'datahubpassword'; + \q + ``` + + nb. SUPERUSER is needed to create 'citext' extension + + On macOS: + + The default table collation ('C') is case and accent sensitive whereas the default ('en_US.utf8') on Debian (which the Python Docker image is based on) is not. See [this](https://postgresql.verite.pro/blog/2019/10/14/nondeterministic-collations.html) for more information on collation types. Without this a couple of tests will fail. + + ```shell + psql + update pg_database set datcollate='en_US.UTF-8', datctype='en_US.UTF-8' where datname = 'test_datahub'; + update pg_database set datcollate='en_US.UTF-8', datctype='en_US.UTF-8' where datname = 'datahub'; + \q + brew services restart postgresql + ``` + +6. Create and activate the virtualenv: ```shell python3.10 -m venv env source env/bin/activate pip install -U pip - or - formally to make sure you have the same version as what is used for cloudfoundry, use buildpack to install the same version e.g. https://github.com/cloudfoundry/python-buildpack/releases e.g. 22.1.2 - python -m pip install pip==22.1.2 ``` -6. Install the dependencies: +7. Install the dependencies: ```shell pip install -r requirements-dev.txt ``` -7. Create an `.env` settings file (it’s gitignored by default): +8. Create a `.env` file (it’s gitignored by default): ```shell - cp config/settings/sample.env config/settings/.env + cp sample-local-dev.env .env ``` -8. Set `DOCKER_DEV=False` and `LOCAL_DEV=True` in `.env` - -9. Make sure you have OpenSearch running locally. If you don't, you can run one in Docker: +11. Make sure you have OpenSearch running locally. ```shell - docker run -p 9200:9200 -e "http.host=0.0.0.0" -e "transport.host=127.0.0.1" -e "plugins.security.disabled=true" opensearchproject/opensearch:1.2.4 + brew install opensearch + brew services start opensearch ``` + +12. Install and run redis -10. Make sure you have redis running locally and that the REDIS_BASE_URL in your `.env` is up-to-date. + ```shell + brew install redis + brew services start redis + ``` -11. Populate the databases and initialise OpenSearch: +13. Populate the databases and initialise OpenSearch: ```shell ./manage.py migrate ./manage.py migrate_search - ./manage.py loadinitialmetadata + # Force is required as `migrate` already loads some metadata + # but loading fixtures will fail without this + ./manage.py loadinitialmetadata --force ./manage.py createinitialrevisions ``` -12. Optionally, you can load some test data: +14. Optionally, you can load some test data: ```shell ./manage.py loaddata fixtures/test_data.yaml @@ -238,7 +269,7 @@ Dependencies: and hence the loaded records won‘t be returned by search endpoints until RQ is started and the queued tasks have run. -13. Create a superuser: +15. Create a superuser: ```shell ./manage.py createsuperuser @@ -246,19 +277,23 @@ Dependencies: (You can enter any valid email address as the username and SSO email user ID.) -14. Start the server: +16. Start the server: ```shell ./manage.py runserver ``` -15. Start RQ (Redis Queue): +17. Start RQ (Redis Queues): + Start short running queue working ```shell - python rq/rq-worker.py + ./rq-run.sh short-running-worker.py ``` - Note that in production the cron-scheduler:1/1, short-running-working:4/4, long-running-worker:4/4 are run in separate instances . + Start long running queue working + ```shell + ./rq-run.sh long-running-worker.py + ``` ## API documentation @@ -600,4 +635,4 @@ Example of environment variables for updating the interaction notification email ``` INTERACTION_NOTIFICATION_API_KEY= EXPORT_NOTIFICATION_NEW_INTERACTION_TEMPLATE_ID= -``` \ No newline at end of file +``` diff --git a/config/settings/sample.env b/config/settings/sample.env deleted file mode 100644 index e26a185f0..000000000 --- a/config/settings/sample.env +++ /dev/null @@ -1,36 +0,0 @@ -DJANGO_SECRET_KEY=dev -DEBUG=True -DATABASE_CREDENTIALS={"username": "postgres", "password": "datahub", "engine": "postgres", "port": 5432, "dbname": "datahub", "host": "localhost", "dbInstanceIdentifier": "db-instance"} -# Set this if you are using a host for OpenSearch other than localhost -# OpenSearch related tests may hang otherwise -#TEST_OPENSEARCH_SERVER=es:9200 -OPENSEARCH_INDEX_PREFIX=test -DOCKER_DEV=True -DEFAULT_BUCKET_AWS_DEFAULT_REGION=eu-west-2 -DEFAULT_BUCKET_AWS_ACCESS_KEY_ID=foo -DEFAULT_BUCKET_AWS_SECRET_ACCESS_KEY=bar -DEFAULT_BUCKET=baz -SSO_ENABLED=True -STAFF_SSO_BASE_URL=http://sso/ -STAFF_SSO_AUTH_TOKEN=sso-token -REDIS_BASE_URL=redis://localhost:6379 -ACTIVITY_STREAM_ACCESS_KEY_ID=some-id -ACTIVITY_STREAM_SECRET_ACCESS_KEY=some-secret -ACTIVITY_STREAM_OUTGOING_URL=http://activity.stream/ -ACTIVITY_STREAM_OUTGOING_ACCESS_KEY_ID=some-outgoing-id -ACTIVITY_STREAM_OUTGOING_SECRET_ACCESS_KEY=some-outgoing-secret -MARKET_ACCESS_ACCESS_KEY_ID=market-access-id -MARKET_ACCESS_SECRET_ACCESS_KEY=market-access-key -PAAS_IP_ALLOWLIST=1.2.3.4 -# Set this when using local environment -#DISABLE_PAAS_IP_CHECK=true -DATA_HUB_FRONTEND_ACCESS_KEY_ID=frontend-key-id -DATA_HUB_FRONTEND_SECRET_ACCESS_KEY=frontend-key - -ADMIN_OAUTH2_ENABLED=True -ADMIN_OAUTH2_BASE_URL=http://localhost:8080 -ADMIN_OAUTH2_TOKEN_FETCH_PATH=/o/token/ -ADMIN_OAUTH2_USER_PROFILE_PATH=/api/v1/user/me/ -ADMIN_OAUTH2_AUTH_PATH=/o/authorize/ -ADMIN_OAUTH2_CLIENT_ID=oauth2-client-id -ADMIN_OAUTH2_CLIENT_SECRET=oauth2-client-secret \ No newline at end of file diff --git a/sample.env b/sample-docker-dev.env similarity index 56% rename from sample.env rename to sample-docker-dev.env index d225d4722..0142f7cd0 100644 --- a/sample.env +++ b/sample-docker-dev.env @@ -1,87 +1,88 @@ -# Environment variables specific to usage with docker-compose -DATABASE_CREDENTIALS={"username": "postgres", "password": "datahub", "engine": "postgres", "port": 5432, "dbname": "datahub", "host": "postgres", "dbInstanceIdentifier": "db-instance"} -POSTGRES_URL=tcp://postgres:5432 +### Local dev ### +PAAS_IP_WHITELIST=1.2.3.4 +DISABLE_PAAS_IP_CHECK=true DEBUG=True -DJANGO_SECRET_KEY=changeme -DJANGO_SETTINGS_MODULE=config.settings.local -COV_TOKEN=${COV_TOKEN} + +### Postgres ### +DATABASE_URL=postgresql://postgres:datahub@postgres/datahub +DATABASE_CREDENTIALS={"username": "postgres", "password": "datahub", "engine": "postgres", "port": 5432, "dbname": "datahub", "host": "postgres", "dbInstanceIdentifier": "db-instance"} + +### Redis ### +REDIS_BASE_URL=redis://redis:6379 + +### OpenSearch ### OPENSEARCH_URL=http://opensearch:9200 OPENSEARCH_INDEX_PREFIX=test_index -REDIS_BASE_URL=redis://redis:6379 -DEFAULT_BUCKET_AWS_DEFAULT_REGION=eu-west-2 -DEFAULT_BUCKET_AWS_ACCESS_KEY_ID=foo -DEFAULT_BUCKET_AWS_SECRET_ACCESS_KEY=bar -DEFAULT_BUCKET=baz +ES_APM_ENABLED=False + +### Django ### +DJANGO_SECRET_KEY=changeme +DJANGO_SETTINGS_MODULE=config.settings.local +# OAuth2 settings for Django Admin access +ADMIN_OAUTH2_ENABLED=False +ADMIN_OAUTH2_TOKEN_FETCH_URL=http://localhost:8100/o/token +ADMIN_OAUTH2_USER_PROFILE_URL= +ADMIN_OAUTH2_AUTH_URL=http://localhost:8100/o/authorize +ADMIN_OAUTH2_CLIENT_ID=oauth2-client-id +ADMIN_OAUTH2_CLIENT_SECRET=oauth2-secret-id +ADMIN_OAUTH2_REDIRECT_URL=http://localhost:8000/oauth/callback +# If you’re working with data-hub-frontend and mock-sso, DJANGO_SUPERUSER_EMAIL should +# be the same as MOCK_SSO_USERNAME in mock-sso’s .env file, and +# DJANGO_SUPERUSER_SSO_EMAIL_USER_ID the same as DJANGO_SUPERUSER_EMAIL in data-hub-api .env file otherwise the user may not exist +DJANGO_SUPERUSER_EMAIL=test@gov.uk +DJANGO_SUPERUSER_PASSWORD=foobarbaz +DJANGO_SUPERUSER_SSO_EMAIL_USER_ID=test@gov.uk +# If SUPERUSER_ACCESS_TOKEN is given a value, an access token for the +# superuser with that value will be created when the server comes up. +# The superuser should have an SSO email user ID set for this to work. +SUPERUSER_ACCESS_TOKEN=ditStaffToken + +### Email Domain Allowlist ### +DIT_EMAIL_DOMAINS=trade.gov.uk,digital.trade.gov.uk + +### PyTest ### +ALLOW_TEST_FIXTURE_SETUP=True + +### Disable output buffering in Python ### +PYTHONUNBUFFERED=1 + +### AWS ### +AWS_DEFAULT_REGION=eu-west-2 +AWS_ACCESS_KEY_ID=foo +AWS_SECRET_ACCESS_KEY=bar +DEFAULT_BUCKET=upload.datahub.dev.uktrade.io SSO_ENABLED=True STAFF_SSO_BASE_URL=http://mock-sso:8080/ STAFF_SSO_AUTH_TOKEN=sso-token + +### Activity Stream config ### ACTIVITY_STREAM_ACCESS_KEY_ID=some-id ACTIVITY_STREAM_SECRET_ACCESS_KEY=some-secret ACTIVITY_STREAM_OUTGOING_URL=http://activity.stream/ ACTIVITY_STREAM_OUTGOING_ACCESS_KEY_ID=some-outgoing-id ACTIVITY_STREAM_OUTGOING_SECRET_ACCESS_KEY=some-outgoing-secret + +### Market Access service config ### MARKET_ACCESS_ACCESS_KEY_ID=market-access-id MARKET_ACCESS_SECRET_ACCESS_KEY=market-access-key -PAAS_IP_ALLOWLIST=1.2.3.4 -# Set this when using local environment -# DISABLE_PAAS_IP_CHECK=true -DIT_EMAIL_DOMAINS=trade.gov.uk,digital.trade.gov.uk +### Data Flow API Config ### DATA_FLOW_API_ACCESS_KEY_ID=data-flow-api-id DATA_FLOW_API_SECRET_ACCESS_KEY=data-flow-api-access-key -DATA_HUB_ENQUIRY_MGMT_HAWK_ID=data-hub-enquiry-mgmt-hawk-id -DATA_HUB_ENQUIRY_MGMT_HAWK_SECRET_KEY=data-hub-enquiry-mgmt-hawk-secret-key + +### Datahub Frontend config ### DATA_HUB_FRONTEND_ACCESS_KEY_ID=frontend-key-id DATA_HUB_FRONTEND_SECRET_ACCESS_KEY=frontend-key -# Determines the docker-compose project - by default, containers with the same -# project name share a network and are able to communicate with eachother -COMPOSE_PROJECT_NAME=data-hub -# Some extra ENV variables to make superuser creation easier on docker copies -# If you're working with data-hub-frontend and mock-sso, DJANGO_SUPERUSER_EMAIL should -# be the same as MOCK_SSO_USERNAME in mock-sso's .env file, and -# DJANGO_SUPERUSER_SSO_EMAIL_USER_ID the same as DJANGO_SUPERUSER_EMAIL in data-hub-api .env file otherwise the user may not exist -DJANGO_SUPERUSER_EMAIL=test@gov.uk -DJANGO_SUPERUSER_PASSWORD=foobarbaz -DJANGO_SUPERUSER_SSO_EMAIL_USER_ID=test@gov.uk - -# If SUPERUSER_ACCESS_TOKEN is given a value, an access token for the -# superuser with that value will be created when the container comes up. -# The superuser should have an SSO email user ID set for this to work. -SUPERUSER_ACCESS_TOKEN=ditStaffToken - -# Settings for Elasticsearch APM. -ES_APM_ENABLED=False -# ES_APM_SERVICE_NAME=datahub -# ES_APM_SECRET_TOKEN= -# ES_APM_SERVER_URL=http://localhost:8200 -# ES_APM_ENVIRONMENT='develop' - -# OAuth2 settings for Django Admin access -ADMIN_OAUTH2_ENABLED=False -ADMIN_OAUTH2_TOKEN_FETCH_URL=http://localhost:8100/o/token -ADMIN_OAUTH2_USER_PROFILE_URL= -ADMIN_OAUTH2_AUTH_URL=http://localhost:8100/o/authorize -ADMIN_OAUTH2_CLIENT_ID=oauth2-client-id -ADMIN_OAUTH2_CLIENT_SECRET=oauth2-secret-id -ADMIN_OAUTH2_REDIRECT_URL=http://localhost:8000/oauth/callback - -# dnb-service settings -DNB_SERVICE_BASE_URL=http://api-dnb:8000/api/ -# Generated through django-rest-framework on dnb-service -DNB_SERVICE_TOKEN=cc373a2a49ce7143817a9d036fa2a0be92da0d6a -DNB_MAX_COMPANIES_IN_TREE_COUNT=1000 - -# Python specific env vars -PYTHONUNBUFFERED=1 +### Dun & Bradstreet service Config ### +DNB_SERVICE_BASE_URL=https://dnb-service.dev.datahub.uktrade.digital/api/ +DNB_SERVICE_TOKEN=some-token -# Consent Service settings +### Consent service config ### CONSENT_SERVICE_BASE_URL=http://mock-third-party-services:8555 CONSENT_SERVICE_HAWK_ID=dummyId CONSENT_SERVICE_HAWK_KEY=dummyKey -DATAHUB_NOTIFICATION_API_KEY= -OMIS_NOTIFICATION_API_KEY= -INVESTMENT_NOTIFICATION_API_KEY= -INTERACTION_NOTIFICATION_API_KEY= -EXPORT_WIN_NOTIFICATION_API_KEY= +### Overseas Market Introduction Service config ### +OMIS_PUBLIC_ACCESS_KEY_ID=access-key-id +OMIS_PUBLIC_SECRET_ACCESS_KEY=secret-access-key diff --git a/sample-local-dev.env b/sample-local-dev.env new file mode 100644 index 000000000..6005f5dc4 --- /dev/null +++ b/sample-local-dev.env @@ -0,0 +1,89 @@ +### Local dev ### +DOCKER_DEV=False +PAAS_IP_WHITELIST=1.2.3.4 +DISABLE_PAAS_IP_CHECK=true +DEBUG=True + +### Postgres ### +DATABASE_URL=postgresql://datahubuser:datahub@localhost/datahub +DATABASE_CREDENTIALS={"username": "datahubuser", "password": "datahubpassword", "engine": "postgres", "port": 5432, "dbname": "datahub", "host": "postgres", "dbInstanceIdentifier": "db-instance"} + +### Redis ### +REDIS_BASE_URL=redis://localhost:6379 + +### OpenSearch ### +OPENSEARCH_URL=http://localhost:9200 +OPENSEARCH_INDEX_PREFIX=test_index +ES_APM_ENABLED=False + +### Django ### +DJANGO_SECRET_KEY=changeme +DJANGO_SETTINGS_MODULE=config.settings.local +# OAuth2 settings for Django Admin access +ADMIN_OAUTH2_ENABLED=False +ADMIN_OAUTH2_TOKEN_FETCH_URL=http://localhost:8100/o/token +ADMIN_OAUTH2_USER_PROFILE_URL= +ADMIN_OAUTH2_AUTH_URL=http://localhost:8100/o/authorize +ADMIN_OAUTH2_CLIENT_ID=oauth2-client-id +ADMIN_OAUTH2_CLIENT_SECRET=oauth2-secret-id +ADMIN_OAUTH2_REDIRECT_URL=http://localhost:8000/oauth/callback +# If you’re working with data-hub-frontend and mock-sso, DJANGO_SUPERUSER_EMAIL should +# be the same as MOCK_SSO_USERNAME in mock-sso’s .env file, and +# DJANGO_SUPERUSER_SSO_EMAIL_USER_ID the same as DJANGO_SUPERUSER_EMAIL in data-hub-api .env file otherwise the user may not exist +DJANGO_SUPERUSER_EMAIL=test@gov.uk +DJANGO_SUPERUSER_PASSWORD=foobarbaz +DJANGO_SUPERUSER_SSO_EMAIL_USER_ID=test@gov.uk +# If SUPERUSER_ACCESS_TOKEN is given a value, an access token for the +# superuser with that value will be created when the server comes up. +# The superuser should have an SSO email user ID set for this to work. +SUPERUSER_ACCESS_TOKEN=ditStaffToken + +### Email Domain Allowlist ### +DIT_EMAIL_DOMAINS=trade.gov.uk,digital.trade.gov.uk + +### PyTest ### +ALLOW_TEST_FIXTURE_SETUP=True + +### Disable output buffering in Python ### +PYTHONUNBUFFERED=1 + +### AWS ### +AWS_DEFAULT_REGION=eu-west-2 +AWS_ACCESS_KEY_ID=foo +AWS_SECRET_ACCESS_KEY=bar +DEFAULT_BUCKET=upload.datahub.dev.uktrade.io +SSO_ENABLED=True +STAFF_SSO_BASE_URL=http://mock-sso:8080/ +STAFF_SSO_AUTH_TOKEN=sso-token + +### Activity Stream config ### +ACTIVITY_STREAM_ACCESS_KEY_ID=some-id +ACTIVITY_STREAM_SECRET_ACCESS_KEY=some-secret +ACTIVITY_STREAM_OUTGOING_URL=http://activity.stream/ +ACTIVITY_STREAM_OUTGOING_ACCESS_KEY_ID=some-outgoing-id +ACTIVITY_STREAM_OUTGOING_SECRET_ACCESS_KEY=some-outgoing-secret + +### Market Access service config ### +MARKET_ACCESS_ACCESS_KEY_ID=market-access-id +MARKET_ACCESS_SECRET_ACCESS_KEY=market-access-key + +### Data Flow API Config ### +DATA_FLOW_API_ACCESS_KEY_ID=data-flow-api-id +DATA_FLOW_API_SECRET_ACCESS_KEY=data-flow-api-access-key + +### Datahub Frontend config ### +DATA_HUB_FRONTEND_ACCESS_KEY_ID=frontend-key-id +DATA_HUB_FRONTEND_SECRET_ACCESS_KEY=frontend-key + +### Dun & Bradstreet service Config ### +DNB_SERVICE_BASE_URL=https://dnb-service.dev.datahub.uktrade.digital/api/ +DNB_SERVICE_TOKEN=some-token + +### Consent service config ### +CONSENT_SERVICE_BASE_URL=http://mock-third-party-services:8555 +CONSENT_SERVICE_HAWK_ID=dummyId +CONSENT_SERVICE_HAWK_KEY=dummyKey + +### Overseas Market Introduction Service config ### +OMIS_PUBLIC_ACCESS_KEY_ID=access-key-id +OMIS_PUBLIC_SECRET_ACCESS_KEY=secret-access-key