Merge pull request #970 from cmu-delphi/release/delphi-epidata-0.3.21

Release Delphi Epidata 0.3.21

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.3.20
+current_version = 0.3.21
 commit = False
 tag = False
 

README.md

@@ -8,7 +8,51 @@ This is the home of [Delphi](https://delphi.cmu.edu/)'s epidemiological data
 API. See our [API documentation](https://cmu-delphi.github.io/delphi-epidata/)
 for details on the available data sets, APIs, and clients.
 
-## COVIDcast
+## Development Quickstart
+
+Requires: Docker, possibly sudo access (depending on your Docker installation and OS).
+
+In the directory where you want to work run the following:
+
+```sh
+# Make folder structure, download dependent repos, and symlink Makefile
+$ curl "https://raw.githubusercontent.com/cmu-delphi/delphi-epidata/dev/dev/local/install.sh" | bash
+```
+
+You should now have the following directory structure:
+
+```sh
+├── driver
+│   ├── .dockerignore -> repos/delphi/delphi-epidata/dev/local/.dockerignore
+│   ├── Makefile -> repos/delphi/delphi-epidata/dev/local/Makefile
+│   ├── repos
+│   │   └── delphi
+│   │       ├── delphi-epidata
+│   │       ├── flu-contest
+│   │       ├── github-deploy-repo
+│   │       ├── nowcast
+│   │       ├── operations
+│   │       └── utils
+```
+
+and you should now be in the `driver` directory.
+You can now execute make commands
+
+```sh
+# Create all docker containers: db, web, and python
+$ [sudo] make all
+
+# Run tests
+$ [sudo] make test
+
+# To drop into debugger on error
+$ [sudo] make test pdb=1
+
+# To test only a subset of tests
+$ [sudo] make test test=repos/delphi/delphi-epidata/integrations/acquisition
+```
+
+# COVIDcast
 
 At the present, our primary focus is developing and expanding the
 [COVIDcast API](https://cmu-delphi.github.io/delphi-epidata/api/covidcast.html),

dev/local/.dockerignore

@@ -0,0 +1,6 @@
+# Ignore everything by default
+*
+# Don't ignore repos dir
+!repos
+# Ignore everything to do with git
+**/*.git

dev/local/Makefile

@@ -0,0 +1,140 @@
+# Docker control panel for delphi-epidata development.
+#
+# Usage: make <command> [pdb=1] [test=<test-subdir>]
+#
+# Assumes you have installed your environment using
+# delphi-epidata/dev/local/install.sh.
+#
+# Checks for the delphi-net bridge and creates if it doesn't exist.
+#
+# Creates all prereq images (delphi_database, delphi_python) only if they don't
+# exist. If you need to rebuild a prereq, you're probably doing something
+# complicated, and can figure out the rebuild command on your own.
+# 
+# 
+# Commands:
+#            
+#   web:     Stops currently-running delphi_web_epidata instances, if any.
+#            Rebuilds delphi_web_epidata image.
+#            Runs image in the background and pipes stdout to a log file.
+# 
+#   db:      Stops currently-running delphi_database_epidata instances, if any.
+#            Rebuilds delphi_database_epidata image.
+#            Runs image in the background and pipes stdout to a log file.
+#            Blocks until database is ready to receive connections.
+# 
+#   python:  Rebuilds delphi_web_python image. You shouldn't need to do this
+#            often; only if you are installing a new environment, or have
+#            made changes to delphi-epidata/dev/docker/python/Dockerfile.
+#
+#   all:     Runs the commands 'web' 'db' and 'python'.
+#
+#   test:    Runs test and integrations in delphi-epidata. If test 
+#            optional arg is provided, then only the tests in that subdir
+#            are run.
+#
+#   clean:   Cleans up dangling Docker images.
+#
+# 
+# Optional arguments:
+#   pdb=1         Drops you into debug mode upon test failure, if running tests.
+#   test=         Only runs tests in the directories provided here, e.g.
+#                 repos/delphi/delphi-epidata/tests/acquisition/covidcast
+
+
+# Set optional argument defaults
+ifdef pdb
+	override pdb=--pdb
+else
+	pdb=
+endif
+
+ifndef test
+	test=repos/delphi/delphi-epidata/tests repos/delphi/delphi-epidata/integrations
+endif
+
+SHELL:=/bin/sh
+
+# Get the Makefile's absolute path: https://stackoverflow.com/a/324782/4784655
+# (if called from a symlink, the path is the location of the symlink)
+CWD:=$(dir $(abspath $(lastword $(MAKEFILE_LIST))))
+NOW:=$(shell date "+%Y-%m-%d")
+LOG_WEB:=delphi_web_epidata_$(NOW).log
+LOG_DB:=delphi_database_epidata_$(NOW).log
+WEB_CONTAINER_ID:=$(shell docker ps -q --filter 'name=delphi_web_epidata')
+DATABASE_CONTAINER_ID:=$(shell docker ps -q --filter 'name=delphi_database_epidata')
+
+
+.PHONY=web
+web:
+	@# Stop container if running
+	@if [ $(WEB_CONTAINER_ID) ]; then\
+		docker stop $(WEB_CONTAINER_ID);\
+	fi
+
+	@# Setup virtual network if it doesn't exist
+	@docker network ls | grep delphi-net || docker network create --driver bridge delphi-net
+
+	@# Build the web_epidata image
+	@cd repos/delphi/delphi-epidata;\
+		docker build -t delphi_web_epidata -f ./devops/Dockerfile .;\
+		cd ../../../
+
+	@# Run the web server
+	@docker run --rm -p 127.0.0.1:10080:80 \
+		--env "SQLALCHEMY_DATABASE_URI=mysql+mysqldb://user:pass@delphi_database_epidata:3306/epidata" \
+		--env "FLASK_SECRET=abc" --env "FLASK_PREFIX=/epidata" \
+		--network delphi-net --name delphi_web_epidata \
+		delphi_web_epidata >$(LOG_WEB) 2>&1 &
+
+.PHONY=db
+db:
+	@# Stop container if running
+	@if [ $(DATABASE_CONTAINER_ID) ]; then\
+		docker stop $(DATABASE_CONTAINER_ID);\
+	fi
+
+	@# Only build prereqs if we need them
+	@docker images delphi_database | grep delphi || \
+		docker build -t delphi_database -f repos/delphi/operations/dev/docker/database/Dockerfile .
+
+	@# Build the database_epidata image
+	@docker build -t delphi_database_epidata \
+		-f repos/delphi/delphi-epidata/dev/docker/database/epidata/Dockerfile .
+
+	@# Run the database
+	@docker run --rm -p 127.0.0.1:13306:3306 \
+		--network delphi-net --name delphi_database_epidata \
+		delphi_database_epidata >$(LOG_DB) 2>&1 &
+
+	@# Block until DB is ready
+	@while true; do \
+		sed -n '/Temporary server stopped/,/mysqld: ready for connections/p' $(LOG_DB) | grep "ready for connections" && break; \
+		tail -1 $(LOG_DB); \
+		sleep 1; \
+	done
+
+.PHONY=py
+py:
+	@# Build the python image
+	@docker build -t delphi_python \
+		-f repos/delphi/operations/dev/docker/python/Dockerfile .
+
+	@docker build -t delphi_web_python \
+		-f repos/delphi/delphi-epidata/dev/docker/python/Dockerfile .
+
+.PHONY=all
+all: web db py
+
+.PHONY=test
+test: 
+	@docker run -i --rm --network delphi-net \
+		--mount type=bind,source=$(CWD)repos/delphi/delphi-epidata,target=/usr/src/app/repos/delphi/delphi-epidata,readonly \
+		--mount type=bind,source=$(CWD)repos/delphi/delphi-epidata/src,target=/usr/src/app/delphi/epidata,readonly \
+		--env "SQLALCHEMY_DATABASE_URI=mysql+mysqldb://user:pass@delphi_database_epidata:3306/epidata" \
+		--env "FLASK_SECRET=abc" \
+		delphi_web_python python -m pytest --import-mode importlib $(pdb) $(test) | tee test_output_$(NOW).log
+
+.PHONY=clean
+clean:
+	@docker images -f "dangling=true" -q | xargs docker rmi >/dev/null 2>&1

dev/local/install.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+# Bootstrap delphi-epidata development
+#
+# Downloads the repos needed for local delphi-epidata development into current dir 
+# and provides a Makefile with Docker control commands.
+#
+# Creates the directory structure:
+#
+#   driver/
+#     .dockerignore
+#     Makefile
+#     repos/
+#       delphi/
+#         operations/
+#         delphi-epidata/
+#         utils/
+#         flu-contest/
+#         nowcast/
+#         github-deploy-repo/
+#       undefx/
+#         py3tester/
+#         undef-analysis/
+#
+# Leaves you in driver, the main workdir.
+#
+
+
+mkdir -p driver/repos/delphi
+cd driver/repos/delphi
+git clone https://github.com/cmu-delphi/operations
+git clone https://github.com/cmu-delphi/delphi-epidata
+git clone https://github.com/cmu-delphi/utils
+git clone https://github.com/cmu-delphi/flu-contest
+git clone https://github.com/cmu-delphi/nowcast
+git clone https://github.com/cmu-delphi/github-deploy-repo
+cd ../../
+
+mkdir -p repos/undefx
+cd repos/undefx
+git clone https://github.com/undefx/py3tester
+git clone https://github.com/undefx/undef-analysis
+cd ../../
+
+ln -s repos/delphi/delphi-epidata/dev/local/Makefile
+ln -s repos/delphi/delphi-epidata/dev/local/.dockerignore

docs/Gemfile.lock

@@ -246,7 +246,7 @@ GEM
     thread_safe (0.3.6)
     typhoeus (1.4.0)
       ethon (>= 0.9.0)
-    tzinfo (1.2.9)
+    tzinfo (1.2.10)
       thread_safe (~> 0.1)
     tzinfo-data (1.2021.1)
       tzinfo (>= 1.0.0)

docs/api/covidcast-signals/chng.md

@@ -191,7 +191,7 @@ all of them, and so this source only represents those encounters known to
 them. Their coverage may vary across the United States, but they report on about
 45% of all doctor's visits nationally.
 
-Standard errors are not available for this data source.
+Standard errors and sample sizes are not available for this data source.
 
 Due to changes in medical-seeking behavior on holidays, this data source has
 upward spikes in the fraction of doctor's visits that are COVID-related around

docs/api/covidcast-signals/doctor-visits.md

@@ -160,7 +160,7 @@ outpatient doctor's visits, but not all of them, and so this source only
 represents those visits known to them. Their coverage may vary across the United
 States.
 
-Standard errors are not available for this data source.
+Standard errors and sample sizes are not available for this data source.
 
 Due to changes in medical-seeking behavior on holidays, this data source has
 upward spikes in the fraction of doctor's visits that are COVID-related around

docs/api/covidcast-signals/dsew-cpr.md

@@ -63,6 +63,8 @@ This data source is susceptible to large corrections that can create strange dat
 
 Not all CPRs have the same lag between the CPR date (listed in the filename) and the date for a particular signal.
 
+Standard errors and sample sizes are not applicable to these metrics.
+
 ### Differences with HHS reports
 
 An analysis comparing the 

docs/api/covidcast-signals/fb-survey.md

@@ -19,17 +19,18 @@ grand_parent: COVIDcast Epidata API
 
 This data source is based on the [COVID-19 Trends and Impact Survey (CTIS)](../../symptom-survey/)
 run by the Delphi group at Carnegie Mellon.
-Facebook directs a random sample of its users to these surveys, which are
-voluntary. Users age 18 or older are eligible to complete the surveys, and
+Facebook directed a random sample of its users to these surveys, which were
+voluntary. Users age 18 or older were eligible to complete the surveys, and
 their survey responses are held by CMU and are sharable with other health
 researchers under a data use agreement. No individual survey responses are
 shared back to Facebook. See our [surveys
 page](https://delphi.cmu.edu/covid19/ctis/) for more detail about how the
 surveys work and how they are used outside the COVIDcast API.
 
 As of November 2021, the average number of Facebook survey responses we
-receive each day is about 40,000, and the total number of survey responses we
-have received is over 25 million.
+received each day was about 40,000. The survey ran from April 6, 2020 to June
+25, 2022, collecting roughly 29.5 million responses in the United States in that
+time.
 
 We produce several sets of signals based on the survey data, listed and
 described in the sections below:

docs/api/covidcast-signals/google-symptoms.md

@@ -116,6 +116,8 @@ popularity of symptoms in searches within each geographical region individually.
 This means that the resulting values of symptom set popularity are **NOT**
 comparable across geographic regions, while the values of different symptom sets are comparable within the same location.
 
+Standard errors and sample sizes are not available for this data source.
+
 More details about the limitations of this dataset are available in [Google's Search
 Trends symptoms dataset documentation](https://storage.googleapis.com/gcp-public-data-symptom-search/COVID-19%20Search%20Trends%20symptoms%20dataset%20documentation%20.pdf).
 

docs/api/covidcast-signals/hhs.md

@@ -32,15 +32,16 @@ adult and pediatric COVID-19 hospital admissions. This sum is used as the
 We also include influenza hospital admissions.
 
 
-| Signal | 7-day average signal | Geography | Resolution | Description |
-| --- | --- | --- | --- | --- |
-| `confirmed_admissions_covid_1d` | `confirmed_admissions_covid_1d_7dav`| state | 1 day | Sum of adult and pediatric confirmed COVID-19 hospital admissions occurring each day. <br/> **Earliest date available:** 2019-12-31 |
-| `confirmed_admissions_covid_1d_prop` | `confirmed_admissions_covid_1d_prop_7dav`| state | 1 day | Sum of adult and pediatric confirmed COVID-19 hospital admissions occurring each day, per 100,000 population. <br/> **Earliest date available:** 2019-12-31 |
-| `sum_confirmed_suspected_admissions_covid_1d` | `sum_confirmed_suspected_admissions_covid_1d_7dav` | state | 1 day | Sum of adult and pediatric confirmed and suspected COVID-19 hospital admissions occurring each day. <br/> **Earliest date available:** 2019-12-31 |
-| `sum_confirmed_suspected_admissions_covid_1d_prop` | `sum_confirmed_suspected_admissions_covid_1d_prop_7dav` | state | 1 day | Sum of adult and pediatric confirmed and suspected COVID-19 hospital admissions occurring each day, per 100,000 population. <br/> **Earliest date available:** 2019-12-31 |
-| `confirmed_admissions_influenza_1d` | `confirmed_admissions_influenza_1d_7dav` | state | 1 day | All confirmed influenza hospital admissions occurring each day. We made this signal available November 1, 2021. <br/> **Earliest issue available:** 2021-09-20 <br/> **Earliest date available:** 2020-01-02 |
-| `confirmed_admissions_influenza_1d_prop` | `confirmed_admissions_influenza_1d_prop_7dav` | state | 1 day | All confirmed influenza hospital admissions occurring each day, per 100,000 population. We made this signal available November 1, 2021. <br/> **Earliest issue available:** 2021-09-20 <br/> **Earliest date available:** 2020-01-02 |
+| Signal and 7-day average signal                                                                                | Description                                                                                                                                                                                                                          |
+|:---------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `confirmed_admissions_covid_1d` and `confirmed_admissions_covid_1d_7dav`                                       | Sum of adult and pediatric confirmed COVID-19 hospital admissions occurring each day. <br/> **Earliest date available:** 2019-12-31                                                                                                  |
+| `confirmed_admissions_covid_1d_prop` and `confirmed_admissions_covid_1d_prop_7dav`                             | Sum of adult and pediatric confirmed COVID-19 hospital admissions occurring each day, per 100,000 population. <br/> **Earliest date available:** 2019-12-31                                                                          |
+| `sum_confirmed_suspected_admissions_covid_1d` and `sum_confirmed_suspected_admissions_covid_1d_7dav`           | Sum of adult and pediatric confirmed and suspected COVID-19 hospital admissions occurring each day. <br/> **Earliest date available:** 2019-12-31                                                                                    |
+| `sum_confirmed_suspected_admissions_covid_1d_prop` and <br/> `sum_confirmed_suspected_admissions_covid_1d_prop_7dav` | Sum of adult and pediatric confirmed and suspected COVID-19 hospital admissions occurring each day, per 100,000 population. <br/> **Earliest date available:** 2019-12-31                                                            |
+| `confirmed_admissions_influenza_1d` and `confirmed_admissions_influenza_1d_7dav`                               | All confirmed influenza hospital admissions occurring each day. We made this signal available November 1, 2021. <br/> **Earliest issue available:** 2021-09-20 <br/> **Earliest date available:** 2020-01-02                         |
+| `confirmed_admissions_influenza_1d_prop` and `confirmed_admissions_influenza_1d_prop_7dav`                     | All confirmed influenza hospital admissions occurring each day, per 100,000 population. We made this signal available November 1, 2021. <br/> **Earliest issue available:** 2021-09-20 <br/> **Earliest date available:** 2020-01-02 |
 
+*for all the above signals & 7-day average signals, their geography is state, and resolution is 1 day.
 
 The 7-day average signals are computed by Delphi by calculating
 moving averages of the preceding 7 days, so e.g. the signal for June 7 is the
@@ -96,6 +97,8 @@ psychiatric and rehabilitation facilities, Indian Health Service (IHS)
 facilities, U.S. Department of Veterans Affairs (VA) facilities, Defense Health
 Agency (DHA) facilities, and religious non-medical facilities.
 
+Standard errors and sample sizes are not applicable to these metrics.
+
 ## Lag and Backfill
 
 HHS issues updates to this timeseries once a week, and occasionally more

docs/api/covidcast-signals/hospital-admissions.md

@@ -68,6 +68,8 @@ hospitalizations, but not all of them, and so this source only represents those
 hospitalizations known to them. Their coverage may vary across the United
 States.
 
+Standard errors and sample sizes are not available for this data source.
+
 ## Qualifying Admissions
 
 We receive two daily data streams of new hospital admissions recorded by the health system partners at each location. One stream is based on electronic medical records, and the other comes from claims records.