Skip to content

Latest commit

 

History

History
126 lines (79 loc) · 4.43 KB

README.md

File metadata and controls

126 lines (79 loc) · 4.43 KB
Raw

logo_and_name

Serve Data with Low-Latency


1. Features 🎍

data-api discovers your datastore data & exposes them as a REST read-only API.

💡 By datastore we mean firestore in datastore mode.

Features include:

  • generate open-api spec definition
  • generate Swagger-UI documentation page
  • expose datastore data via key-value or property-filtering.
  • cache responses with configurable duration
  • manage permissions with api-keys or openID tokens.

💡 data-api is great to create a low-latency API for your BigQuery data (that you can export to datastore with one sql query. More below).


2. Deploy 🚀

Run

gcloud run deploy data-api \
  --source . \
  --set-env-vars "PROJECT=$PROJECT,DATABASE=$DATABASE"

with:

  • $PROJECT the Google Cloud Project where you datastore database resides
  • $DATABASE the name of your datastore database (if not given, it will use the default database).

You can then get the url of your data-api Cloud Run Service by running:

gcloud run services describe --format "value(status.url)"

3. Exposed Routes 🚚

💡 Datastore Reminder:

  • Data in datastore is organized in a hierarchy: database/namespace/kind (like the database/schema/table hierarchy in relational databases).
  • Inside a kind (think table), the entities (think rows) have a key (a string or integer) and a value (a dict).

data-api Cloud Run service:

  • exposes the data of the database defined as environment variable at deploy time (see above).
  • considers each namespace as a different api (which has its own open-api spec definition and Swagger UI).
  • exposes the following routes (with GET method):
url Description
/ Redirects to /api/
/api/ Returns the list of namespaces in database
/api/<namespace>/ Returns details on namespace api including its kinds and urls
/api/<namespace>/openapi.json Returns the openapi spec definition of the namespace api
/api/<namespace>/swagger-ui.html Returns the Swagger UI (documentation portal) of the namespace api
/api/<namespace>/<kind>/ Returns a list of entity values of kind
/api/<namespace>/<kind>/<key> Returns the entity value of key
/api/<namespace>/<kind>/?foo=bar Returns a list of entity values of kind for which foo property is equal to bar

4. [BONUS] Export BigQuery table to datastore ✍️

data-api is great to create a low-latency API for your BigQuery data.

You can export a BigQuery table into datastore with one sql command to make it available to data-api:

call bigfunctions.eu.export_table_to_datastore(
  'your-project.dataset.table',
  'user_id',
  'your-project/your-database/default/users'
);

💡 If you don't want to use the public bigfunction, you can deploy the function in your own BigQuery project. Check the function documentation.


5. Contribute 👋

Any contribution is more than welcome 🤗!

  • Add a ⭐ on the repo to show your support
  • Join our Slack and talk with us
  • Raise an issue to raise a bug or suggest improvements
  • Open a PR!

6. TODO 📝

  • manage permissions with a metadata key-value
  • make queries work when filtering with integer (using the datastore schema)
  • extend this concept to other backends