-
Notifications
You must be signed in to change notification settings - Fork 3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #5497 from OpenBB-finance/feature/improved-docs
Improve docs
- Loading branch information
Showing
34 changed files
with
1,513 additions
and
537 deletions.
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
4 changes: 2 additions & 2 deletions
4
openbb_platform/extensions/tests/utils/integration_tests_api_generator.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
{ | ||
"label": "Contributing", | ||
"position": 2 | ||
} |
4 changes: 4 additions & 0 deletions
4
website/content/platform/contributing/extension-development/_category_.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
{ | ||
"label": "Extension Development", | ||
"position": 2 | ||
} |
78 changes: 78 additions & 0 deletions
78
website/content/platform/contributing/extension-development/custom_data.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,78 @@ | ||
--- | ||
title: Add a custom data source | ||
sidebar_position: 2 | ||
description: Add a custom data source to the OpenBB Platform. | ||
keywords: [openbb platform, extension, custom data source, contributing, documentation] | ||
--- | ||
|
||
import HeadTitle from '@site/src/components/General/HeadTitle.tsx'; | ||
|
||
<HeadTitle title="Add a custom data source - Platform | OpenBB Docs" /> | ||
|
||
You will get your data either from a CSV file, local database or from an API endpoint. | ||
|
||
If you don't want or don't need to partake in the data standardization framework, you have the option to add all the logic straight inside the router file. This is usually the case when you are returning custom data from your local CSV file, or similar. Keep in mind that we also serve the REST API and that you shouldn't send non-serializable objects as a response (e.g. a pandas dataframe). | ||
|
||
Saying that, we highly recommend following the standardization framework, as it will make your life easier in the long run and unlock a set of features that are only available to standardized data. | ||
|
||
When standardizing, all data is defined using two different pydantic models: | ||
|
||
1. Define the [query parameters](https://github.com/OpenBB-finance/OpenBBTerminal/tree/feature/openbb-sdk-v4/openbb_platform/platform/provider/openbb_provider/abstract/query_params.py) model. | ||
2. Define the resulting [data schema](https://github.com/OpenBB-finance/OpenBBTerminal/tree/feature/openbb-sdk-v4/openbb_platform/platform/provider/openbb_provider/abstract/data.py) model. | ||
|
||
> The models can be entirely custom, or inherit from the OpenBB standardized models. | ||
> They enforce a safe and consistent data structure, validation and type checking. | ||
We call this the ***Know-Your-Data*** principle. | ||
|
||
After you've defined both models, you'll need to define a `Fetcher` class which contains three methods: | ||
|
||
1. `transform_query` - transforms the query parameters to the format of the API endpoint. | ||
2. `extract_data` - makes the request to the API endpoint and returns the raw data. | ||
3. `transform_data` - transforms the raw data into the defined data model. | ||
|
||
> Note that the `Fetcher` should inherit from the [`Fetcher`](https://github.com/OpenBB-finance/OpenBBTerminal/tree/feature/openbb-sdk-v4/openbb_platform/platform/provider/openbb_provider/abstract/fetcher.py) class, which is a generic class that receives the query parameters and the data model as type parameters. | ||
After finalizing your models, you need to make them visible to the Openbb Platform. This is done by adding the `Fetcher` to the `__init__.py` file of the `<your_package_name>/<your_module_name>` folder as part of the [`Provider`](https://github.com/OpenBB-finance/OpenBBTerminal/tree/feature/openbb-sdk-v4/openbb_platform/platform/provider/openbb_provider/abstract/provider.py). | ||
|
||
Any command, that uses the `Fetcher` class you've just defined, will be calling the `transform_query`, `extract_data` and `transform_data` methods under the hood in order to get the data and output it do the end user. | ||
|
||
If you're not sure what's a command and why is it even using the `Fetcher` class, follow along! | ||
|
||
## OpenBB Platform commands | ||
|
||
The OpenBB Platform will enable you to query and output your data in a very simple way. | ||
|
||
> Any Platform endpoint will be available both from a Python interface and the API. | ||
The command definition on the Platform follows [FastAPI](https://fastapi.tiangolo.com/) conventions, meaning that you'll be creating **endpoints**. | ||
|
||
The Cookiecutter template generates for you a `router.py` file with a set of examples that you can follow, namely: | ||
|
||
- Perform a simple `GET` and `POST` request - without worrying on any custom data definition. | ||
- Using a custom data definition so you get your data the exact way you want it. | ||
|
||
You can expect the following endpoint structure when using a `Fetcher` to serve the data: | ||
|
||
```python | ||
@router.command(model="Example") | ||
def model_example( | ||
cc: CommandContext, | ||
provider_choices: ProviderChoices, | ||
standard_params: StandardParams, | ||
extra_params: ExtraParams, | ||
) -> OBBject[BaseModel]: | ||
"""Example Data.""" | ||
return OBBject(results=Query(**locals()).execute()) | ||
``` | ||
|
||
Let's break it down: | ||
|
||
- `@router.command(...)` - this tells the OpenBB Platform that this is a command. | ||
- `model="Example"` - this is the name of the `Fetcher` dictionary key that you've defined in the `__init__.py` file of the `<your_package_name>/<your_module_name>` folder. | ||
- `cc: CommandContext` - this contains a set of user and system settings that is useful during the execution of the command - eg. api keys. | ||
- `provider_choices: ProviderChoices` - all the providers that implement the `Example` `Fetcher`. | ||
- `standard_params: StandardParams` - standardized parameters that are common to all providers that implement the `Example` `Fetcher`. | ||
- `extra_params: ExtraParams` - it contains the provider specific arguments that are not standardized. | ||
|
||
You only need to change the `model` parameter to the name of the `Fetcher` dictionary key and everything else will be handled by the OpenBB Platform. |
25 changes: 25 additions & 0 deletions
25
website/content/platform/contributing/extension-development/index.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
--- | ||
title: Extension Development | ||
sidebar_position: 1 | ||
description: Learn how to develop Extensions inside the OpenBB Platform. | ||
keywords: [openbb platform, introduction, extensions, contributing, documentation] | ||
--- | ||
|
||
import HeadTitle from '@site/src/components/General/HeadTitle.tsx'; | ||
|
||
<HeadTitle title="Extension Development - Platform | OpenBB Docs" /> | ||
|
||
We have a Cookiecutter template that will help you get started. It serves as a jumpstart for your extensions development, so you can focus on the data and not on the boilerplate. | ||
|
||
Please refer to the [Cookiecutter template](https://github.com/OpenBB-finance/openbb-cookiecutter) and follow the instructions there. | ||
|
||
This section will walk you through the steps of adding a new extension to the OpenBB Platform. | ||
|
||
The high level steps are: | ||
|
||
- Generate the extension structure | ||
- Install your dependencies | ||
- Install your new package | ||
- Use your extension (either from Python or the API interface) | ||
- QA your extension | ||
- Share your extension with the community |
91 changes: 91 additions & 0 deletions
91
website/content/platform/contributing/extension-development/qa.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,91 @@ | ||
--- | ||
title: Extension QA | ||
sidebar_position: 3 | ||
description: Learn how to QA your extension. | ||
keywords: [openbb platform, introduction, qa, contributing, extension, documentation] | ||
--- | ||
|
||
import HeadTitle from '@site/src/components/General/HeadTitle.tsx'; | ||
|
||
<HeadTitle title="How to QA your extension - Platform | OpenBB Docs" /> | ||
|
||
We are strong believers in the QA process and we want to make sure that all the extensions that are added to the OpenBB Platform are of high quality. To ensure this, we have a set of QA tools that you can use to test your extension. | ||
|
||
Primarily, we have tools that semi-automate the creation of unit and integration tests. | ||
|
||
> The QA tools are still in development and we are constantly improving them. | ||
## Unit tests | ||
|
||
Each `Fetcher` comes equipped with a `test` method that will ensure that it is implemented correctly and that it is returning the expected data. It also ensures that all types are correct and that the data is valid. | ||
|
||
To create unit tests for your Fetchers, you can run the following command: | ||
|
||
```bash | ||
python openbb_platform/providers/tests/utils/unit_test_generator.py | ||
``` | ||
|
||
> Note that you should be running this file from the root of the repository. | ||
The automatic unit test generation will add unit tests for all the fetchers available in a given provider. | ||
|
||
> Note that sometimes manual intervention is needed. For example, adjusting out-of-top level imports or adding specific arguments for a given fetcher. | ||
## Integration tests | ||
|
||
The integration tests are a bit more complex than the unit tests, as we want to test both the Python interface and the API interface. For this, we have two scripts that will help you generate the integration tests. | ||
|
||
To generate the integration tests for the Python interface, you can run the following command: | ||
|
||
```bash | ||
python openbb_platform/extensions/tests/utils/integration_tests_generator.py | ||
``` | ||
|
||
To generate the integration tests for the API interface, you can run the following command: | ||
|
||
```bash | ||
python openbb_platform/extensions/tests/utils/integration_tests_api_generator.py | ||
``` | ||
|
||
When testing the API interface, you'll need to run the OpenBB Platform locally before running the tests. To do so, you can run the following command: | ||
|
||
```bash | ||
uvicorn openbb_platform.platform.core.openbb_core.api.rest_api:app --host 0.0.0.0 --port 8000 --reload | ||
``` | ||
|
||
These automated tests are a great way to reduce the amount of code you need to write, but they are not a replacement for manual testing and might require tweaking. That's why we have unit tests that test the generated integration tests to ensure they cover all providers and parameters. | ||
|
||
To run the tests we can do: | ||
|
||
- Unit tests only: | ||
|
||
```bash | ||
pytest openbb_platform -m "not integration" | ||
``` | ||
|
||
- Integration tests only: | ||
|
||
```bash | ||
pytest openbb_platform -m integration | ||
``` | ||
|
||
- Both integration and unit tests: | ||
|
||
```bash | ||
pytest openbb_platform | ||
``` | ||
|
||
## Import time | ||
|
||
We aim to have a short import time for the package. To measure that we use `tuna`. | ||
|
||
- <https://pypi.org/project/tuna/> | ||
|
||
To visualize the import time breakdown by module and find potential bottlenecks, run the | ||
following commands from `openbb_platform` directory: | ||
|
||
```bash | ||
pip install tuna | ||
python -X importtime openbb/__init__.py 2> import.log | ||
tuna import.log | ||
``` |
Oops, something went wrong.