From 05ec060c1310b0430e74c62e9ee108a02d6a0ee5 Mon Sep 17 00:00:00 2001 From: ilaigigi Date: Fri, 13 Dec 2024 14:47:52 +0200 Subject: [PATCH 1/7] changing docker installation instructions --- .../03_contributors_quick_start.rst | 20 ++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/contributing-docs/03_contributors_quick_start.rst b/contributing-docs/03_contributors_quick_start.rst index 121323c5c9bdb..98ba5d832e39b 100644 --- a/contributing-docs/03_contributors_quick_start.rst +++ b/contributing-docs/03_contributors_quick_start.rst @@ -82,16 +82,18 @@ Docker Community Edition .. code-block:: bash sudo apt-get update - sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin + sudo apt-get install docker-ce docker-ce-cli containerd.io -3. Creating group for docker and adding current user to it +3. Manage docker as non-root user .. code-block:: bash sudo groupadd docker sudo usermod -aG docker $USER -Note : After adding user to docker group Logout and Login again for group membership re-evaluation. +Note : This is done so a non-root user can access the `docker` command. +After adding user to docker group Logout and Login again for group membership re-evaluation. +On some Linux distributions, the system automatically creates this group. 4. Test Docker installation @@ -99,6 +101,8 @@ Note : After adding user to docker group Logout and Login again for group member docker run hello-world +Note : Read more about `Linux post-installation steps for Docker Engine `_. + Colima ------ If you use Colima as your container runtimes engine, please follow the next steps: @@ -121,6 +125,13 @@ Docker Compose -------------- 1. Installing latest version of Docker Compose +Install plugin using the repository: + +.. code-block:: bash + sudo apt-get update + sudo apt-get install docker-compose-plugin + +Install plugin manually: .. code-block:: bash @@ -134,6 +145,9 @@ Docker Compose sudo chmod +x /usr/local/bin/docker-compose +Note: This option requires you to manage updates manually. +It is recommended that you set up Docker's repository for easier maintenance. + 2. Verifying installation .. code-block:: bash From 5db751ac1e11abef7a4a78fc8816e93de945cfc0 Mon Sep 17 00:00:00 2001 From: ilaigigi Date: Fri, 13 Dec 2024 14:50:14 +0200 Subject: [PATCH 2/7] rewording docker installation instructions --- contributing-docs/03_contributors_quick_start.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/contributing-docs/03_contributors_quick_start.rst b/contributing-docs/03_contributors_quick_start.rst index 98ba5d832e39b..4b13b224a90be 100644 --- a/contributing-docs/03_contributors_quick_start.rst +++ b/contributing-docs/03_contributors_quick_start.rst @@ -124,14 +124,14 @@ If you use Colima as your container runtimes engine, please follow the next step Docker Compose -------------- -1. Installing latest version of Docker Compose -Install plugin using the repository: +1. Installing latest version of the Docker Compose plugin +Install using the repository: .. code-block:: bash sudo apt-get update sudo apt-get install docker-compose-plugin -Install plugin manually: +Install manually: .. code-block:: bash From 65b8f16e62bfa0b713fff56b7b4422b02cce67a1 Mon Sep 17 00:00:00 2001 From: ilaigigi Date: Fri, 13 Dec 2024 15:28:54 +0200 Subject: [PATCH 3/7] updating breeze setup instructions --- .../03_contributors_quick_start.rst | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/contributing-docs/03_contributors_quick_start.rst b/contributing-docs/03_contributors_quick_start.rst index 4b13b224a90be..ae079383d6eb0 100644 --- a/contributing-docs/03_contributors_quick_start.rst +++ b/contributing-docs/03_contributors_quick_start.rst @@ -169,7 +169,7 @@ Setting up virtual-env .. code-block:: bash - sudo apt install openssl sqlite default-libmysqlclient-dev libmysqlclient-dev postgresql + sudo apt install openssl sqlite3 default-libmysqlclient-dev libmysqlclient-dev postgresql If you want to install all airflow providers, more system dependencies might be needed. For example on Debian/Ubuntu like system, this command will install all necessary dependencies that should be installed when you use @@ -312,7 +312,6 @@ Setting up Breeze .. code-block:: bash - root@b76fcb399bb6:/opt/airflow# root@b76fcb399bb6:/opt/airflow# exit 8. You can stop the environment (which means deleting the databases and database servers running in the @@ -329,7 +328,10 @@ Using Breeze 1. Starting breeze environment using ``breeze start-airflow`` starts Breeze environment with last configuration run( In this case python and backend will be picked up from last execution ``breeze --python 3.9 --backend postgres``) It also automatically starts webserver, backend and scheduler. It drops you in tmux with scheduler in bottom left - and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate + and webserver in bottom right. Use ``[Ctrl + B] and Arrow keys`` to navigate. + Keep in mind, you need ``pre-commit`` installed for this to work or you will be prompted with + ``FileNotFoundError: [Errno 2] No such file or directory: 'pre-commit'`` + when attempting to invoke ``breeze start-airflow``. .. code-block:: bash @@ -427,13 +429,22 @@ Using Breeze 4. Stopping breeze +If Airflow was started with ``breeze start-airflow``: .. code-block:: bash root@f3619b74c59a:/opt/airflow# stop_airflow + breeze down + +If Airflow was started with ``breeze --python 3.9 --backend postgres``: + +.. code-block:: bash + root@f3619b74c59a:/opt/airflow# exit breeze down +Note : ``stop_airflow`` is available only when Airflow is started with ``breeze start-airflow``. + 5. Knowing more about Breeze .. code-block:: bash From 7ac6bb8cffa9ac2d129fdba3babbee78b638921f Mon Sep 17 00:00:00 2001 From: ilaigigi Date: Fri, 13 Dec 2024 15:35:54 +0200 Subject: [PATCH 4/7] updating airflow local venv setup instructions --- contributing-docs/03_contributors_quick_start.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contributing-docs/03_contributors_quick_start.rst b/contributing-docs/03_contributors_quick_start.rst index ae079383d6eb0..7f86f081b7e8d 100644 --- a/contributing-docs/03_contributors_quick_start.rst +++ b/contributing-docs/03_contributors_quick_start.rst @@ -636,7 +636,7 @@ Installing airflow in the local venv .. code-block:: bash - sudo apt-get install sqlite libsqlite3-dev default-libmysqlclient-dev postgresql + sudo apt-get install sqlite3 libsqlite3-dev default-libmysqlclient-dev postgresql ./scripts/tools/initialize_virtualenv.py From 43625c5ad8ba6dc86bfcb9d87a1e1721e337c95c Mon Sep 17 00:00:00 2001 From: ilaigigi Date: Fri, 13 Dec 2024 15:41:50 +0200 Subject: [PATCH 5/7] fixing spacing --- contributing-docs/03_contributors_quick_start.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/contributing-docs/03_contributors_quick_start.rst b/contributing-docs/03_contributors_quick_start.rst index 7f86f081b7e8d..bf56783f09917 100644 --- a/contributing-docs/03_contributors_quick_start.rst +++ b/contributing-docs/03_contributors_quick_start.rst @@ -125,6 +125,7 @@ Docker Compose -------------- 1. Installing latest version of the Docker Compose plugin + Install using the repository: .. code-block:: bash @@ -429,6 +430,7 @@ Using Breeze 4. Stopping breeze + If Airflow was started with ``breeze start-airflow``: .. code-block:: bash From 7bb3a12e0103f59b096a1dc937f1465c41e111b5 Mon Sep 17 00:00:00 2001 From: ilaigigi Date: Fri, 13 Dec 2024 15:42:45 +0200 Subject: [PATCH 6/7] fixing spacings --- contributing-docs/03_contributors_quick_start.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/contributing-docs/03_contributors_quick_start.rst b/contributing-docs/03_contributors_quick_start.rst index bf56783f09917..54e09eb9c63ed 100644 --- a/contributing-docs/03_contributors_quick_start.rst +++ b/contributing-docs/03_contributors_quick_start.rst @@ -129,6 +129,7 @@ Docker Compose Install using the repository: .. code-block:: bash + sudo apt-get update sudo apt-get install docker-compose-plugin From e159326958c3ae3691e1fc82f3323dd8c3e726f2 Mon Sep 17 00:00:00 2001 From: ilaigigi Date: Fri, 20 Dec 2024 21:34:44 +0200 Subject: [PATCH 7/7] using double quotes for code in rsT --- .../03_contributors_quick_start.rst | 39 +++++-------------- 1 file changed, 9 insertions(+), 30 deletions(-) diff --git a/contributing-docs/03_contributors_quick_start.rst b/contributing-docs/03_contributors_quick_start.rst index 54e09eb9c63ed..0c6fef5c029f9 100644 --- a/contributing-docs/03_contributors_quick_start.rst +++ b/contributing-docs/03_contributors_quick_start.rst @@ -91,7 +91,7 @@ Docker Community Edition sudo groupadd docker sudo usermod -aG docker $USER -Note : This is done so a non-root user can access the `docker` command. +Note : This is done so a non-root user can access the ``docker`` command. After adding user to docker group Logout and Login again for group membership re-evaluation. On some Linux distributions, the system automatically creates this group. @@ -225,8 +225,8 @@ Set it to true for windows. git config core.autocrlf true -Typical development tasks -######################### +Setting up Breeze +----------------- For many of the development tasks you will need ``Breeze`` to be configured. ``Breeze`` is a development environment which uses docker and docker-compose and its main purpose is to provide a consistent @@ -235,9 +235,6 @@ syndrome - because not only others can reproduce easily what you do, but also th the same environment to run all tests - so you should be able to easily reproduce the same failures you see in CI in your local environment. -Setting up Breeze ------------------ - 1. Install ``uv`` or ``pipx``. We recommend to install ``uv`` as general purpose python development environment - you can install it via https://docs.astral.sh/uv/getting-started/installation/ or you can install ``pipx`` (>=1.2.1) - follow the instructions in `Install pipx `_ @@ -491,27 +488,13 @@ To avoid burden on CI infrastructure and to save time, Pre-commit hooks can be r started to use Python 3.9+ features in Airflow and accompanying scripts. -Installing pre-commit is best done with ``uv`` (recommended) or ``pipx``: - -This will install ``pre-commit`` with ``uv``, and it will change it to use ``uv`` to install its own -virtualenvs. - -.. code-block:: bash - - uv tool install pre-commit --with pre-commit-uv - -or - -.. code-block:: bash - - pipx install pre-commit +Installing pre-commit is best done with ``uv`` (recommended) or ``pipx``. You can add ``uv`` support for ``pre-commit`` even you install it with ``pipx`` using the commands (then pre-commit will use ``uv`` to create virtualenvs for the hooks): .. code-block:: bash - pipx install pre-commit pipx inject pre-commit pre-commit-uv # optionally if you want to use uv to install virtualenvs 1. Installing required packages @@ -528,7 +511,7 @@ on macOS, install via brew install libxml2 -2. Installing pre-commit (if you have not done it yet): +2. Installing pre-commit: .. code-block:: bash @@ -539,7 +522,7 @@ or .. code-block:: bash pipx install pre-commit - pipx install inject pre-commit pre-commit-uv + pipx install inject pre-commit pre-commit-uv # optional, configures pre-commit to use uv to install virtualenvs 3. Go to your project directory @@ -690,10 +673,6 @@ All Tests are inside ./tests directory. - Running specific type of test - - Types of tests - - - Running specific type of test - .. code-block:: bash breeze --backend postgres --postgres-version 15 --python 3.9 --db-reset testing tests --test-type Core @@ -701,8 +680,6 @@ All Tests are inside ./tests directory. - Running Integration test for specific test type - - Running an Integration Test - .. code-block:: bash breeze --backend postgres --postgres-version 15 --python 3.9 --db-reset testing tests --test-type All --integration mongo @@ -713,7 +690,9 @@ All Tests are inside ./tests directory. 09_testing.rst - - |Local and Remote Debugging in IDE| +- Similarly to regular development, you can also debug while testing using your IDE, for more information, you may refer to + + |Local and Remote Debugging in IDE| .. |Local and Remote Debugging in IDE| raw:: html