Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix logging for astro CLI local parse and pytest commands #1792

Merged
merged 1 commit into from
Jan 30, 2025

Conversation

pritt20
Copy link
Contributor

@pritt20 pritt20 commented Jan 30, 2025

Description

We recently shipped a change to suppress the image build logs and display user friendly messaging to users but we also suppressed the logs for astro dev parse and astro dev pytest commands. As a result these logs are now only available with --versbosity=debug flag, which is not ideal during troubleshooting failures.

As part of this PR changes, logs for astro dev parse and astro dev pytest commands will be tagged to warn log level, which is default and therefore user will not have to use --versbosity=debug flag to get these logs in case of error and failures.

🎟 Issue(s)

Related #1789

🧪 Functional Testing

Tested the changes locally. Please find the details below:

astro dev parse

Success:

astro dev parse 

Checking your DAGs for errors,
this might take a minute if you haven't run this command before…
✔ Project image has been updated
68dd7555f3502f48f03695a12764da5182826803785908c18edf6e4eec89c6a6
Astro Runtime Version: 12.2.0
============================= test session starts ==============================
platform linux -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0
rootdir: /usr/local/airflow
plugins: anyio-4.4.0, time-machine-2.15.0
collected 1 item

.astro/test_dag_integrity_default.py .                                   [100%]

============================== 1 passed in 2.37s ===============================

✔ no errors detected in your DAGs 

Failure:

astro dev parse                  

Checking your DAGs for errors,
this might take a minute if you haven't run this command before…
✔ Project image has been updated
6e8a859f125abc9be134058ddff149c67938615667d8b176a7efba3f0eee1a6b
Astro Runtime Version: 12.2.0
============================= test session starts ==============================
platform linux -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0
rootdir: /usr/local/airflow
plugins: anyio-4.4.0, time-machine-2.15.0
collected 1 item

.astro/test_dag_integrity_default.py F                                   [100%]

=================================== FAILURES ===================================
_______________ test_file_imports[dags/example_etl_galaxies.py] ________________

rel_path = 'dags/example_etl_galaxies.py'
rv = 'Traceback (most recent call last):\n  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed\n ...s/example_etl_galaxies.py", line 19, in <module>\n    import pand as pd\nModuleNotFoundError: No module named \'pand\''

    @pytest.mark.parametrize(
        "rel_path, rv", get_import_errors(), ids=[x[0] for x in get_import_errors()]
    )
    def test_file_imports(rel_path, rv):
        """Test for import errors on a file"""
        if rv != "No import errors":
            # If rv is not "No import errors," consider it a failed test
>           raise Exception(f"{rel_path} failed to import with message \n {rv}")
E           Exception: dags/example_etl_galaxies.py failed to import with message 
E            Traceback (most recent call last):
E             File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
E             File "/usr/local/airflow/dags/example_etl_galaxies.py", line 19, in <module>
E               import pand as pd
E           ModuleNotFoundError: No module named 'pand'

.astro/test_dag_integrity_default.py:133: Exception
=========================== short test summary info ============================
FAILED .astro/test_dag_integrity_default.py::test_file_imports[dags/example_etl_galaxies.py]
============================== 1 failed in 1.81s ===============================
Error: See above for errors detected in your DAGs

astro dev pytest

Success:

astro dev pytest

Running Pytest
This may take a minute if you have not run this command before…
✔ Project image has been updated
ccd16e601065d84137942d9ab3148f0a8275f2a88babfc3395b914f10d858e7a
Astro Runtime Version: 12.2.0
============================= test session starts ==============================
platform linux -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0
rootdir: /usr/local/airflow
plugins: anyio-4.4.0, time-machine-2.15.0
collected 3 items

tests/dags/test_dag_example.py ...                                       [100%]

============================== 3 passed in 1.82s ===============================

✔ All Pytests passed!

Failure:

astro dev pytest

Running Pytest
This may take a minute if you have not run this command before…
✔ Project image has been updated
3cc99a83c93ec15066d7b8eed451e558e129316e47198218b09504784fe00886
Astro Runtime Version: 12.2.0
============================= test session starts ==============================
platform linux -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0
rootdir: /usr/local/airflow
plugins: anyio-4.4.0, time-machine-2.15.0
collected 4 items

tests/dags/test_dag_example.py .Fss                                      [100%]

=================================== FAILURES ===================================
_______________ test_file_imports[dags/example_etl_galaxies.py] ________________

rel_path = 'dags/example_etl_galaxies.py'
rv = 'Traceback (most recent call last):\n  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed\n ...s/example_etl_galaxies.py", line 19, in <module>\n    import pand as pd\nModuleNotFoundError: No module named \'pand\''

    @pytest.mark.parametrize(
        "rel_path,rv", get_import_errors(), ids=[x[0] for x in get_import_errors()]
    )
    def test_file_imports(rel_path, rv):
        """Test for import errors on a file"""
        if rel_path and rv:
>           raise Exception(f"{rel_path} failed to import with message \n {rv}")
E           Exception: dags/example_etl_galaxies.py failed to import with message 
E            Traceback (most recent call last):
E             File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
E             File "/usr/local/airflow/dags/example_etl_galaxies.py", line 19, in <module>
E               import pand as pd
E           ModuleNotFoundError: No module named 'pand'

tests/dags/test_dag_example.py:56: Exception
=========================== short test summary info ============================
FAILED tests/dags/test_dag_example.py::test_file_imports[dags/example_etl_galaxies.py]
==================== 1 failed, 1 passed, 2 skipped in 1.47s ====================
Error: pytests failed

📋 Checklist

  • Rebased from the main (or release if patching) branch (before testing)
  • Ran make test before taking out of draft
  • Ran make lint before taking out of draft
  • Added/updated applicable tests
  • Tested against Astro-API (if necessary).
  • Tested against Houston-API and Astronomer (if necessary).
  • Communicated to/tagged owners of respective clients potentially impacted by these changes.
  • Updated any related documentation

Copy link
Contributor

@neel-astro neel-astro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@schnie schnie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @pritt20, this makes sense!

@pritt20 pritt20 merged commit a8abaea into main Jan 30, 2025
5 checks passed
@pritt20 pritt20 deleted the fix_error_logging branch January 30, 2025 15:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants