Store v2 reports in GCS instead of Redis #960

Swatinem · 2024-11-05T13:43:57Z

Worker has some dedicated code to move an uploaded report from Redis over to GCS. Instead of having that code, we can just store the report into GCS in the first place.

Additionally, this PR also cleans up the services.archive, replacing it fully with the shared.api_archive.archive.

It also forwards all the task arguments directly to the Upload task, in addition to storing it in Redis, to be forward compatible to some more planned upload processing improvements.

Depends on codecov/shared#415

codecov · 2024-11-05T13:55:49Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.25%. Comparing base (ffcc446) to head (782fb8a).

✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #960      +/-   ##
==========================================
- Coverage   96.25%   96.25%   -0.01%     
==========================================
  Files         826      826              
  Lines       19048    19041       -7     
==========================================
- Hits        18334    18327       -7     
  Misses        714      714

Flag	Coverage Δ
unit	`92.53% <100.00%> (+0.03%)`	⬆️
unit-latest-uploader	`92.53% <100.00%> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

codecov-notifications · 2024-11-05T13:55:52Z

Codecov Report

All modified and coverable lines are covered by tests ✅

✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

codecov-qa · 2024-11-05T13:55:56Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
2652	2	2650	6

View the top 2 failed tests by shortest run time

upload/tests/test_upload.py::UploadHandlerRouteTest::test_successful_upload_v2_slash

Stack Traces | 0.05s run time

self = &lt;MagicMock name='write_file' id='139671276570912'&gt;
args = ('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', &lt;ANY&gt;, &lt;ANY&gt;)
kwargs = {}
expected = call('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', &lt;ANY&gt;, &lt;ANY&gt;)
actual = call('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)
_error_message = &lt;function NonCallableMock.assert_called_with.&lt;locals&gt;._error_message at 0x7f077c6e7b00&gt;
cause = None

    def assert_called_with(self, /, *args, **kwargs):
        """assert that the last call was made with the specified arguments.
    
        Raises an AssertionError if the args and keyword args passed in are
        different to the last call to the mock."""
        if self.call_args is None:
            expected = self._format_mock_call_signature(args, kwargs)
            actual = 'not called.'
            error_message = ('expected call not found.\nExpected: %s\n  Actual: %s'
                    % (expected, actual))
            raise AssertionError(error_message)
    
        def _error_message():
            msg = self._format_mock_failure_message(args, kwargs)
            return msg
        expected = self._call_matcher(_Call((args, kwargs), two=True))
        actual = self._call_matcher(self.call_args)
        if actual != expected:
            cause = expected if isinstance(expected, Exception) else None
&gt;           raise AssertionError(_error_message()) from cause
E           AssertionError: expected call not found.
E           Expected: write_file('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', &lt;ANY&gt;, &lt;ANY&gt;)
E             Actual: write_file('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)

.../local/lib/python3.12/unittest/mock.py:949: AssertionError

During handling of the above exception, another exception occurred:

self = &lt;upload.tests.test_upload.UploadHandlerRouteTest testMethod=test_successful_upload_v2_slash&gt;
mock_repo_provider_service = &lt;MagicMock name='get_adapter' id='139670663700240'&gt;
mock_dispatch_upload = &lt;MagicMock name='dispatch_upload_task' id='139670137903856'&gt;
mock_uuid4 = &lt;MagicMock name='uuid4' id='139671605272432'&gt;
mock_get_redis = &lt;MagicMock name='get_redis_connection' id='139670130925712'&gt;
mock_write_file = &lt;MagicMock name='write_file' id='139671276570912'&gt;

    @patch("shared.api_archive.archive.ArchiveService.write_file")
    @patch("upload.views.legacy.get_redis_connection")
    @patch("upload.views.legacy.uuid4")
    @patch("upload.views.legacy.dispatch_upload_task")
    @patch("services.repo_providers.RepoProviderService.get_adapter")
    @override_settings(CODECOV_DASHBOARD_URL="https://app.codecov.io")
    def test_successful_upload_v2_slash(
        self,
        mock_repo_provider_service,
        mock_dispatch_upload,
        mock_uuid4,
        mock_get_redis,
        mock_write_file,
    ):
        class MockRepoProviderAdapter:
            async def get_commit(self, commit, token):
                return {"message": "This is not a merge commit"}
    
        mock_get_redis.return_value = MockRedis()
        mock_repo_provider_service.return_value = MockRepoProviderAdapter()
        mock_uuid4.return_value = (
            "dec1f00b-1883-40d0-afd6-6dcb876510be"  # this will be the reportid
        )
    
        query_params = {
            "commit": "b521e55aef79b101f48e2544837ca99a7fa3bf6b",
            "token": "a03e5d02-9495-4413-b0d8-05651bb2e842",
            "pr": "456",
            "branch": "",
            "flags": "",
            "build_url": "",
            "package": "",
        }
    
        response = self._post_slash(
            kwargs={"version": "v2"}, query=query_params, data="coverage report"
        )
    
        assert response.status_code == 200
    
        headers = response.headers
    
        assert headers["access-control-allow-origin"] == "*"
        assert (
            headers["access-control-allow-headers"]
            == "Origin, Content-Type, Accept, X-User-Agent"
        )
        assert headers["content-type"] != "text/plain"
    
        archive_service = ArchiveService(self.repo)
        datetime = timezone.now().strftime("%Y-%m-%d")
        repo_hash = archive_service.get_archive_hash(self.repo)
        expected_url = f"v4/raw/{datetime}/{repo_hash}/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt"
    
&gt;       mock_write_file.assert_called_with(expected_url, ANY, ANY)
E       AssertionError: expected call not found.
E       Expected: write_file('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', &lt;ANY&gt;, &lt;ANY&gt;)
E         Actual: write_file('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)
E       
E       pytest introspection follows:
E       
E       Args:
E       assert ('v4/raw/2024...erage report') == ('v4/raw/2024... &lt;ANY&gt;, &lt;ANY&gt;)
E         
E         Right contains one more item: &lt;ANY&gt;
E         Use -v to get more diff
E       Kwargs:
E       assert {'is_already_gzipped': False} == {}
E         
E         Left contains 1 more item:
E         {'is_already_gzipped': False}
E         Use -v to get more diff

upload/tests/test_upload.py:1100: AssertionError

upload/tests/test_upload.py::UploadHandlerRouteTest::test_successful_upload_v2

Stack Traces | 0.059s run time

self = &lt;MagicMock name='write_file' id='139670940359360'&gt;
args = ('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', &lt;ANY&gt;, &lt;ANY&gt;)
kwargs = {}
expected = call('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', &lt;ANY&gt;, &lt;ANY&gt;)
actual = call('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)
_error_message = &lt;function NonCallableMock.assert_called_with.&lt;locals&gt;._error_message at 0x7f07acefd760&gt;
cause = None

    def assert_called_with(self, /, *args, **kwargs):
        """assert that the last call was made with the specified arguments.
    
        Raises an AssertionError if the args and keyword args passed in are
        different to the last call to the mock."""
        if self.call_args is None:
            expected = self._format_mock_call_signature(args, kwargs)
            actual = 'not called.'
            error_message = ('expected call not found.\nExpected: %s\n  Actual: %s'
                    % (expected, actual))
            raise AssertionError(error_message)
    
        def _error_message():
            msg = self._format_mock_failure_message(args, kwargs)
            return msg
        expected = self._call_matcher(_Call((args, kwargs), two=True))
        actual = self._call_matcher(self.call_args)
        if actual != expected:
            cause = expected if isinstance(expected, Exception) else None
&gt;           raise AssertionError(_error_message()) from cause
E           AssertionError: expected call not found.
E           Expected: write_file('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', &lt;ANY&gt;, &lt;ANY&gt;)
E             Actual: write_file('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)

.../local/lib/python3.12/unittest/mock.py:949: AssertionError

During handling of the above exception, another exception occurred:

self = &lt;upload.tests.test_upload.UploadHandlerRouteTest testMethod=test_successful_upload_v2&gt;
mock_repo_provider_service = &lt;MagicMock name='get_adapter' id='139670663059200'&gt;
mock_dispatch_upload = &lt;MagicMock name='dispatch_upload_task' id='139670938203056'&gt;
mock_uuid4 = &lt;MagicMock name='uuid4' id='139671601606784'&gt;
mock_get_redis = &lt;MagicMock name='get_redis_connection' id='139670940003040'&gt;
mock_write_file = &lt;MagicMock name='write_file' id='139670940359360'&gt;

    @patch("shared.api_archive.archive.ArchiveService.write_file")
    @patch("upload.views.legacy.get_redis_connection")
    @patch("upload.views.legacy.uuid4")
    @patch("upload.views.legacy.dispatch_upload_task")
    @patch("services.repo_providers.RepoProviderService.get_adapter")
    @override_settings(CODECOV_DASHBOARD_URL="https://app.codecov.io")
    def test_successful_upload_v2(
        self,
        mock_repo_provider_service,
        mock_dispatch_upload,
        mock_uuid4,
        mock_get_redis,
        mock_write_file,
    ):
        class MockRepoProviderAdapter:
            async def get_commit(self, commit, token):
                return {"message": "This is not a merge commit"}
    
        mock_get_redis.return_value = MockRedis()
        mock_repo_provider_service.return_value = MockRepoProviderAdapter()
        mock_uuid4.return_value = (
            "dec1f00b-1883-40d0-afd6-6dcb876510be"  # this will be the reportid
        )
    
        query_params = {
            "commit": "b521e55aef79b101f48e2544837ca99a7fa3bf6b",
            "token": "a03e5d02-9495-4413-b0d8-05651bb2e842",
            "pr": "456",
            "branch": "",
            "flags": "",
            "build_url": "",
            "package": "",
        }
    
        response = self._post(
            kwargs={"version": "v2"}, query=query_params, data="coverage report"
        )
    
        assert response.status_code == 200
    
        headers = response.headers
    
        assert headers["access-control-allow-origin"] == "*"
        assert (
            headers["access-control-allow-headers"]
            == "Origin, Content-Type, Accept, X-User-Agent"
        )
        assert headers["content-type"] != "text/plain"
    
        archive_service = ArchiveService(self.repo)
        datetime = timezone.now().strftime("%Y-%m-%d")
        repo_hash = archive_service.get_archive_hash(self.repo)
        expected_url = f"v4/raw/{datetime}/{repo_hash}/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt"
    
&gt;       mock_write_file.assert_called_with(expected_url, ANY, ANY)
E       AssertionError: expected call not found.
E       Expected: write_file('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', &lt;ANY&gt;, &lt;ANY&gt;)
E         Actual: write_file('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)
E       
E       pytest introspection follows:
E       
E       Args:
E       assert ('v4/raw/2024...erage report') == ('v4/raw/2024... &lt;ANY&gt;, &lt;ANY&gt;)
E         
E         Right contains one more item: &lt;ANY&gt;
E         Use -v to get more diff
E       Kwargs:
E       assert {'is_already_gzipped': False} == {}
E         
E         Left contains 1 more item:
E         {'is_already_gzipped': False}
E         Use -v to get more diff

upload/tests/test_upload.py:1019: AssertionError

To view individual test run time comparison to the main branch, go to the Test Analytics Dashboard

github-actions · 2024-11-06T10:49:37Z

This PR includes changes to shared. Please review them here: codecov/shared@ebafd80...ce68890

codecov-public-qa · 2024-11-06T11:03:37Z

Test Failures Detected: Due to failing tests, we cannot provide coverage reports at this time.

❌ Failed Test Results:

Completed 2658 tests with 2 failed, 2650 passed and 6 skipped.

View the full list of failed tests

pytest

Class name: upload.tests.test_upload.UploadHandlerRouteTest
Test name: test_successful_upload_v2

self = <MagicMock name='write_file' id='139670940359360'>
args = ('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', <ANY>, <ANY>)
kwargs = {}
expected = call('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', <ANY>, <ANY>)
actual = call('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)
_error_message = <function NonCallableMock.assert_called_with.<locals>._error_message at 0x7f07acefd760>
cause = None

    def assert_called_with(self, /, *args, **kwargs):
        """assert that the last call was made with the specified arguments.
    
        Raises an AssertionError if the args and keyword args passed in are
        different to the last call to the mock."""
        if self.call_args is None:
            expected = self._format_mock_call_signature(args, kwargs)
            actual = 'not called.'
            error_message = ('expected call not found.\nExpected: %s\n  Actual: %s'
                    % (expected, actual))
            raise AssertionError(error_message)
    
        def _error_message():
            msg = self._format_mock_failure_message(args, kwargs)
            return msg
        expected = self._call_matcher(_Call((args, kwargs), two=True))
        actual = self._call_matcher(self.call_args)
        if actual != expected:
            cause = expected if isinstance(expected, Exception) else None
>           raise AssertionError(_error_message()) from cause
E           AssertionError: expected call not found.
E           Expected: write_file('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', <ANY>, <ANY>)
E             Actual: write_file('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)

.../local/lib/python3.12/unittest/mock.py:949: AssertionError

During handling of the above exception, another exception occurred:

self = <upload.tests.test_upload.UploadHandlerRouteTest testMethod=test_successful_upload_v2>
mock_repo_provider_service = <MagicMock name='get_adapter' id='139670663059200'>
mock_dispatch_upload = <MagicMock name='dispatch_upload_task' id='139670938203056'>
mock_uuid4 = <MagicMock name='uuid4' id='139671601606784'>
mock_get_redis = <MagicMock name='get_redis_connection' id='139670940003040'>
mock_write_file = <MagicMock name='write_file' id='139670940359360'>

    @patch("shared.api_archive.archive.ArchiveService.write_file")
    @patch("upload.views.legacy.get_redis_connection")
    @patch("upload.views.legacy.uuid4")
    @patch("upload.views.legacy.dispatch_upload_task")
    @patch("services.repo_providers.RepoProviderService.get_adapter")
    @override_settings(CODECOV_DASHBOARD_URL="https://app.codecov.io")
    def test_successful_upload_v2(
        self,
        mock_repo_provider_service,
        mock_dispatch_upload,
        mock_uuid4,
        mock_get_redis,
        mock_write_file,
    ):
        class MockRepoProviderAdapter:
            async def get_commit(self, commit, token):
                return {"message": "This is not a merge commit"}
    
        mock_get_redis.return_value = MockRedis()
        mock_repo_provider_service.return_value = MockRepoProviderAdapter()
        mock_uuid4.return_value = (
            "dec1f00b-1883-40d0-afd6-6dcb876510be"  # this will be the reportid
        )
    
        query_params = {
            "commit": "b521e55aef79b101f48e2544837ca99a7fa3bf6b",
            "token": "a03e5d02-9495-4413-b0d8-05651bb2e842",
            "pr": "456",
            "branch": "",
            "flags": "",
            "build_url": "",
            "package": "",
        }
    
        response = self._post(
            kwargs={"version": "v2"}, query=query_params, data="coverage report"
        )
    
        assert response.status_code == 200
    
        headers = response.headers
    
        assert headers["access-control-allow-origin"] == "*"
        assert (
            headers["access-control-allow-headers"]
            == "Origin, Content-Type, Accept, X-User-Agent"
        )
        assert headers["content-type"] != "text/plain"
    
        archive_service = ArchiveService(self.repo)
        datetime = timezone.now().strftime("%Y-%m-%d")
        repo_hash = archive_service.get_archive_hash(self.repo)
        expected_url = f"v4/raw/{datetime}/{repo_hash}/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt"
    
>       mock_write_file.assert_called_with(expected_url, ANY, ANY)
E       AssertionError: expected call not found.
E       Expected: write_file('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', <ANY>, <ANY>)
E         Actual: write_file('.../C6F3531E9BA57B5CE7FF4ED1BA28CC99/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)
E       
E       pytest introspection follows:
E       
E       Args:
E       assert ('v4/raw/2024...erage report') == ('v4/raw/2024... <ANY>, <ANY>)
E         
E         Right contains one more item: <ANY>
E         Use -v to get more diff
E       Kwargs:
E       assert {'is_already_gzipped': False} == {}
E         
E         Left contains 1 more item:
E         {'is_already_gzipped': False}
E         Use -v to get more diff

upload/tests/test_upload.py:1019: AssertionError

Class name: upload.tests.test_upload.UploadHandlerRouteTest
Test name: test_successful_upload_v2_slash

self = <MagicMock name='write_file' id='139671276570912'>
args = ('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', <ANY>, <ANY>)
kwargs = {}
expected = call('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', <ANY>, <ANY>)
actual = call('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)
_error_message = <function NonCallableMock.assert_called_with.<locals>._error_message at 0x7f077c6e7b00>
cause = None

    def assert_called_with(self, /, *args, **kwargs):
        """assert that the last call was made with the specified arguments.
    
        Raises an AssertionError if the args and keyword args passed in are
        different to the last call to the mock."""
        if self.call_args is None:
            expected = self._format_mock_call_signature(args, kwargs)
            actual = 'not called.'
            error_message = ('expected call not found.\nExpected: %s\n  Actual: %s'
                    % (expected, actual))
            raise AssertionError(error_message)
    
        def _error_message():
            msg = self._format_mock_failure_message(args, kwargs)
            return msg
        expected = self._call_matcher(_Call((args, kwargs), two=True))
        actual = self._call_matcher(self.call_args)
        if actual != expected:
            cause = expected if isinstance(expected, Exception) else None
>           raise AssertionError(_error_message()) from cause
E           AssertionError: expected call not found.
E           Expected: write_file('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', <ANY>, <ANY>)
E             Actual: write_file('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)

.../local/lib/python3.12/unittest/mock.py:949: AssertionError

During handling of the above exception, another exception occurred:

self = <upload.tests.test_upload.UploadHandlerRouteTest testMethod=test_successful_upload_v2_slash>
mock_repo_provider_service = <MagicMock name='get_adapter' id='139670663700240'>
mock_dispatch_upload = <MagicMock name='dispatch_upload_task' id='139670137903856'>
mock_uuid4 = <MagicMock name='uuid4' id='139671605272432'>
mock_get_redis = <MagicMock name='get_redis_connection' id='139670130925712'>
mock_write_file = <MagicMock name='write_file' id='139671276570912'>

    @patch("shared.api_archive.archive.ArchiveService.write_file")
    @patch("upload.views.legacy.get_redis_connection")
    @patch("upload.views.legacy.uuid4")
    @patch("upload.views.legacy.dispatch_upload_task")
    @patch("services.repo_providers.RepoProviderService.get_adapter")
    @override_settings(CODECOV_DASHBOARD_URL="https://app.codecov.io")
    def test_successful_upload_v2_slash(
        self,
        mock_repo_provider_service,
        mock_dispatch_upload,
        mock_uuid4,
        mock_get_redis,
        mock_write_file,
    ):
        class MockRepoProviderAdapter:
            async def get_commit(self, commit, token):
                return {"message": "This is not a merge commit"}
    
        mock_get_redis.return_value = MockRedis()
        mock_repo_provider_service.return_value = MockRepoProviderAdapter()
        mock_uuid4.return_value = (
            "dec1f00b-1883-40d0-afd6-6dcb876510be"  # this will be the reportid
        )
    
        query_params = {
            "commit": "b521e55aef79b101f48e2544837ca99a7fa3bf6b",
            "token": "a03e5d02-9495-4413-b0d8-05651bb2e842",
            "pr": "456",
            "branch": "",
            "flags": "",
            "build_url": "",
            "package": "",
        }
    
        response = self._post_slash(
            kwargs={"version": "v2"}, query=query_params, data="coverage report"
        )
    
        assert response.status_code == 200
    
        headers = response.headers
    
        assert headers["access-control-allow-origin"] == "*"
        assert (
            headers["access-control-allow-headers"]
            == "Origin, Content-Type, Accept, X-User-Agent"
        )
        assert headers["content-type"] != "text/plain"
    
        archive_service = ArchiveService(self.repo)
        datetime = timezone.now().strftime("%Y-%m-%d")
        repo_hash = archive_service.get_archive_hash(self.repo)
        expected_url = f"v4/raw/{datetime}/{repo_hash}/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt"
    
>       mock_write_file.assert_called_with(expected_url, ANY, ANY)
E       AssertionError: expected call not found.
E       Expected: write_file('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', <ANY>, <ANY>)
E         Actual: write_file('.../C7E92ABAB30E18809BEF745DF9CD8DDB/b521e55aef79b101f48e2544837ca99a7fa3bf6b/dec1f00b-1883-40d0-afd6-6dcb876510be.txt', b'coverage report', is_already_gzipped=False)
E       
E       pytest introspection follows:
E       
E       Args:
E       assert ('v4/raw/2024...erage report') == ('v4/raw/2024... <ANY>, <ANY>)
E         
E         Right contains one more item: <ANY>
E         Use -v to get more diff
E       Kwargs:
E       assert {'is_already_gzipped': False} == {}
E         
E         Left contains 1 more item:
E         {'is_already_gzipped': False}
E         Use -v to get more diff

upload/tests/test_upload.py:1100: AssertionError

Worker has some dedicated code to move an uploaded report from Redis over to GCS. Instead of having that code, we can just store the report into GCS in the first place. Additionally, this PR also cleans up the `services.archive`, replacing it fully with the `shared.api_archive.archive`. It also forwards all the task arguments directly to the `Upload` task, in addition to storing it in Redis, to be forward compatible to some more planned upload processing improvements.

matt-codecov

in future PRs could you put the actual logical changes in their own commit(s) so they're easier to find/review?

does worker already know that the uploads may no longer be in redis? or does that PR still need to be written

it doesn't look like the newer endpoint saves the report to redis in the first place

matt-codecov · 2024-11-06T22:09:24Z

upload/views/legacy.py

+                # FIXME(swatinem): this is using the `default_path` even for the shelter path,
+                # how does that even work? shouldn't this be the same as the `path`?
+                # or is shelter rather making sure that the `upload_params`
+                # `storage_path` and `url` are set appropriately?


do we need to answer this Q before merging?

Working on getting an answer in https://sentry.slack.com/archives/C04NC9NNPEG/p1730968157331319?thread_ts=1730814495.218099&cid=C04NC9NNPEG
I believe when using shelter, it ignores the generated upload_url in favor of returning the storage_path back to the user which is then both user for the upload on the user end, as well as the read on the worker end.

Swatinem · 2024-11-07T08:40:33Z

in future PRs could you put the actual logical changes in their own commit(s) so they're easier to find/review?

Yes, I will try :-) One thing kind of led to the other, and I ended up just completely remove the duplicated ArchiveService/Storage in API, rather switching over to the one in shared.

does worker already know that the uploads may no longer be in redis?

Yes, the worker has backwards compatibility code for:

Move the upload from redis to GCS based on redis_key, moving: https://github.com/codecov/worker/blob/d369849095133aaf11c77d0afb8b4afd0cd55481/tasks/upload.py#L190-L202
Otherwise, it will use the url key to find an already stored upload.
It will then also optionally create the Upload object based on the upload_id: https://github.com/codecov/worker/blob/d369849095133aaf11c77d0afb8b4afd0cd55481/tasks/upload.py#L499-L503

Swatinem requested a review from adrian-codecov November 5, 2024 13:43

Swatinem self-assigned this Nov 5, 2024

Swatinem requested review from a team as code owners November 5, 2024 13:43

Swatinem force-pushed the swatinem/v2-store branch from 53e4286 to 09b0993 Compare November 6, 2024 10:49

Swatinem force-pushed the swatinem/v2-store branch from 09b0993 to 38d2647 Compare November 6, 2024 11:31

Swatinem force-pushed the swatinem/v2-store branch from 38d2647 to 782fb8a Compare November 6, 2024 11:51

matt-codecov reviewed Nov 6, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store v2 reports in GCS instead of Redis #960

Store v2 reports in GCS instead of Redis #960

Swatinem commented Nov 5, 2024 •

edited

Loading

codecov bot commented Nov 5, 2024 •

edited

Loading

codecov-notifications bot commented Nov 5, 2024 •

edited

Loading

codecov-qa bot commented Nov 5, 2024 •

edited

Loading

github-actions bot commented Nov 6, 2024

codecov-public-qa bot commented Nov 6, 2024 •

edited

Loading

pytest

matt-codecov left a comment

matt-codecov Nov 6, 2024

Swatinem Nov 7, 2024

Swatinem commented Nov 7, 2024

Store v2 reports in GCS instead of Redis #960

Are you sure you want to change the base?

Store v2 reports in GCS instead of Redis #960

Conversation

Swatinem commented Nov 5, 2024 • edited Loading

codecov bot commented Nov 5, 2024 • edited Loading

Codecov Report

codecov-notifications bot commented Nov 5, 2024 • edited Loading

Codecov Report

codecov-qa bot commented Nov 5, 2024 • edited Loading

❌ 2 Tests Failed:

github-actions bot commented Nov 6, 2024

codecov-public-qa bot commented Nov 6, 2024 • edited Loading

❌ Failed Test Results:

pytest

matt-codecov left a comment

Choose a reason for hiding this comment

matt-codecov Nov 6, 2024

Choose a reason for hiding this comment

Swatinem Nov 7, 2024

Choose a reason for hiding this comment

Swatinem commented Nov 7, 2024

Swatinem commented Nov 5, 2024 •

edited

Loading

codecov bot commented Nov 5, 2024 •

edited

Loading

codecov-notifications bot commented Nov 5, 2024 •

edited

Loading

codecov-qa bot commented Nov 5, 2024 •

edited

Loading

codecov-public-qa bot commented Nov 6, 2024 •

edited

Loading