Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing Parent Request #, Duration, and Response Size fields #78

Open
BurnzZ opened this issue Jan 22, 2024 · 1 comment
Open

Missing Parent Request #, Duration, and Response Size fields #78

BurnzZ opened this issue Jan 22, 2024 · 1 comment
Labels

Comments

@BurnzZ
Copy link

BurnzZ commented Jan 22, 2024

Currently, the requests coming from scrapy_zyte_api.providers.ZyteApiProvider doesn't create the Parent Request # field in Scrapy Cloud.

image

In the example above, Request 1 should have a Parent Request # field which is missing.

Note that when reverting the changes from the PR #73, we get the Parent Request # field back which comes from the other request which is filtered in the new scrapinghub-entrypoint-scrapy version.

It would seem that after filtering out one of the duplicate requests, the request.meta.setdefault(HS_PARENT_ID_KEY) should somehow be copied into the other request (code ref).

Reproducible example:

class ParentSpider(scrapy.Spider):
    name = "parent"

    def start_requests(self):
        yield scrapy.Request(
            url="https://books.toscrape.com",
            callback=self.parse_nav,
        )

    def parse_nav(self, response: DummyResponse, navigation: ProductNavigation):
        for request in navigation.items:
            yield request.to_scrapy(
                callback=self.parse_item,
            )

    def parse_item(self, response: DummyResponse, product: Product):
        yield product
@BurnzZ BurnzZ added the bug label Jan 22, 2024
@BurnzZ
Copy link
Author

BurnzZ commented Feb 23, 2024

Note that Duration and Response Size are also missing.

@BurnzZ BurnzZ changed the title Missing Parent Request # field from ZyteApiProvider Missing Parent Request #, Duration, and Response Size fields Feb 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant