Fix `fullPage` rendering and loading #65

brandon93s · 2020-11-25T22:59:55Z

Opening for discussion/proposal. This PR does the following:

Fixes the scroll implementation for fullPage
Implements an auto-scale-down to capture full-page without repeat

Benefits/drawbacks:

Will scale down the device ratio until the image fits
Works well for small to medium-sized websites
Can result in a low-quality images for massive pages
Single image implementation; no complex image stitching

Todo:

Proceed w/ simple scale-down approach?
Document scaling approach when using fullPage
Tests

Expand for sample screenshots

Expand for related links

brandon93s · 2020-11-25T23:08:52Z

index.js

-		// Scroll one viewport at a time, pausing to let content load
-		const viewportHeight = viewportOptions.height;
-		let viewportIncrement = 0;
-		while (viewportIncrement + viewportHeight < bodyBoundingHeight) {


This loop was inaccessible before due to bodyBoundingHeight being an object. autoScroll below accomplishes the same goal of scrolling to force content load.

brandon93s · 2020-11-25T23:41:18Z

index.js

+		const maxTextureSize = await page.evaluate(() => {
+			const canvas = document.createElement('canvas');
+			const webGL = canvas.getContext('webgl');
+			return webGL.getParameter(webGL.MAX_TEXTURE_SIZE);
 		});


MAX_TEXTURE_SIZE is the number of pixels we can go in any dimmension before duplicate rendering will occur. This is determined at runtime as it can/will be different per environment. Using this, we can adjust our deviceScaleFactor to fit the page within this dimension.

Note: The same issue can technically happen horizontally. If there's a use case out there, we should scroll and scale in both dimensions ( maybe a follow-up PR? ).

brandon93s · 2020-11-26T00:58:33Z

index.js

@@ -119,8 +119,6 @@ const parseCookie = (url, cookie) => {
 	return returnValue;
 };

-const imagesHaveLoaded = () => [...document.images].map(element => element.complete);


This was returning an array of true/false before; always truthy and thus never waited.

sindresorhus · 2020-11-28T03:06:00Z

Thanks for working on this 🙌🏻

sindresorhus · 2020-11-28T03:06:21Z

Will scale down the device ratio until the image fits

I don't see how that will work in practice. Most websites are long enough to make the quality unusable.

sindresorhus · 2020-11-28T03:09:24Z

index.js

+			return !isBottom;
+		};
+
+		while (await autoScroll()) { /* eslint-disable-line no-await-in-loop */


Should this have a timeout? Otherwise, it would just go on forever on sites with infinite auto-loading.

sindresorhus · 2020-11-28T03:09:44Z

index.js

-			/* eslint-disable no-undef */
-			window.scrollTo(0, 0);
-			/* eslint-enable no-undef */
+		// Workaround for chromium height limitations:  https://bugs.chromium.org/p/chromium/issues/detail?id=770769#c12


Suggested change

// Workaround for chromium height limitations: https://bugs.chromium.org/p/chromium/issues/detail?id=770769#c12

// Workaround for Chromium height limitations: https://bugs.chromium.org/p/chromium/issues/detail?id=770769#c12

brandon93s · 2020-11-28T03:15:32Z

Will scale down the device ratio until the image fits

I don't see how that will work in practice. Most websites are long enough to make the quality unusable.

Review the sample screenshots in the description; hopefully, those can inform whether this is a viable compromise. Or, at a minimum, a bridge to a proper chromium fix. Otherwise, we'd need to explore the "image stitching" approach.

sindresorhus · 2020-11-28T06:02:20Z

Review the sample screenshots in the description; hopefully, those can inform whether this is a viable compromise. Or, at a minimum, a bridge to a proper chromium fix. Otherwise, we'd need to explore the "image stitching" approach.

I don't think it's viable. Yes, it can work on some sites, but for most, it cannot. And I don't like that the flag makes everything nondeterministic. You have no idea of the quality of the screenshot upfront. I'd honestly rather have no --full-page flag than a broken one.

sindresorhus · 2020-11-28T06:04:02Z

Could we have a temporary flag that changed --full-height to mean as much as possible by scaling while still keeping the set width/height options?

brandon93s · 2020-11-29T00:15:09Z

Could we have a temporary flag that changed --full-height to mean as much as possible by scaling while still keeping the set width/height options?

To make sure I understand: are you proposing a new flag / option called full-height which would work as proposed in this PR as an alternative to full-page while we wait on Chromium or implement an alternative? Would we remove full-page in the interim since it's broken?

sindresorhus · 2020-12-01T09:00:44Z

No, just a typo, fixed:

Could we have a temporary flag that changed --full-page to mean as much as possible by scaling while still keeping the set width/height options?

sindresorhus · 2020-12-01T09:02:07Z

Basically, scale the site down as much as possible while still respecting the set width and then capture as much height as possible, although it won't be everything.

sindresorhus · 2020-12-07T06:02:59Z

Maybe you can find some inspiration in https://github.com/morteza-fsh/puppeteer-full-page-screenshot

sindresorhus · 2021-02-07T13:01:41Z

Does https://github.com/puppeteer/puppeteer/releases/tag/v7.0.0 help at all with this?

brandon93s · 2021-02-07T16:20:44Z

Unfortunately, the underlying chromium limitation is still outstanding and is not addressed by the puppeteer release.

Kikobeats · 2021-11-06T18:14:43Z

I tested this PR locally with puppeteer v11 and I didn't any change compared with the default behavior.

target: https://www.apple.com/es/iphone/

before

![before](https://user-images.githubusercontent.com/2096101/140619676-a0df96d9-5306-4074-bd55-59f435513b92.png)

after

![after](https://user-images.githubusercontent.com/2096101/140619683-5f55c697-8141-48eb-acbb-2f367480ef38.png)

sindresorhus · 2022-12-10T20:17:23Z

Closing as this is not moving forward and we cannot keep it open forever.

Fullpage fixes

2ea5760

brandon93s commented Nov 25, 2020

View reviewed changes

brandon93s marked this pull request as ready for review November 25, 2020 23:12

No more funny math

53fbf25

brandon93s commented Nov 25, 2020

View reviewed changes

Enhance scroll detection

c68bd89

brandon93s commented Nov 26, 2020

View reviewed changes

Refactor token to promise

53938e3

sindresorhus reviewed Nov 28, 2020

View reviewed changes

Base automatically changed from master to main January 23, 2021 08:29

sindresorhus mentioned this pull request Jan 23, 2022

workaround fullPage clipping bug by using clip #95

Closed

sindresorhus closed this Dec 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix `fullPage` rendering and loading #65

Fix `fullPage` rendering and loading #65

brandon93s commented Nov 25, 2020 •

edited

Loading

brandon93s Nov 25, 2020 •

edited

Loading

brandon93s Nov 25, 2020

brandon93s Nov 26, 2020

sindresorhus commented Nov 28, 2020

sindresorhus commented Nov 28, 2020

sindresorhus Nov 28, 2020

sindresorhus Nov 28, 2020

brandon93s commented Nov 28, 2020

sindresorhus commented Nov 28, 2020

sindresorhus commented Nov 28, 2020

brandon93s commented Nov 29, 2020 •

edited

Loading

sindresorhus commented Dec 1, 2020

sindresorhus commented Dec 1, 2020

sindresorhus commented Dec 7, 2020

sindresorhus commented Feb 7, 2021

brandon93s commented Feb 7, 2021

Kikobeats commented Nov 6, 2021 •

edited

Loading

sindresorhus commented Dec 10, 2022

	// Workaround for chromium height limitations: https://bugs.chromium.org/p/chromium/issues/detail?id=770769#c12
	// Workaround for Chromium height limitations: https://bugs.chromium.org/p/chromium/issues/detail?id=770769#c12

Fix fullPage rendering and loading #65

Fix fullPage rendering and loading #65

Conversation

brandon93s commented Nov 25, 2020 • edited Loading

brandon93s Nov 25, 2020 • edited Loading

Choose a reason for hiding this comment

brandon93s Nov 25, 2020

Choose a reason for hiding this comment

brandon93s Nov 26, 2020

Choose a reason for hiding this comment

sindresorhus commented Nov 28, 2020

sindresorhus commented Nov 28, 2020

sindresorhus Nov 28, 2020

Choose a reason for hiding this comment

sindresorhus Nov 28, 2020

Choose a reason for hiding this comment

brandon93s commented Nov 28, 2020

sindresorhus commented Nov 28, 2020

sindresorhus commented Nov 28, 2020

brandon93s commented Nov 29, 2020 • edited Loading

sindresorhus commented Dec 1, 2020

sindresorhus commented Dec 1, 2020

sindresorhus commented Dec 7, 2020

sindresorhus commented Feb 7, 2021

brandon93s commented Feb 7, 2021

Kikobeats commented Nov 6, 2021 • edited Loading

sindresorhus commented Dec 10, 2022

Fix `fullPage` rendering and loading #65

Fix `fullPage` rendering and loading #65

brandon93s commented Nov 25, 2020 •

edited

Loading

brandon93s Nov 25, 2020 •

edited

Loading

brandon93s commented Nov 29, 2020 •

edited

Loading

Kikobeats commented Nov 6, 2021 •

edited

Loading