Understanding / Improving element sizing

[hatsumatsu]

Trying to integrate ar.js in a project and following the efforts to create an es6 build of thee lib 🙏 I'm currently trying to understand how the sizing of the video source and the threejs renderer works.

I would love to help out fixing potential bugs or simplifying the current API methods but might need some intro into how the current system works.

Reading the code this is what I understand:

Despite ARController and ARSource supporting custom width/height configurations they seem to be always initiated with a size of 640×480px. On mobile devices in portrait orientation, requesting a MediaStream with these constraints results in a video source of 480×640px (see test chart below). The ARController however does not react to this but still runs in 640x480. Logging its properties also shows that it runs in "landscape" orientation.

The threejs renderer –at least in some examples– also runs in 640×480 no matter the screen aspect ratio and the device orientation.

The webcam video is than scaled to fullscreen via ARSource.onResizeElement()

AR.js/three.js/src/threex/threex-artoolkitsource.js

Line 306 in c6661e2

ARjs.Source.prototype.onResizeElement = function () {

This looks good.

Next step, the threejs renderer is sized and positioned via ARSource.copyElementSizeTo()

AR.js/three.js/src/threex/threex-artoolkitsource.js

Line 360 in c6661e2

ARjs.Source.prototype.copyElementSizeTo = function (otherElement) {

This is where I got lost: Instead of just copying the style properties calculated in ARSource.onResizeElement() there is a special handling of portrait mode including some magic numbers:

AR.js/three.js/src/threex/threex-artoolkitsource.js

Line 372 in c6661e2

otherElement.style.width = (parseInt(otherElement.style.height) * 4 / 3) + "px";

This probably handles the missing sync between the video source and the ARController and threejs renderer size...

Given that some examples resize the threejs renderer based on the (assumed) video size and some based on the browser window it seems like pure magic to me that this works at all 🔮

Could anyone with a deeper knowledg of arjs let me know if I got this right or I am missing something.
Thanks a lot!

Tested on Pixel 5 & iPhone 6

	Video Source	threejs renderer
Portrait	480x640	640x480
Landscape	640x480	640x480

[hatsumatsu]

This is what I would like to try:

Goal

Initiate the ARSource, the ARContext and the threeJS renderer with the same dimensions to avoid wasting resources on offscreen calculations and simplify sizing and positioning.

ARSource video source dimensions === ARContext canvas dimensions === threejs renderer dimensions (to be precise same aspect ratio, probably higher resolution)

The sizing and positioning of the source element and the threejs canvas could then be done via CSS only:

position: absolute;
left: 0;
top: 0;
width: 100%;
height: 100%;
object-fit:  cover;

This would also free us from running everything in a fullscreen layout and would enable having an instance in any container element.

Steps to achieve this

1. Make ARSource aware of their source dimensions

Currently THREEx.ArToolkitSource has no property for the actual source dimensions. the sourceWidthand sourceHeight passed as parameters are just hints. Saving the actual source dimensions would also save us from re-calculating it in methods like resizeElement():

AR.js/three.js/src/threex/threex-artoolkitsource.js

Line 314 in aaad984

// compute sourceWidth, sourceHeight

2. Refactor examples

... to work like:

• Init THREEx.ArToolkitSource with the default of 640x480px
• Init THREEx.ArToolkitContext with the actual source dimensions reported from THREEx.ArToolkitSource
• Init threejs renderer with the actual source dimensions reported from THREEx.ArToolkitSource
• Add orientationchange event to re-init the above to account for orientation change.

Optional

Introduce an optional container parameter to THREEx.ArToolkitSource to specify a DOM element the source will be placed in. This would make the whole thing design-agnostic and enables the simplified sizing / positioning described above.

Questions

@kalwalt @nicolocarpignoli @nickw1 (+ anyone who knows the inner working of this lib better than me 🙏) Do you think this could be a good idea or do you see any conceptual problems with how jsartoolkit works?

Could initiating a portrait-oriented THREEx.ArToolkitContext work with the mysterious camera_para.dat file? 🔮

Thanks a lot!

[nickw1]

@hatsumatsu thanks for your input! Just to make clear, I haven't done any work on the jsartoolkit side of things (markers and NFT), only the location-based part of AR.js. @kalwalt and @nicolocarpignoli would probably have a better idea.

[kalwalt]

@hatsumatsu i merged the ES6 branch into dev. I thinking that we could remove the portrait and landscape mode in artoolkit5-js

  } else {


     this.ctx.save();


     if(this.orientation === 'portrait') {
       this.ctx.translate(this.canvas.width, 0);
       this.ctx.rotate(Math.PI / 2);
       this.ctx.drawImage(sourceImage, 0, 0, this.canvas.height, this.canvas.width); // draw video
     } else {
       this.ctx.drawImage(sourceImage, 0, 0, this.canvas.width, this.canvas.height); // draw video
     }


     this.ctx.restore();


     let imageData = this.ctx.getImageData(0, 0, this.canvas.width, this.canvas.height);
     data = imageData.data;
   }

https://github.com/AR-js-org/artoolkit5-js/blob/3f4f24603e7b19385c26d7d6006f74abe8cd70c2/src/ARController.js#L1328-L1344

and removing all the code containing canvas and HTMLElement inside artoolkit5-js. Passing only imageData to the _copyImageToHeap(sourceImage) {
https://github.com/AR-js-org/artoolkit5-js/blob/3f4f24603e7b19385c26d7d6006f74abe8cd70c2/src/ARController.js#L1310 and then to the arController.process() function.
In this way it became agnostic of orientation and instead we could do this logic inside Ar.js.