How does the anti-aliasing version works ?

[lgarczyn]

I've looked through the code, but I can't find anywhere how the AA version works.

It's not a render texture per the Readme, but I don't see how you can use a stencil to do this kind of effect.

[mob-sakai]

The basic principles of AA masking mode can be easily tested even without SoftMaskForUGUI.

First, here’s the default Mask.
As you can see, the jagged edges are clearly visible.

Next, let’s try reducing Image.color.a.
Surprisingly, the jagged edges become less noticeable.
This is because the clipping sensitivity indirectly increases.

However, as it stands, it will disappear when CanvasGroup.alpha is reduced, so a bit more adjustment is needed.

[lgarczyn]

I feel like I must be missing some important points, but wouldn't reducing the image.color.a just... Make the image transparent?

[mob-sakai]

The Image.color.a I’m referring to is the masking graphic (i.e., a graphic with a Mask component).

In the UI/Default shader, the stencil mask is implemented as follows:

#ifdef UNITY_UI_ALPHACLIP
clip (color.a - 0.001);
#endif

In other words, pixels with an alpha value near zero are clipped.
Looking at the alpha histogram of the masking graphic, there’s a noticeable gap at an alpha value of 0, which results in jagged edges.

Now, let’s try multiplying the alpha by a small value.
The gap becomes smaller than before, and the edges are now smooth, without jagged.

[lgarczyn]

The Image.color.a I’m referring to is the masking graphic (i.e., a graphic with a Mask component).

In the UI/Default shader, the stencil mask is implemented as follows:

#ifdef UNITY_UI_ALPHACLIP
clip (color.a - 0.001);
#endif

In other words, pixels with an alpha value near zero are clipped.
Looking at the alpha histogram of the masking graphic, there’s a noticeable gap at an alpha value of 0, which results in jagged edges.

Now, let’s try multiplying the alpha by a small value.
The gap becomes smaller than before, and the edges are now smooth, without jagged.

Right, but whatever happens, the mask still only has 1 bit of depth. Offsetting the clipping value of the mask will simply appear to grow or shrink the mask by one pixel, but shouldn't give smooth edges.

[mob-sakai]

In clipping, referencing the bilinearly interpolated pixel values for small gaps provides a stencil result with a higher resolution than the actual pixel size.

BTW, this method cannot be used with point-filtered sprites.

[lgarczyn]

In clipping, referencing the bilinearly interpolated pixel values for small gaps provides a stencil result with a higher resolution than the actual pixel size.

BTW, this method cannot be used with point-filtered sprites.

Oh! So the clipping is still perfectly binary, but it solves the issue of a mask with too low resolution

[mob-sakai]

Exactly!