Question regarding scroll margins and SIXELs

[jerch]

Sorry, if you already stated it elsewhere and I overlooked it - I wonder how a VT340 deals with scroll margins in SIXEL phase.

My idea so far - imho the 300+ line only supported top/bottom restrictions and left/right was a 400+ thing (plz correct me, if thats already wrong).

Now regarding SIXELs - do scroll margins affect SIXEL printing by any means? Does DECOM change here anything? Has DECAWM any influence on SIXEL printing width?

My current expectations from applying basic text mechanics are (given SIXEL scrolling is set):

• printing SIXELs within an active text scroll region cannot leave the region, thus SIXELs would not overlap a bottom scroll border, instead the image gets scrolled within the scroll region, if it happens to be higher. I would assume the same for left/right borders, but that one prolly cannot be tested on a VT340.
• SIXEL printing outside of an active text scroll region would dump the pixels to full viewport extend, until it may cross scroll borders into an active region (then the rule above gets applied for image parts within the scroll region)

Following your findings in other threads and what you wrote so far about SIXEL mechanics my assumption above is certainly wrong. From your writings I would conclude, that SIXELs are not scroll border aware at all, due their more or less direct screen buffer printing skipping most text buffer indirections.

Could you shed light on that mystery on my end? Ofc if needed, I can create a few test cases.

[j4james]

Scroll margins definitely do have an effect - there 's a margin clipping test case here:
https://github.com/hackerb9/vt340test/blob/main/j4james/margin_clipping.sh
And you can see the results here:
https://github.com/hackerb9/vt340test/blob/main/j4james/margin_clipping.png

This only tests top/bottom margins, because that's all that is supported on the VT340, but I'd expect left/right to work the same way (except for scrolling).

I haven't tested anything related to DECOM or DECAWM, because I can't actually think of any difference they would make to sixel output (other than their effect on the text cursor position, which ultimately determines the sixel starting position). What exactly did you have in mind?

[hackerb9]

From your writings I would conclude, that SIXELs are not scroll border aware at all, due their more or less direct screen buffer printing skipping most text buffer indirections.

Please let me know where I wrote that as I should definitely make it clearer. I would say that sixels typically work as if they were part of the text. The illusion is only broken in the corner cases (as @j4james has documented).

Could you shed light on that mystery on my end? Ofc if needed, I can create a few test cases.

You are correct that there are no side margins on a VT340 and @j4james is correct that top and bottom margins do affect sixels.

DECAWM does not impact sixels. The documentation says that anything outside of graphics window is to be simply truncated, not wrapped. (Note that the graphics window is not necessarily the same as the what the text window is displaying. By moving the page, it is possible to draw sixels that appear to be truncated offscreen at the time of drawing and then scroll them onto the screen later. Even so, sixels are never wrapped.)

DECOM should have exactly the impact you'd expect, positioning the cursor relative to the screen or the scrolling region, but I have not tested it.

If you would send a few test cases, I would happily run them for you.

[jerch]

Thx for the answers.

About DECOM:
The question comes from the fact, that if DECOM is set, the text cursor cannot move outside of margins, thus I was wondering if sixel painting would be restricted likewise (e.g. cannot paint pixels below bottom margin on vt340).
And yes, I would also assume to apply the same behavior to left/right borders (xtermjs currently does not support those either, but I want to add support as it is a very convenient text output mode for complicated screen setups.)

About DECAWM:
This was just to make sure, DECAWM would never wrap sixel image parts, that would get truncated at the right border, to the next line below (in the same sense as it wraps text content). So the always truncating graphics window behavior answers that.
This question might look weird in the first place, but we kinda have that issue on almost all TEs implementing sixels with a tile-to-textcell abstraction under reflow logic. It gets evident when you try to resize those terminals with a sixel dump in the viewport.

Prolly will make another test case with scroll margins set and text scrolling happening, because its still not quite clear to me, how sixels pixels (from sixels) on the graphics window will exactly "morph" under these conditions.

[j4james]

I think the margin_clipping test I linked above should actually cover everything you're asking. It tests the rendering of an image starting above the top margin (which is allowed), and extending into the margin area until it overflows the bottom margin, which triggers a scroll.

The segment of the image that was above the margin remains static, while the segment inside the margin boundaries is scrolled up. This basically results in the middle segment of the image being erased as it is scrolled under the top margin.

This first scroll is then followed by an RI (reverse index) below the top margin to test scrolling the image down again (an additional segment of the test pattern is output to fill the line gap that is created by that movement).

Finally an image is output below the bottom margin, large enough that it should actually overflow the bottom of the page. But this doesn't display anything, and doesn't cause the page to scroll, because images below the bottom margin are ignored.

If there is an error in your implementation you are likely to see areas of red in the image where something hasn't been appropriately clipped or scrolled out of view.

This is all in the test pattern on the left, btw. The one on the right is testing how the margin effects the background fill.

The question comes from the fact, that if DECOM is set, the text cursor cannot move outside of margins, thus I was wondering if sixel painting would be restricted likewise (e.g. cannot paint pixels below bottom margin on vt340).

As mentioned above, even without DECOM set you can't paint below the bottom margin, so I don't think setting DECOM is likely to make any difference.

[hackerb9]

Prolly will make another test case with scroll margins set and text scrolling happening, because its still not quite clear to me, how sixels pixels (from sixels) on the graphics window will exactly "morph" under these conditions.

Sure, send it on over and I'll try it out.

[hackerb9]

I was curious about scrolling regions and wrote up a little script to try it out myself. I'm not sure that I'm testing exactly what is being asked for, but the results are pleasingly unsurprising. Nothing weird going on here:

Without scrolling regions:

With a scrolling region:

With both a scrolling region and origin mode enabled:

[jerch]

Wow thx a lot - yepp, thats exactly what I would have tested.

[jerch]

To summarize your test results, thats what I get from it (plz correct any point that I got wrong to avoid the next broken implementation detail, haha):

1. if the text cursor cannot leave the scroll region (DECOM on), a sixel dump within would scroll rows like text, thus scroll the sixels up in the scroll region, truncating when reaching the top border (as seen in 3rd picture)
2. if a scroll region smaller than the viewport is set, things are conditional:
   • if the initial text cursor was outside upon sixel start, it would print sixels normally (upper half of 2nd picture)
   • once the cursor reaches the scroll region, it gets caught and cannot leave it anymore, treated as under rule 1. (lower part of 2nd picture). Bottom border cannot be crossed by normal sixel printing progression.
3. No or full viewport scroll region is just a special case of the 1. rule applied to all cols/rows.

If thats right, things can be applied in a similar fashion to left/right borders. What might be surprising there is the switch from outside to inside scroll borders handling. Example:

EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
EEEEEEEEEEEEEXEEEEEEEEEEEEEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEE#########EEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEE#########EEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEE#########EEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEE#########EEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE

Where '###' denotes the scrolling window and you start outputting sixels from position X. Following the rules above I'd assume the following output:

EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
EEEEEEEEEEEEEa0sssa1sssa2sssa3sssa4sssa5
EEEEEEEEEEEEEb0sssb1sssb2sssb3sssb4sssb5
EEEEEEEEEEEEEc0sssw1sssw2sEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEsx1sssx2sEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEsy1sssy2sEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEsz1sssz2sEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE

'a0' denotes here the row ('a') and the col('0') within the sixel data, as they would have mapped to text cells on a big flat canvas. The scrolling happens only in the window (see how 'w' follows 'b'), futhermore the cursor cannot leave anymore the scrolling window (truncated on left/right borders).
Would that be the an acceptable handling for left/right borders, what you think?

(This is kinda offtopic in a repo for vt340 tests. I can move it elsewhere, if you want.)

[j4james]

if the text cursor cannot leave the scroll region (DECOM on), a sixel dump within would scroll rows like text, thus scroll the sixels up in the scroll region, truncating when reaching the top border (as seen in 3rd picture)

Just to be absolutely clear, DECOM has no effect on the image itself in that test - only the text position. If you replace every occurrence of echo -n ${CSI}H with echo -n "${CSI}${top}H" instead of setting DECOM, you'll get the exact same result. All DECOM is doing is changing the cursor position, which in turn effects the position at which the image is output.

Would that be the an acceptable handling for left/right borders, what you think?

I don't think I'd agree with this. Lines a and b are definitely wrong, because even if the image could start off outside the left margin, it can't extend past the right margin, so the first two lines should also be truncated at that point. And the more I think about, the more I think that images shouldn't even be allowed outside the left margin.

If that's not the case, then I'd expect the image to always be visible outside the left margin, and truncated on the right, even once inside the top/bottom margins. But then it's going to get really weird once you start scrolling, because only the center portion of the image would scroll, but the last sixel line on the left would still continue to be redrawn.

By that logic, I'd expect your test pattern to look like this (note the z0 in the bottom left):

EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
EEEEEEEEEEEEEa0sssa1sssa2sEEEEEEEEEEEEEE
EEEEEEEEEEEEEb0sssb1sssb2sEEEEEEEEEEEEEE
EEEEEEEEEEEEEc0sssw1sssw2sEEEEEEEEEEEEEE
EEEEEEEEEEEEEd0sssx1sssx2sEEEEEEEEEEEEEE
EEEEEEEEEEEEEe0sssy1sssy2sEEEEEEEEEEEEEE
EEEEEEEEEEEEEz0sssz1sssz2sEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE

Only in practice it would be even weirder, because the sixel height is not likely to be an exact multiple of the line height, so the z0 area would be a weird amalgamation of sixel segments overriding each other at different y offsets.

In short, I think the most sensible approach would be to disallow images starting outside the left margin. This follows a similar pattern to the ICH and DCH controls, which also aren't allowed outside the left/right margins, but can be used outside the top/bottom margins.

[jerch]

@j4james I see your points, and I think we only differ in 2 aspects:

• never cross right border
  While my version allows that, if the cursor is not within the scroll window (basically testing for both dimensions at once), your version would check (and forbid) border crossing separately. I want that to be as close as possible to text behavior and gonna do some tests on the VT525 regarding scroll borders, once it is up and running.
• continue left sixels even if outside
  Hmm this looks not right to me. Why should the image disregard the left border and always start at initial offset? It looks to me like a violation of the border constraints, that I tend to weigh higher. Also it is not much better in terms of "image preservation" as it introduces more nasty "pixel amalgamation" side effects with that e-z line succession and in the lines above (c-w, d-x, e-y).

In short, I think the most sensible approach would be to disallow images starting outside the left margin. This follows a similar pattern to the ICH and DCH controls, which also aren't allowed outside the left/right margins, but can be used outside the top/bottom margins.

Yes, this seems like a good way to circumvent most of those ugly pixel amalgamation effects.

Edit:
About your DECOM remark - yes thats totally clear.

[j4james]

I want that to be as close as possible to text behavior and gonna do some tests on the VT525 regarding scroll borders, once it is up and running.

I completely agree. I thought what I was proposing was close to the text behavior, but I'm beginning to have my doubts now. I just did a couple of tests on the 5 terminals I know of that support left/right margins, and I got 4 different results! At least most of them agreed on not ever crossing the right border, but I wouldn't necessarily trust that that is correct. I'm happy to wait and see what your VT525 really does.

Btw, I was thinking it might be nice to setup a documentation/testing repo similar to this one for your VT525. Beside the DECPS audio, and this margin discussion, I'm sure there are a lot more features that aren't fully documented that would be worth testing.

[hackerb9]

To summarize your test results, thats what I get from it (plz correct any point that I got wrong to avoid the next broken implementation detail, haha):

1. if the text cursor cannot leave the scroll region (DECOM on), a sixel dump within would scroll rows like text, thus scroll the sixels up in the scroll region, truncating when reaching the top border (as seen in 3rd picture)

Yes, that is the way sixels work when DECSDM (Sixel Display Mode) is disabled.

2. if a scroll region smaller than the viewport is set, things are conditional:
   
   * if the initial text cursor was outside upon sixel start, it would print sixels normally (upper half of 2nd picture)
   * once the cursor reaches the scroll region, it gets caught and cannot leave it anymore, treated as under rule 1. (lower part of 2nd picture). Bottom border cannot be crossed by normal sixel printing progression.

Correct, but the cursor starting outside or inside the scroll region does not need to be handled specially, other than ensuring it is never below the bottom of the region.

3. No or full viewport scroll region is just a special case of the 1. rule applied to all cols/rows.

Again, correct, but I think it is actually simpler than that. No matter whether DECOM is on or off and whether a scrolling region is defined some things are always true (presuming DECSDM is disabled):

1. Sixels scroll exactly like text.
2. An unset scrolling region is equivalent to all lines being the scrolling region.
3. If the cursor is below the scrolling region before printing output, it is moved to the last line of the region.
4. When the region scrolls up, the topmost line is replaced with the line below it, which in turn is replaced by the line below that, and so on. That continues until the bottommost line, which is simply cleared.

Given the above, the only contingency is whether DECOM is set before output. If it is on and the cursor is above the region, then the cursor is moved to the first line of the region. Note that this is the same for text and sixel output, so it is likely already implemented.

If thats right, things can be applied in a similar fashion to left/right borders. What might be surprising there is the switch from outside to inside scroll borders handling.

The VT525 is beyond my knowledge, but I can only say that, whatever you do, make sure sixel scrolling in regions works exactly the same as text scrolling. (Unless DECSDM is set, in which case, sixels never trigger scrolling and output is truncated to the region).

(This is kinda offtopic in a repo for vt340 tests. I can move it elsewhere, if you want.)

It's fine here and is kind of interesting to me, although I cannot help very much.

As for a setting up a special repository for in-depth VT525 documentation & testing, I think that'd be cool. But, until you do, feel free to use these pages to discuss or even submit pull requests with documentation and I'll publish them here.

[jerch]

@j4james Yes thats a good idea, I am sure there are a lot of things to test, the sheer amount of sequences of the late VT devices is overwhelming compared to whats supported in all common TEs these days (which makes them all look like poor clones). I am not advocating for full vt500+ emulation (imho some things are just dead and not worth to be revived), yet it is a good opportunity to fix emulation details for existing stuff, and to dig for concepts and sequence gems, that are worth to be implemented (like left/right borders, some mass actions like rect area functions etc. pp, funny - most things are even older from 300/400 line).

@hackerb9 Yepp, your list of contraints is the better overall summary, thx.

And about DECSDM, what I shamelessly left out of considerations above - can you explain more in detail, how that behaves? (sorry for being a slowpoke) Given DECSDM is set:

• Is DECOM respected for sixel painting start, thus would implicitly start painting at scroll region offset (DECOM on) or viewport home (DECOM off)?
• If the above is true, how does a smaller scroll region affect sixel output, that started outside (DECOM off) at the bottom border? Would it just stop there without scrolling (effectively truncating lower image parts)? I'd assume it never will pass the bottom border from normal image progression, no matter how DECSDM or DECOM is set. If thats true, scroll border constraints would always be active for sixel output, independent of any other setting.

Edit: About no scroll region being explicitly set - the vt520 docs state somewhere, that the margins are set to the viewport margins by default (lines per screen for bottom, page borders for left/right). If thats also true for vt340, an implementation could be simplified to always respect scroll margins, which happens to be the same as the full viewport in that case. (Note that the page concept is not reflected on most TEs, but kinda reduced to a single one.)

[dankamongmen]

if someone has a VT525, i'd love to know the story on #16 (does DECERA apply to sixels?)

[jerch]

@dankamongmen Imho VT525 cannot really answer that, because DEC never released a VT500+ device with sixel caps (prolly would have been VT540). For some reason they stopped the whole sixel thing with VT level4, where DECERA belongs to.

But reading the mechanics about DECERA imho gives us enough clues to answer that question - it is meant to replace stuff in the given rect with SP. Now writing SPs yourself into those cells would erase sixel pixels, thus I'd say we should treat it the same way with DECERA. (all imho)

[dankamongmen]

@dankamongmen Imho VT525 cannot really answer that, because DEC never released a VT500+ device with sixel caps (prolly would have been VT540). For some reason they stopped the whole sixel thing with VT level4, where DECERA belongs to.

But reading the mechanics about DECERA imho gives us enough clues to answer that question - it is meant to replace stuff in the given rect with SP. Now writing SPs yourself into those cells would erase sixel pixels, thus I'd say we should treat it the same way with DECERA. (all imho)

ahhhh, didn't realize that about the vt5xx, thanks. I agree with your interpretation, but @ThomasDickey seems to want confirmation before he'll merge a patch if mine effecting this behavior.

[jerch]

Ofc with newer SGR caps things are getting way more tricky - what if the terminal supports transparent in BG? Should this keep the current pixels untouched? Or is SP an empty char, thus would still wipe FG, and we treat sixels as FG content by default (effectively wiping the cell and let some other background thingy shine through)? I currently have no answer to that, and support for transparent in BG is still very limited among TEs.

Edit:
My gut feeling says - lets treat sixel pixels as FG content, thats composable from previous FG/BG content (incl. sixel pixels), and SP is an empty char clearing FG pixels. Would make things much easier on implementation side.

[hackerb9]

ahhhh, didn't realize that about the vt5xx, thanks.

@jerch is right. That's why I purchased a VT340 to test and document: it was (as far as I know) the most advanced machine DEC made for rendering sixels. (I could be mistaken as I think they may have made some sixel laser printers, but that's a whole 'nother ball of wax.)

I agree with your interpretation, but @ThomasDickey seems to want confirmation before he'll merge a patch if mine effecting this behavior.

Hard to confirm something that never existed before. But the maxim that sixels are supposed to act like text is a reasonable one to follow and I think the burden of confirmation in hardware should be on a behaviour that treats sixels differently. That is, I would gently suggest that Mr. Dickey consider accepting @dankamongmen's patch so that sixels will be treated like text, unless evidence to the contrary in DEC hardware surfaces.¹

---

¹ Who knows, maybe archaeologists will unearth a VT540 from an alternate timeline... Speaking of which, some digital-anthropologist should interview the old Digital employees and ask: “Who or what killed sixels?” From my experience with a VT340, my bet is on baudrate limitations, especially when compared to PCs which had no such limit.

[j4james]

Is DECOM respected for sixel painting start, thus would implicitly start painting at scroll region offset (DECOM on) or viewport home (DECOM off)?

That's something I never considered, but my guess is that DECOM (and margins in general) would have no effect in DECSDM mode. I've always thought of it as a kind of VT240 compatibility mode, and I know that margins don't constrain the image on the VT240 (with or without DECOM). I'd be very interested to see if that's also true of the VT340.

[hackerb9]

Ofc with newer SGR caps things are getting way more tricky - what if the terminal supports transparent in BG?

There is a "transparent" color for text background in the VT525? I could see that being potentially useful! Do they also have a "contrast" color for foreground which is guaranteed to contrast with whatever color the background is?

[Side note: I was just writing a utility that writes text via sixels (using convert label:foo sixel:-) and was considering using transparency for the background so I wouldn't have to detect the user's terminal colors. I decided against it because sixel has only 1-bit transparency which ruins the antialiasing vector fonts need at low resolution. That and there is no "contrast" color. I have to detect the foreground color anyhow, so I might as well get the background.]

I may be wrong, but to my mind sixels (and a space character) are both foreground. I would expect a space character would obliterate any sixels just as sixels would obliterate any text. However, I would hope that text with a transparent background could overlay on top of sixels in the same way that sixels with a transparent background already overlay on top of text. Why should that be when transparent text doesn't accumulate upon other text? My reasoning is simple: because it would be immensely useful.

[dankamongmen]

I may be wrong, but to my mind sixels (and a space character) are both foreground. I would expect a space character would obliterate any sixels just as sixels would obliterate any text.

this is the case on all terminal emulators of which i am aware save Alacritty's graphics branch (and of course it does not apply to other graphics protocols).

However, I would hope that text with a transparent background could overlay on top of sixels in the same way that sixels with a transparent background already overlay on top of text. Why should that be when transparent text doesn't accumulate upon other text? My reasoning is simple: because it would be immensely useful.

you can do this with the kitty protocol. if you haven't read my https://nick-black.com/dankwiki/index.php/Theory_and_Practice_of_Sprixels, you might enjoy it.

and yes, this is indeed useful.

[hackerb9]

Is DECOM respected for sixel painting start, thus would implicitly start painting at scroll region offset (DECOM on) or viewport home (DECOM off)?

That's something I never considered, but my guess is that DECOM (and margins in general) would have no effect in DECSDM mode. I've always thought of it as a kind of VT240 compatibility mode, and I know that margins don't constrain the image on the VT240 (with or without DECOM). I'd be very interested to see if that's also true of the VT340.

BINGO! @j4james is right again. I tested it and DECSDM ignores DECOM and DECSTBM, just like the VT240. Here's the MediaCopy from my VT340.

[jerch]

There is a "transparent" color for text background in the VT525? I could see that being potentially useful! Do they also have a "contrast" color for foreground which is guaranteed to contrast with whatever color the background is?

Ah nope, the VT525 only supports 16 colors. The transparent SGR variant is much newer and defined in ITU T.416 for extended colors (e.g. CSI 38 : 1 m or CSI 48 : 1 m). What I was trying to say above regarding this - we need an agreement on how sixel pixels are related to FG/BG for those newer SGR capabilities, otherwise TEs will handle these cases different again.
The question, how that should be handled and how sixels would fit in, is greatly related to the layering discussion in contour-terminal/terminal-good-image-protocol#11 (ideally that protocol can use sixel as transport encoding in the end).

I may be wrong, but to my mind sixels (and a space character) are both foreground. I would expect a space character would obliterate any sixels just as sixels would obliterate any text.

Yes thats the most straight forward take on the matter imho. But is that true for transparent sixels on a VT340? I'd have thought, that text glyph pixels (and BG color pixels) stay on the screen for unset sixel pixels?
I agree with SP to treat it as "clearing" character.

I would hope that text with a transparent background could overlay on top of sixels in the same way that sixels with a transparent background already overlay on top of text. Why should that be when transparent text doesn't accumulate upon other text? My reasoning is simple: because it would be immensely useful.

For transparent BG imho that should be discussed, whether the transparency means to leave prvious BG untouched, or to clear and let something further in the background shine through. Leaving previous content untouched gives the better caps to app devs, but is likely to be much harder to implement on TE side (pulls in image composition with shifting layers).
For transparent FG - haha I have no idea yet, what to make out of this without wading in deep image processing waters (it is also legal in terms of the T.416 definitions).

BINGO! @j4james is right again. I tested it and DECSDM ignores DECOM and DECSTBM, just like the VT240. Here's the MediaCopy from my VT340.

Sweet, thx for testing. Though your results make it a poor mans graphics plot thingy, guess it is only there for vt240 backward compat?

[j4james]

I would hope that text with a transparent background could overlay on top of sixels in the same way that sixels with a transparent background already overlay on top of text. Why should that be when transparent text doesn't accumulate upon other text? My reasoning is simple: because it would be immensely useful.

I don't really see how you can allow one without the other. Say you write out some text, then partially overlay that with an image so some text is still visible, then write over that area again with "transparent" text. If you can't write text on top of text, you're going to have a weird situation where some parts of the background are being completely erased, while other areas show through - it's just going to look broken.

I think if you want to do this sort of thing, you really need to design it into the protocol from the start. And personally I don't think it's worth the overhead and will just lead to poorer performing TEs (which already struggle to handle basic sixel output).

[hackerb9]

I would hope that text with a transparent background could overlay on top of sixels in the same way that sixels with a transparent background already overlay on top of text. Why should that be when transparent text doesn't accumulate upon other text?

I don't really see how you can allow one without the other.

Makes sense, Depending upon what the "transparent background" attribute means, I think it could still be somewhat handy as a shortcut for what I already do using sixels and a font renderer. [Example of sixels writing text on top of text: tput home; cat -v sdmhome.six ; tput home; cat sdmhome.six]

This reminds me, I've been meaning to ask you if you would see any problem with terminal emulators setting their default colour palette for sixels to work more like the VT340.

In particular, it's trivial to write a sixel image that uses the current foreground and background colours on an actual VT340: you'd just use #7 and #0 without sending a definition for them. The sdmhome.six image does something like that: it uses #7 for the foreground so it will match the text color, but then omits the background not because it needs transparency but because it cuts the file size in half. [Edit: file size has an outsized impact on VT340 performance due to the serial port bottleneck.]

I know it doesn't make sense to allow the reverse, where setting sixel colours affects the text foreground and background. That means an emulator's colour palette will never work exactly like a VT340 which also remembers the last defined palette. I have seen two image files that took advantage of that feature, sending the palette in one sixel image and the image data in the next image, but it seems pretty rare and I think having access to the current fg and bg colours may be more valuable.

(On the other hand, there is a workaround for discovering the fg/bg colours and using them, but there is little recourse if the colour palette is not preserved. Hmm... I may have just talked myself out of what I was asking. [Edit: It is late at night and I wasn't thinking clearly. Discovering the fg/bg colours requires an interactive program and cannot be done in an image file, whereas the colour palette could be mechanically copied from the previous image and pasted into the next one.])

Anyhow, I'd like to hear what you think.

[jerch]

... I have seen two image files that took advantage of that feature, sending the palette in one sixel image and the image data in the next image...

Have not investigated the outputs, but I think mpv has a setting to reduce "palette definition pressure" on single frames:

--vo-sixel-threshold=<threshold> (default: -1)
    Has no effect with fixed palette. Defines the threshold to change the palette
    - as percentage of the number of colors,
    e.g. 20 will change the palette when the number of colors changed by 20%.
    It's a simple measure to reduce the number of palette changes,
    because it can be slow in some terminals (xterm). 
    The default (-1) will choose a palette on every frame and will have better quality.

[jerch]

ontopic interlude:
Thx all for the discussions and testing the scroll behavior and DECSDM, collected enough evidence to fix the implementation on my side.

[j4james]

This reminds me, I've been meaning to ask you if you would see any problem with terminal emulators setting their default colour palette for sixels to work more like the VT340.

For my own implementation, I've been trying to find a balance between supporting modern sixel usage, while still providing accurate emulation of the original DEC terminals, and I have encountered a few complications.

First off, if you want to support background colors correctly, you've got to be able to configure the palette size to match the target terminal. As you know, on the VT240 you need the fourth color defined to change the background, while on the VT340 it will be sixteenth color. So the terminal has to be able to downgrade the color range for that to work.

Then there's the issue of palette animation. For now I've dropped support for global palette animation, but I still support palette changes within the current image. If you update a color before terminating the sixel sequence, all instances of that color in the current image will be refreshed (you can see an example of this in the spinning ball demo).

I've also been considering supporting animation via a separate sixel sequence, but limited to updating the most recent image (or possible the current viewport). I'm not yet sure how workable that's going to be though.

When it comes to palette inheritance, I've tried to exactly match the original DEC semantics, so you can set the palette in one image and reuse those values in subsequent images. But for this to work correctly, you need to set the palette in the right order, namely #1, #2, #3, and not #0, #1, #2, otherwise the later images will get the colors off by one.

I think Jexer is one example of an app that gets this wrong, and I'm been meaning to submit a PR to fix that, but considering the project is now archived I'm not sure how likely that is to be accepted.

Finally there's the relationship between the image palette and the text palette. By default I treat them separately, but if the palette size is configured to 16 or 4, then I'll also update the text palette when the image palette is changed as you would expect. For now this is just the foreground and background colors, but I'm still working out the exact semantics.

So as things stand, the defaults are mostly compatible with modern apps, while still providing good legacy support if you configure the palette size appropriately. I'm not sure if that'll end up being an option in the Windows Terminal settings, but either way I'd like to be able to configure things via a standard escape sequence too. I was considering something like DECSCL, or DECTID, but I haven't decided on anything yet.

The sdmhome.six image does something like that: it uses #7 for the foreground so it will match the text color, but then omits the background not because it needs transparency but because it cuts the file size in half.

For me this kind of works by default, because the default sixel palette entries for #7 and #0 are likely to be reasonably close to the text foreground and background colors, but that's not guaranteed. If you've selected 16-color mode, though, then the palettes are somewhat shared (as describe above), so you should get an exact match.

[hackerb9]

First off, if you want to support background colors correctly, you've got to be able to configure the palette size to match the target terminal. As you know, on the VT240 you need the fourth color defined to change the background, while on the VT340 it will be sixteenth color. So the terminal has to be able to downgrade the color range for that to work.

If I recall correctly, DEC's ANSI Compliant Printing Protocol addresses a similar issue for 32 color and 4 color sixel printers and its recommendation was for modern applications to simply use the higher number as older devices were designed specifically for this issue and will take an index modulo their own palette limit. I read that to mean setting color #16 on a VT240 would actually set #4. Is that not the case?

[j4james]

I read that to mean setting color #16 on a VT240 would actually set #4. Is that not the case?

Yes and no. The VT240 is a special case I think, because they try to do smart downgrading of larger palettes. So if you set more than 4 colors, it'll try to remap the later color numbers to the closest match in the initial 4. On the VT340, though, it's a simple mod 16 mapping.

Either way, this doesn't help a modern terminal that supports 256 or more colors when it's trying to render a 4-color VT240 image or a 16-color VT340 image. The potential benefit is for older terminals when rendering newer images.

[hackerb9]

Either way, this doesn't help a modern terminal that supports 256 or more colors when it's trying to render a 4-color VT240 image or a 16-color VT340 image. The potential benefit is for older terminals when rendering newer images.

Makes sense to me.

So as things stand, the defaults are mostly compatible with modern apps, while still providing good legacy support if you configure the palette size appropriately.

Pretty much everything you've described sounds good. I'm pleased you've thought of a way for palette animation to work. (By the way, did you ever see that I added horizontal motion to your vertical Boink demo? Now we just need to put them together to get a proper bounce!)

I'm not sure if that'll end up being an option in the Windows Terminal settings, but either way I'd like to be able to configure things via a standard escape sequence too. I was considering something like DECSCL, or DECTID, but I haven't decided on anything yet.

Please do not overload DECSCL to also mean palette size changes. It does a hard reset on a terminal.

I am pretty sure DECTID would also be a bad choice. The documentation suggests that it is a superficial change, akin to changing the UserAgent of a browser. While my VT340 does not respond to that sequence, the Set-Up menu has a Primary DA Response setting akin to the option the VT525 manual says DECTID maps to. I tested it out and changing it to VT240 does not actually affect things like the palette size or DECSDM. It only changes the terminal ID as seen by the DA response.

From our past conversations, I recall you have a distaste for newfangled escape sequences. That's healthy as it encourages us to go to the extra effort to build upon the existing foundation instead of just making up new sequences. But it seems palette size was one of the sequences DEC never got around to adding. We could use an existing sequence to mean "change the palette size", but it would not build upon anything and can only cause confusion, a doppelganger that looks the same, has the same name, but acts differently.

Please consider using XTSMGRAPHICS. It makes getting and setting the palette size unambiguous and causes no problems for existing terminals.

[dankamongmen]

Please consider using XTSMGRAPHICS. It makes getting and setting the palette size unambiguous and causes no problems for existing terminals.

yes. it is also pretty widespread among terminals supporting Sixel (i was in fact using its support as my litmus for sixel support in its entirety up until a few weeks ago, as opposed to the '4' feature in DA1). it has clean semantics and also handles maximum bitmap geometry, if that's a meaningful feature in the implementation.

[j4james]

By the way, did you ever see that I added horizontal motion to your vertical Boink demo? Now we just need to put them together to get a proper bounce!

Yeah, I thought that was a brilliant idea! I was also wondering whether it was possible to make it bounce across all the edges of the screen. I was hoping you could do the vertical movement the same way you've done the horizontal, in place of my IL/DL hack (which I think you said had some glitchy effects).

Unfortunately I don't know of any modern terminals that actually support virtual window scrolling like that, never mind combining it with sixel, but it is on my TODO list. The Windows console does at least have the underlying concept of virtual windows to some extent, so we're part of the way there.

Please do not overload DECSCL to also mean palette size changes. It does a hard reset on a terminal.

I'm not sure I understand your objection. The target audience for this are users that are running legacy VT applications that require strict compatibility with older hardware. So they're going to be setting DECSCL anyway - I'm just downgrading the sixel implementation to match the rest of the system at the same time.

Please consider using XTSMGRAPHICS.

I probably wasn't clear, but the intention here isn't just to change the palette size - it's to set the sixel implementation to a specific conformance level. Palette size is one aspect of that, but there's a whole lot more to it. For example, at the VT240 level, sixel scrolling is disabled, and the pixel aspect ratio is fixed at 2:1. I'm pretty sure nobody would want me repurposing XTSMGRAPHICS to trigger that sort of thing, but I don't think it's an unreasonable outcome when someone has specifically requested level 2 conformance.

I am pretty sure DECTID would also be a bad choice.

Yeah, I'm not that keen on the idea either. I was just considering it because I was looking for a wider range of options than I could get with DECSCL - specifically a way to select VT382 compatibility. I also know some commercial terminal emulators trigger VT382 compatibility via their terminal ID setting (although they don't support DECTID). But since I haven't been able to find an official DECTID number for the VT382, it doesn't really help.

Anyway, none of this is decided yet. These were just some idea I was considering. My initial implementation will probably be hardcoded with the max palette size, and whatever options are necessary to work with modern applications. At some point, though, I do need to provide a way for users to configure the system for use with legacy apps, and I can't really add new settings in the Windows console.

[dankamongmen]

I'm not sure I understand your objection. The target audience for this are users that are running legacy VT applications that require strict compatibility with older hardware. So they're going to be setting DECSCL anyway - I'm just downgrading the sixel implementation to match the rest of the system at the same time.

btw, having gathered @j4james that this seems to be your personal audience / motivation, what are some examples of these apps? just for my edification. are they primarily in an industrial/retail context?

[j4james]

I honestly don't know actually. I have a friend who does consulting for clients that work with ancient hardware, and occasionally needs to recommend a terminal emulator to replace whatever old boxes they're getting rid of, but I don't know the specifics of what they're doing. I wouldn't really have thought of his clients as industrial or retail, but I'm not really sure. I do know there was someone asking for VT340 support in the Windows Terminal issue tracker last year, and they wanted to replace a bunch of terminals at a nuclear power station. I guess that would classify as industrial.

[hackerb9]

I can't really add new settings in the Windows console.

Well, that seems to be the problem. It didn't make sense to me that one would change the core emulator setting via escape sequence instead of just putting it in the emulator's options. On the other hand, the VT525 appears to do exactly that to switch to Wyse mode, so maybe those are the escape sequences to look at extending.

I do not think DECSCL is a good choice for selecting VT240 sixel behaviour for a number of reasons, but perhaps the most important is that the VT340 can already emulate the VT240's sixel behaviour well enough via DECSDM (sixel scrolling) and DECSTGLT (colormap). Aspect ratio is a red-herring as any application designed for the VT240 would not select an aspect ratio. The sixel documentation states that if no aspect ratio is chosen, the default is 2:1, so any properly implemented emulator would already handle that case.

I do know there was someone asking for VT340 support in the Windows Terminal issue tracker last year, and they wanted to replace a bunch of terminals at a nuclear power station. I guess that would classify as industrial.

⚛⏻🪴❗ If a nuclear power plant isn't industrial, I don't know what is.

[jerch]

If a nuclear power plant isn't industrial, I don't know what is.

They are still running their stuff on 30ys+ old hardware? I am somewhat scared. Well, at least DEC was known to build things quite long lasting. But 30ys+? Com'on...

[j4james]

It didn't make sense to me that one would change the core emulator setting via escape sequence

You mean like VT52 mode? Or Tektronix Mode? Or even say selecting the TIA conformance level? It's pretty useful being able to have a startup script for an application that automatically switches the system into an appropriate compatibility mode first.

I do not think DECSCL is a good choice for selecting VT240 sixel behaviour for a number of reasons

Can you phrase one of these reasons in the form of a bug report? In other words, imagine I've already implemented this. Now can you give me an example of an application that doesn't work because of it?

the VT340 can already emulate the VT240's sixel behaviour well enough via DECSDM (sixel scrolling) and DECSTGLT (colormap).

That's a very liberal definition of "well enough". I can come up with several VT240 examples that I'm sure will not work on a VT340. If you don't care about that level of compatibility that's fine - I'm not trying to push anyone else to take the same approach as me - I just don't understand why you're so opposed to what I'm doing. How is it harming you?

[hackerb9]

How is it harming you?

Harm? I must be miscommunicating again. My apologies. Let's skip over items like this where I sound unreasonably opposed and cooperate when my suggestions are more clearly helpful.