Fix textLinesMutator when dealing with insertions/deletions at the end of pad

[webzwo0i]

Will clean this up tomorrow (fixing eslint etc). This should fix #5214 #5213 and replace #5215 and #2911

Also adds some comments and additional coverage.

[webzwo0i]

As a side note: The commits that fixed #2802 should be re-evaluated.

[webzwo0i]

This should be ready now.

[rhansen]

• The "chore Changeset.js: use template strings" commit missed one: Line 1383 (in outputMutOp()).
• The "easysync tests: move to separate files" commit is difficult to review, mostly because stuff was reordered. Please split this commit into two: One that splits easysync.js into multiple files but preserves relative order, and one that reorders stuff.
• The "easysync tests: move to separate files" commit introduces a //TODO: why isn't this |2+2*x|1+4 ? comment to src/tests/frontend/specs/easysync-assembler.js. The comment doesn't belong in that commit.
• Please keep the first paragraph of each commit message short—ideally less than 50 chars but definitely less than 70. If you need to say more, put it in a separate paragraph (blank line between the paragraphs).
• Please add the regression tests in the commit that fixes the bug.

[webzwo0i]

Thanks for the input @rhansen! Will clean up.

Regarding putting the test into the same commit as the fix, I thought we also put the test in a separate commit (before all other commits) as stated somewhere in CONTRIBUTING.md. But in this case, as it's fixing multiple bugs in one PR, I'll definitely include them into the correct commit.

[rhansen]

Rebased onto latest develop.

[rhansen]

Regarding putting the test into the same commit as the fix, I thought we also put the test in a separate commit (before all other commits) as stated somewhere in CONTRIBUTING.md.

It's OK to add an xit() test before the commit that fixes the bug, then change the xit() to it() in the commit that fixes the bug. I generally don't like that approach for a couple of reasons:

• It disconnects the addition of the new test from the reason the new test was added (as a regression test for a bugfix). This makes it harder for reviewers to determine that the new test is actually a proper regression test for the bugfix.
• Mocha doesn't attempt xit() tests. Other test frameworks support the concept of "expected to fail" tests that are still run but don't result in overall failure unless they actually pass (in which case the "expected to fail" should be converted to the usual "expected to pass").

[rhansen]

@webzwo0i I rebased onto latest develop and made some changes to address my feedback. The "easysync tests: move to separate files" commit was split into 5 different commits to make it easier to review. I think the only thing left is to add the regression tests to the commits that fix the associated bugs.

[webzwo0i]

Thanks! I added the tests to the appropriate commits, added poolOrArray to easysync-assembler and also refined the commit messages of the last three commits.

Rebased onto latest develop

[rhansen]

This line makes the following assumptions:

• this._curSplice.length > 2
• this._curLine === this._curSplice[0] + (this._curSplice.length - 3) (in other words, the current line is the last entry in this._curSplice)

It's not immediately obvious to me that either assumption is guaranteed to be true. Examples:

• If this._curLine < this._curSplice[0] then this._isCurLineInSplice() will return true even if this._curSplice.length === 2.
• If this._curLine === this._curSplice[0] and this._curSplice.length === 4, then the first removed line should be this._curSplice[this._curSplice.length - 2].

Maybe those assumptions are guaranteed to be true due to (undocumented?) invariants. If so, it would be good to assert them here.

[rhansen]

It would be good to assert that this._curCol === 0.

[rhansen]

What if this._curCol is greater than or equal to the current line's length? In that case, shouldn't we set this._curCol to 0 and increment this._curLine before this check? Or is that case not possible due to invariants?

If invariants make that case impossible, I think it would be worthwhile to add an assert statement.

[webzwo0i]

skip:

• if L present, curCol is set to 0 in all branches (in skipLines)
• if L not present, curCol will be +=N (number of skipped chars) [0]

insert:

• if L present, curCol is set to 0 after applying L, because of the "multiline end with newline" constraint
• if L is present, but curLine was not in splice, curCol isn't touched
• if L is not present and we are processing the last line of lines array, curCol is increased by N chars [1]
• if L is not present and we are not processing the last line, curCol is increased by N chars [2]

remove:

• no changes to curCol

In cases [1] and [2] curCol calculation is safe, as a text string is given to the respective function (the term that's added to curCol is derived from from text).

Also after calling close or skipping over multiple L that don't have attributes, the splice is not active anymore and it's possible that curCol is > 0. So there are four possible "start" states:

• splice not active, curCol is 0 [X1]
• splice not active, curCol > 0 [X2]
• splice active, curCol is 0 [X3]
• splice active, curCol > 0 [X4]

So for all 4 "start" states, I'll add coverage for [0] exceeding the number of chars in the line (ie a text of "test\n" and skipping over 6 chars so we end in the middle of the next line), followed by a call to removeLines.

If we want to assert I think it's better to do it where curCol is changed.

[rhansen]

What about line markers? If the first character is a line marker and this._curCol === 1, shouldn't it be treated the same as if this._curCol === 0? Or is that not possible due to invariants? Unfortunately, I don't think it's possible to add an assert here because TextLinesMutator doesn't have access to the attributes.

[webzwo0i]

Maybe I don't understand, but the ops that are applied to lines shouldn't care about lineMarker's special meaning in the editor. ie lineMarker is special in the editor, because it's not rendered and it changes the "meaning" of the whole line, but when it comes to Changeset.js whatever attributes apply to the lineMarker are only applied to this single, "real" character. So we should not treat this._curCol === 1 as this._curCol === 0

[rhansen]

If remaining is the empty string, then this._curSplice[sline] will not end with a newline. Don't all entries in this._lines need to end with a newline character?

diff --git a/src/static/js/Changeset.js b/src/static/js/Changeset.js
index 5e0bc34fc..defc08852 100644
--- a/src/static/js/Changeset.js
+++ b/src/static/js/Changeset.js
@@ -918,11 +918,23 @@ class TextLinesMutator {
         removed = nextKLinesText(L - 1);
         this._curSplice[1] += L - 1;
         const sline = this._curSplice.length - 1;
-        removed = this._curSplice[sline].substring(this._curCol) + removed;
-        // Is there a line left?
-        const remaining = this._linesGet(this._curSplice[0] + this._curSplice[1]) || '';
-        this._curSplice[sline] = this._curSplice[sline].substring(0, this._curCol) + remaining;
-        this._curSplice[1] += remaining ? 1 : 0;
+        removed = this._curSplice[sline].slice(this._curCol) + removed;
+        this._curSplice[sline] = this._curSplice[sline].slice(0, this._curCol);
+        // The current line's newline was deleted, which means that the line after the splice (if
+        // there is one) should be joined with what remains of the current line.
+        const n = this._curSplice[0] + this._curSplice[1];
+        if (n < this._linesLength()) {
+          // There is a line that can be joined with the current line.
+          // TODO: What if the joined line starts with a line marker?
+          // TODO: What if the joined line has line attributes that differ from the current line's
+          // line attributes?
+          this._curSplice[sline] += this._linesGet(n);
+          this._curSplice[1] += 1;
+        } else {
+          // There is no line that can be joined with the current line. The current line's newline
+          // was removed, but all lines must end with a newline so add it back.
+          this._curSplice[sline] += '\n';
+        }
       }
     } else {
       // Nothing that is removed is in splice. Implies curCol === 0.

[rhansen]

I don't think this is right. If my understanding is correct, each entry in this._lines represents a line of the document and must end with a newline character. By pushing to this._curSplice, this code creates a new line at the end of the document, one that line doesn't end with a newline. This code should instead insert the text after column this._curCol (which should be before the line's newline) in line this._curLine.

I think the actual bug is inside this._putCurLineInSplice(): That method should not attempt to read past the end of this._lines or delete lines that don't exist in this._lines. Something like this (untested):

diff --git a/src/static/js/Changeset.js b/src/static/js/Changeset.js
index 5e0bc34fc..930a5e187 100644
--- a/src/static/js/Changeset.js
+++ b/src/static/js/Changeset.js
@@ -823,11 +823,18 @@ class TextLinesMutator {
    * @returns {number} the index of the added line in curSplice
    */
   _putCurLineInSplice() {
-    if (!this._isCurLineInSplice()) {
-      this._curSplice.push(this._linesGet(this._curSplice[0] + this._curSplice[1]));
-      this._curSplice[1]++;
+    assert(this._inSplice, 'not in splice');
+    assert(this._curLine >= this._curSplice[0], 'curLine is before splice start');
+    while (this._curSplice[0] + this._curSplice.length - 2 < this._curLine + 1) {
+      const n = this._curSplice[0] + this._curSplice[1];
+      if (n < this._linesLength()) {
+        this._curSplice.push(this._linesGet(n));
+        this._curSplice[1]++;
+      } else {
+        // No more lines available to add to the splice. Create a new line from scratch.
+        this._curSplice.push('\n');
+      }
     }
-    // TODO should be the same as this._curSplice.length - 1
     return 2 + this._curLine - this._curSplice[0];
   }
 

[webzwo0i]

Thanks for pointing to _putCurLineInSplice. Will take a look.

Let's make sure newlines are present at all times

[rhansen]

What should happen if an op removes the last line's newline but that op is immediately followed by another op that restores the newline? For example:

const text = 'x\n';
const lines = [text];
const cs = '=1|1-1|1+1$\n'; // Keep a char, then remove a newline, then add a newline.
Changeset.mutateTextLines(cs, lines);
assert.equal(lines.join(''), 'x\n');
assert.equal(Changeset.applyToText(cs, text), 'x\n');

Similarly, what if an op adds text to the end of the document without a trailing newline, then the next op adds the newline:

const text = 'x\n';
const lines = [text];
const cs = '|1=2+1|1+1$y\n'; // Keep existing text, add a char, then add a newline.
Changeset.mutateTextLines(cs, lines);
assert.equal(lines.join(''), 'x\ny\n');
assert.equal(Changeset.applyToText(cs, text), 'x\ny\n');

If we force newlines to always be present in each entry in this._lines then we're going to end up inserting extra newlines when we don't want them. I think the way to fix this is to strip the newlines from this._lines and add them back before returning.

[rhansen]

If remaining === '', then that means that this._curCol points just past the line's newline, correct? If so, I think the better fix would be to set this._curCol to 0 and increment this._curLine before doing anything else.

Also, this._curCol pointing just past the newline feels like a violation of an invariant. Maybe instead of checking if remaining !== '', it should assert that and we should fix whatever logic is causing this._curCol to point past the newline.

[webzwo0i]

There is not necessarly a newline at every line. Etherpad uses it with lines that always end with newlines (e.g. rep.lines), except for mutateAttributionLines, but this has other logic.
Also take a look at the test case, this is valid imo.
Update: I do think now, that it should end with a newline now. So when it comes to "valid" ignore the newline for now.

[webzwo0i]

I do understand what you wrote in the first paragraph, though. If there is a newline, we would never go "beyond" that. Let me test

[webzwo0i]

OK, I think I ensure that newlines are everywhere first. It's possible that I didn't saw missing newlines because it's dealing with edits at the end of the pad.

[rhansen]

Any update @webzwo0i?

[webzwo0i]

Will have time in the next few days. I'll probably tackle curCol/curLine behavior first, including asserting the invariants. For newlines behavior I need to take a look at mutateAttributionLines

[rhansen]

I rebased onto latest develop and pushed a couple of fixup commits.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[LeaChemoul]

Any update on thoses fixup commits ?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[JohnMcLear]

Bump @webzwo0i -- any progress?

[JohnMcLear]

@dependabot rebase

[SamTV12345]

@webzwo0i Do you have some time to fix this pr?