Except for numbers and upper/lower case ASCII letters, I actually haven't implemented control code escapes the way bash implements them

$ echo -n $'ф' | xxd
00000000: d184                                     ..
$ echo -n $'\cф' | xxd
00000000: 1184                                     ..
$ echo -n $'🟥' | xxd
00000000: f09f 9fa5                                ....
$ echo -n $'\c🟥' | xxd
00000000: 109f 9fa5                                ....
$ echo -n $'^M' | xxd
00000000: 0d                                       .
$ echo -n $'\c^M' | xxd
00000000: 0d                                       .
$ echo -n $'1' | xxd
00000000: 31                                       1
$ echo -n $'\c1' | xxd
00000000: 11                                       .

$ echo -n ' ' | xxd -b
00000000: 00100000                                                
$ echo -n $'\c ' | xxd -b  # null
$ echo -n '!' | xxd -b
00000000: 00100001                                               !
$ echo -n $'\c!' | xxd -b
00000000: 00000001                                               .
$ echo -n '/' | xxd -b
00000000: 00101111                                               /
$ echo -n $'\c/' | xxd -b
00000000: 00001111                                               .
$ echo -n '0' | xxd -b
00000000: 00110000                                               0
$ echo -n $'\c0' | xxd -b
00000000: 00010000                                               .
$ echo -n '?' | xxd -b
00000000: 00111111                                               ?
$ echo -n $'\c?' | xxd -b
00000000: 01111111                                               .
$ echo -n '@' | xxd -b
00000000: 01000000                                               @
$ echo -n $'\c@' | xxd -b  # null value
$ echo -n 'A' | xxd -b
00000000: 01000001                                               A
$ echo -n $'\cA' | xxd -b
00000000: 00000001                                               .
$ echo -n 'O' | xxd -b
00000000: 01001111                                               O
$ echo -n $'\cO' | xxd -b
00000000: 00001111                                               .
$ echo -n 'P' | xxd -b
00000000: 01010000                                               P
$ echo -n $'\cP' | xxd -b
00000000: 00010000                                               .
$ echo -n '_' | xxd -b
00000000: 01011111                                               _
$ echo -n $'\c_' | xxd -b
00000000: 00011111                                               .
$ echo -n '`' | xxd -b
00000000: 01100000                                               `
$ echo -n $'\c`' | xxd -b
$ echo -n 'a' | xxd -b
00000000: 01100001                                               a
$ echo -n $'\ca' | xxd -b
00000000: 00000001                                               .
$ echo -n 'o' | xxd -b
00000000: 01101111                                               o
$ echo -n $'\co' | xxd -b
00000000: 00001111                                               .
$ echo -n 'p' | xxd -b
00000000: 01110000                                               p
$ echo -n $'\cp' | xxd -b
00000000: 00010000                                               .
$ echo -n '~' | xxd -b
00000000: 01111110                                               ~
$ echo -n $'\c~' | xxd -b
00000000: 00011110                                               .
$ echo -n ' ' | xxd -b
00000000: 00100000                                                
$ echo -n $'\c ' | xxd -b  # null char
$ echo -n '!' | xxd -b
00000000: 00100001                                               !
$ echo -n $'\c!' | xxd -b
00000000: 00000001                                               .
$ echo -n 罿 | xxd -b
00000000: 11100111 10111101 10111111                             ...
$ echo -n $'\c罿' | xxd -b
00000000: 00000111 10111101 10111111                             ...

If someone finds it in the bash source code and tells me what it's doing, I can implement it correctly.

Also, ANSI-C quoted strings are a bash thing, not a sh thing, we may want to have a flag to let users toggle this behavior.

There will be a mismatch in parsing because in bash you can go past the last code point 0x10FFFF, but you can't do that in node (or Python):

$ echo -n $'\U10FFFF'  | xxd
00000000: f48f bfbf                                ....
$ echo -n $'\U110000'  | xxd
00000000: f490 8080                                ....
$ node
Welcome to Node.js v14.14.0.
Type ".help" for more information.
> String.fromCodePoint(0x10FFFF)
'�'
> String.fromCodePoint(0x110000)
Uncaught RangeError: Invalid code point 1114112
    at Function.fromCodePoint (<anonymous>)
> 
$ python3
Python 3.9.0+ (default, Oct 19 2020, 09:51:18) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> "\U0010FFFF"
'\U0010ffff'
>>> "\U00110000"
  File "<stdin>", line 1
    "\U00110000"
                ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 0-9: illegal Unicode character

Control characters are proving to be a real pain. It's complicated by the fact that my bash uses UTF-8 (and other encodings are supported) but JS uses UTF-16. I'm thinking to only do the control characters that make sense and throwing an error for the rest.

I tried reverse engineering what bash does with \c codes by adding this code in test/yargs-parser.cjs:

    function toBinaryString (buf) {
      let result = []
      for (let b of buf) {
        result.push(b.toString(2).padStart(8, '0'))
      }
      return result.join(' ')
    }

    async function check (ansiString, encoding = 'utf8') {
      const result = parser(['--foo', ansiString]).foo + '' // convert integers back to str
      const resultBuffer = Buffer.from(result, encoding)

      const { stdout } = await exec(
        'printf "%s" ' + ansiString,
        { shell: '/bin/bash', encoding: 'buffer' }
      )
      if (!resultBuffer.equals(stdout)) {
        const binaryResult = toBinaryString(resultBuffer);
        const binaryStdout = stdout.length ? toBinaryString(stdout) : '00000000 (null)';
        if (binaryResult !== binaryStdout) {
          console.log(JSON.stringify(ansiString), ansiString[4], result.length, resultBuffer, resultBuffer.length, stdout, stdout.length)
          console.log(toBinaryString(Buffer.from(ansiString[4], 'utf8')))
          console.log(binaryStdout)
          // con sole.log(binaryResult)
          console.log()
        }
        if (stdout.length) {
          // throw 'aoeu'
        }
      } else {
        // console.log(JSON.stringify(ansiString), 'works correctly')
      }
      // resultBuffer.should.deep.equal(stdout)
    }

    it('parses all control codes in ANSI-C quoted strings like bash', async () => {
      for (let i = 1; i <= 0x10FFFF; i++) {
        let chr =  String.fromCodePoint(i);
        if (chr === "'" || chr === "\\") {  // TODO: escape
          continue
        }
        await check("$'\\c" + chr + "'")
        // await check("$'\\c" + chr + " '")
      }
    // }).timeout(5 * 60 * 60 * 1000)
    }).timeout(60 * 1000)

@bcoe could we get this merged into upstream?

@jonluca I still haven't finished control codes, at a minimum it needs to raise an error for the control codes that aren't handled.

If you'd like, feel free to copy paste parts of this code to curlconverter. Take a look at the source of where Chrome generates the curl command, you don't need to handle everything (like control codes) https://github.com/ChromeDevTools/devtools-frontend/blob/90e5470c94875d4417e5d31e2ab84f9b0682f33c/front_end/network/NetworkLogView.js#L2297-L2326

@verhovsky @jonluca I am supportive of this feature, but since it's breaking and fairly specialized, I'd like for it to be behind the configuration option --bash-ansi-strings, then folks who want the behavior for their library just need to turn the feature on in their package.json, by adding it to a yargs stanza.

I named it --parse-bash-ansi-c-strings instead of --bash-ansi-strings. It's more verbose but it's more correct, there's no such thing as "ANSI strings". We could name it --bash-c-strings or --bash-c-quoted-strings. The point is that they are bash-specific string syntax that uses C's quoting rules.

@verhovsky I'm getting back to open source after a break earlier this year. Could I bother you for a rebase 👏

Working on driving yargs-parser down to zero open PRs.

@verhovsky this feature looks quite solid to me, except for a couple a broken test in optimize, is this ready to move out of draft?

Thank you for the contribution 👍

@verhovsky friendly nudge, need any help getting this over finish line?

@bcoe sorry for the inactivity. I realized I put this feature in the wrong place. It needs to go in the tokenizer and I think I have to improve the tokenizer to parse $-prefixed strings first.

yargs has 2 modes of operation, either someone (or a script) calls a JS script through bash, which means bash takes the user input command string and parses it (this is the step that parses ANSI-C quoted strings), splitting it into words which then get passed to Node and it exposes that as an array of strings which then get passed to yargs. The other mode is when yargs gets passed a string, presumably this string is a raw bash command that needs to be parsed using bash's rules to build the array of arguments first.

I added my code to work after the first step (i.e. to operate on the array), which is not the right place for this to go. It needs to happen in lib/tokenize-arg-string.ts but I noticed the bash parsing there is quite rudimentary, I thought it would be good to do real parsing. So I marked the PR as draft and started looking into how to parse bash in JavaScript. I had these ideas:

1. bash's parsing is implemented with yacc, here https://git.savannah.gnu.org/cgit/bash.git/tree/parse.y . I thought we could run it with a JS yacc implementation (i.e. jison), but if I understand it correctly, the yacc grammar is used to generate a C program, so you'd still need to rewrite that stuff in JavaScript
2. use someone else's JS bash parser. I only see js-shell-parse and bash-parser, both are kinda unmaintained and neither seem to handle ANSI-C quoted strings. What else do they not handle?
3. convert the C code that does shell parsing in a simpler shell such as dash https://git.kernel.org/pub/scm/utils/dash/dash.git/tree/src/parser.c
4. compile parsing logic from either bash or dash to Wasm
5. build on the shell parsing from curl-to-go, which seems a bit better

(Edit: I ended up using tree-sitter-bash instead of yargs for parsing Bash and implementing option parsing myself)

You already had basically this conversation in #302

@verhovsky thanks for keeping me in the loop 👍

One thought, perhaps we could add an approach for providing an extension to the tokenizer, so that you could inject a parser like tree-sitter-bash?

I'm going to go ahead and close this, since we've stalled. But please feel free to open the conversation back up at any point.