Lexically scoped file handles

[jpco]

(#239 got me really hopped up on the idea of "lexical scope all the things!" Forgive me :)

Es has a couple of places where it opens a file and holds its descriptor, hidden from user code, for some long-ish period of time. The most obvious example of this is the file handling for shell input, which got that whole big mailing-list discussion early on; even if the shell input is on stdin, messing around with stdin doesn't mess around with shell input.

See the following script:

#!/usr/local/bin/es

# readwow sets stdin to wowfile
# and reads and evals the first line
echo >  readwow 'exec {< wowfile}'
echo >> readwow 'eval <=%read'

echo >  wowfile 'how = good'
echo >> wowfile 'echo hello world'

. readwow       # set stdin to wowfile, eval {how = good}
echo all $how   # check that how was set
eval <=%read    # read and eval {echo hello world} from stdin
echo neat       # keep evaluating shell input

Even when run in a way that explicitly sets the shell input as stdin, the right thing happens -- setting stdin to wowfile doesn't muck up the input that the shell is reading:

; cat test.es | es
all good
hello world
neat

Another example is the prior, pre-#65, behavior for writing to history without readline. This was a performance hack, I think, and it's not too great a loss that it's gone, but it's still notable that the shell runtime had this file descriptor that the user couldn't get at modulo /proc/fd shenanigans.

I think these are evidence that hiding file handles (that is, descriptors) in es would have utility. How would you pull these examples out to the shell level -- as we did with history in #65 and I would like to do with shell input in #178 -- and keep that file-handle hiding? The immediate answer, as usual, ought to be lexical scope.

Given es already has a layer of abstraction between user fds and real fds, this doesn't seem so crazy. There's some prior art with lexical file scope being added after dynamic scope was the only default, in Perl.

There would be a lot of questions to answer. Say with the following script

#!/usr/local/bin/es

fn my-fn {
	%create 1 my-output {        # the critical line
		echo hello, world
		outer-fn
		echo goodbye, world
	}
}

fn outer-fn {
	echo whose stdout?
}

my-fn
echo got:
cat my-output

questions would be:

1. Would we want to change the existing %openfile behavior to create a lexical file scope within the block, or create a new construct for this that could let us do the more explicit let (fd = <={%write-to my-output}) {...}? I tend to prefer improving the existing shell features, rather than adding new ones, and changing the existing stuff to lexical scope should result in no change in almost all cases. (We'd need to add an escape hatch so that we could keep dynamic scope -- or maybe just make sure to keep the exec {> foo} escape hatch that exists now.)

2. How would we solve the -- to coin a term -- "upward fdarg" problem? For the %write-history example, the lexical binding would need to be on the outside of the function, so when opening the history file the file handle would need to be passed "up" to that binding. How would these handles/fds be represented when printed in a %closure? How can they be correctly inherited in the environment by subshells? For lifetime management, we'd need to do some kind of fd-reference tracking like the GC does now. Is an Fd * a type of Term * or a kind of Tree * -- or could we model it some other way?

Maybe these questions are signs that this is not the best idea. But it seems to me that there's something here; I am certainly not very happy with how quickly file descriptor use becomes dizzying as it gets even a little bit advanced.

[jpco]

This is not really an "issue", so moving it to a "discussion".

Maybe backing up from lexical file scope to what I'm really thinking about. In es as in other shells, running a single command with output directed to some file is concise and nice, no matter the command.

do-stuff > filename

But let's look at the following C function:

void doit(void) {
	int fd = open("./zoom", O_WRONLY | O_CREAT | O_TRUNC, 0666);
	dprintf(fd, "hello\n");
	other_stuff();
	dprintf(fd, "world\n");
	close(fd);
}

To recreate that in es, we need to:

fn doit {
	let (fd = <=%newfd) {
		exec {%create $fd ./zoom {}}
		unwind-protect {
			%dup 1 $fd {echo hello}
			other-stuff
			%dup 1 $fd {echo world}
		} {
			exec {%close $fd {}}
		}
	}
}

This involves:

1. Manually grabbing an fd to use, since there's no way to just open a file on a fresh fd like in C's open(3). This is better than just picking some high number and hoping for the best, though.
2. No syntactic sugar at all, so basically every actual operation is buried within the file-handling stuff.
3. Manually closing the file (incl. the unwind-protect in case of exception)
4. No actual protection from other code messing with fd other than the obscurity granted by using %newfd.

None of these seem ideal. Unfortunately I don't see a way to make the following work as a "lexically-scoped" redirection, either. There's just no real difference between the echo call from the other-stuff call that would make one have the redirection applied in its body but not the other.

fn doit {
	{
		echo hello
		other-stuff
		echo world
	} > ./zoom
}

A somewhat more tenable alternative could look something like the following.

fn doit {
	let (>[f] ./zoom) {
		echo hello >[$f=1]
		other-stuff
		echo world >[$f=1]
	}
}

It's a little new syntax (both the let and putting variables in redirections are invalid now) and I am picturing that it would be backed by, essentially, a pool of reference-counted "real" fds, potentially de-duplicated, keyed by inode and/or filename. When the last of these named/refcounted file references gets collected and the refcount reaches zero, the fd can be automatically closed.

Another side benefit of a more "robust" real-fd pool which tracks filenames where it knows them: internal error messages could include the file name in more cases. This would be particularly useful in the Input *-replacing case I am thinking of, where parse errors would use this mechanism to print the file name when possible.

[jpco]

As another example of where this is a problem, esdebug opens exec {<>[3] /dev/tty} for interacting with the terminal. But if the user program does something with fd 3 themselves, then they'll stomp over the esdebug fd and cause mayhem. esdebug is not exactly the most shining example of es scripting, but still.

I'm realizing now that my let (>[f] file) idea is essentially just the bash/zsh exec {var}>~/file concept. I think that's basically fine; using variables to refer to files is natural, and it makes sense to depend on es' fairly powerful variable scoping mechanisms. Lexical scope makes it more complicated to implement, but there's more upside, too.

[jpco]

I continue to tumble this idea in my head trying to get it shiny. Here's what I have so far.

I want to be able to open a file (with a mode specified, either in %openfile-style (r w a)^('' +) syntax or "sugary" </>/>> syntax) and refer to that open file by variable name, and then I want to be able to use dup syntax to "apply" that open file to a descriptor, like cat <[0=$f] (potentially without the $). Critically, I do not want the file to be visible to the user as an fd number, except in scopes where it has been "applied".

Internally, this should be fairly easily achievable with an array/list of structs looking something like

typedef struct {
	int refs;
	int fd;
	char *name;
} Filehandle;

When refs hits 0, fd can be closed and the struct can be freed or reused (of course, right now, we're not addressing what actually manages refs). The name field isn't strictly necessary, but could be nice for things like error messages. Some kind of tag field could be necessary for cases where the same file is open twice at different offsets (or two different files are open by the same name) and a child process which inherits the fds needs to accurately associate variables with handles.

The externalization story would be the big question, though.

Initially I pictured this as a special kind of binding, so that instead of assignment syntax like a = file we'd have slightly more complex syntax like >[a] file. However, this is very specific to file handling, it's awkwardly similar to the normal syntax in certain cases (>[a] file vs. exec {>[3] file} seems very confusing), it would require an implicit call to a hook function to avoid losing the ability to spoof file-opening, and it isn't really flexible enough to print a list like a $f b.

So, for maximum flexibility with minimal "magic", a filehandle should probably just be a kind of Term * (or Tree * embedded within Term *, whatever). But if we had, say, f = <={%filehandle r name}, then what would echo $f print? What would let (f = <={%filehandle r name}) {echo {}} look like? There are basically three options here:

1. Make fresh syntax for this new thing,
2. Fit it into existing syntax, or
3. Weasel out by just printing the filename or something outside of %closures.

Fitting an opaque filehandle into existing syntax might look something like the following dodge:

; f = <={%filehandle r name}
; echo $f
<={%filehandle r name}  # potentially including a tag

This would create no new syntax, and would only require one novel parsing behavior, which would be making closure-parsing handle

%closure(f=<={%filehandle r name}){}

So that all seems very pleasantly straightforward. However, a troubling scenario would be something like

; ./dosomething.es arg arg $f

since the script would get <={%filehandle r name} as an argument without any obvious signal to parse it. This could be just considered okay, or could be prevented by forbidding these "opaque" handles to be passed in argv to external binaries (probably not good), or in cases like this the handle could "devolve" into a file name (probably too magic, and we probably don't want to lean too hard on filenames, since they can become wrong with a simple cd), or the subprocess could try to eagerly parse this special case (bad!)

There would be fewer edge cases to worry about with fresh syntax -- but what would that be? Ideally it would be something not specific to file handles, so that it could be generalized. I am imagining something that specifies a function to fetch/generate a new handle, as well as arguments to that function. Like, I don't know, @<%file tag r name>, but preferably less ugly.

I think I need to keep tumbling this idea in my head for a bit. The core idea still seems sound and useful to me, though.