Lexically scoped file handles

[jpco]

(#239 got me really hopped up on the idea of "lexical scope all the things!" Forgive me :)

Es has a couple of places where it opens a file and holds its descriptor, hidden from user code, for some long-ish period of time. The most obvious example of this is the file handling for shell input, which got that whole big mailing-list discussion early on; even if the shell input is on stdin, messing around with stdin doesn't mess around with shell input.

See the following script:

#!/usr/local/bin/es

# readwow sets stdin to wowfile
# and reads and evals the first line
echo >  readwow 'exec {< wowfile}'
echo >> readwow 'eval <=%read'

echo >  wowfile 'how = good'
echo >> wowfile 'echo hello world'

. readwow       # set stdin to wowfile, eval {how = good}
echo all $how   # check that how was set
eval <=%read    # read and eval {echo hello world} from stdin
echo neat       # keep evaluating shell input

Even when run in a way that explicitly sets the shell input as stdin, the right thing happens -- setting stdin to wowfile doesn't muck up the input that the shell is reading:

; cat test.es | es
all good
hello world
neat

Another example is the prior, pre-#65, behavior for writing to history without readline. This was a performance hack, I think, and it's not too great a loss that it's gone, but it's still notable that the shell runtime had this file descriptor that the user couldn't get at modulo /proc/fd shenanigans.

I think these are evidence that hiding file handles (that is, descriptors) in es would have utility. How would you pull these examples out to the shell level -- as we did with history in #65 and I would like to do with shell input in #178 -- and keep that file-handle hiding? The immediate answer, as usual, ought to be lexical scope.

Given es already has a layer of abstraction between user fds and real fds, this doesn't seem so crazy. There's some prior art with lexical file scope being added after dynamic scope was the only default, in Perl.

There would be a lot of questions to answer. Say with the following script

#!/usr/local/bin/es

fn my-fn {
	%create 1 my-output {        # the critical line
		echo hello, world
		outer-fn
		echo goodbye, world
	}
}

fn outer-fn {
	echo whose stdout?
}

my-fn
echo got:
cat my-output

questions would be:

1. Would we want to change the existing %openfile behavior to create a lexical file scope within the block, or create a new construct for this that could let us do the more explicit let (fd = <={%write-to my-output}) {...}? I tend to prefer improving the existing shell features, rather than adding new ones, and changing the existing stuff to lexical scope should result in no change in almost all cases. (We'd need to add an escape hatch so that we could keep dynamic scope -- or maybe just make sure to keep the exec {> foo} escape hatch that exists now.)

2. How would we solve the -- to coin a term -- "upward fdarg" problem? For the %write-history example, the lexical binding would need to be on the outside of the function, so when opening the history file the file handle would need to be passed "up" to that binding. How would these handles/fds be represented when printed in a %closure? How can they be correctly inherited in the environment by subshells? For lifetime management, we'd need to do some kind of fd-reference tracking like the GC does now. Is an Fd * a type of Term * or a kind of Tree * -- or could we model it some other way?

Maybe these questions are signs that this is not the best idea. But it seems to me that there's something here; I am certainly not very happy with how quickly file descriptor use becomes dizzying as it gets even a little bit advanced.