Implement Eval (dynamic evaluation of expression/code)?

[JBEJBE]

VBS, Python and other languages can dynamically call a function via "Eval". The combination of this proposed
native "Eval" and VB control arrays would open the field for TB as a truely exceptional language, mixing compilation and dynamic interpretation.

Example:
func = "res = Call AnswerToLife(42)"
Eval(func) ' TB now calls Function AnswerToLife with value 42, pulling result into res
Debug.Print res ' Hopefully true according to hitchhiker's guide ;-)

Yes, this could get complicated, but I think it would be worth the effort.

[Greedquest]

Yes this seems useful, definitely in the realm of "Solving problems is the point" from the language design principles. At the expense of "Readable code is better than terse code" as dynamic string expressions cannot be debugged at design/compile time, "Fewer ways are better than many ways" ("There should be only one obvious way") - since it allows rewriting any normal code to now use late bound stringified syntax.

For those reasons two ideas spring to mind:

• Where possible, consider the reason Eval is necessary and instead make a feature request to support that directly. E.g. I have often wanted to do Set a = Eval("New " & TypeName(someObj)) to create a new instance of a class from an unknown object. However that could be supported with a different feature request Set a = New (TypeOf someObj). So try as much as possible to improve the language to cover all use cases of Eval.
• We could limit the scope of Eval - make it so that it can not be used in place of regular code as easily. For example limit it so that it cannot declare new variables/names (that may shadow existing ones), maybe even prevent assigning values to or reading values from existing variables. Eval is surely a target for malicious code injection so limiting what it can do may be sensible, especially given tB's business focus "Solving problems is the point".

[mwolfe02]

The Microsoft Access Application object includes an Eval() method that does what you are describing, but does it via the Access database engine Expression Service (ACEES.DLL), which has some important implications (e.g., IIf gets treated as a true ternary operator rather than as a function).

[JBEJBE]

@mike: Yes, perfectly right. The major drawback of this solution seems, that Access needs to be installed on the end user's pc. If that's not the case, Access Eval() won't run. Or does a workaround exist? The proposed native Eval() would alway work, regardless of external DLL's. This is especially true, if you want to support other operating systens (Linux etc.) .

@Greedquest: Yes very good points. Eval() needs to be controlled, otherwise it's sort of Evil().

[rexxitall]

As said a pcode interpreter would do the trick. But to get this really working you will need to implement a complete interpreter too. Would be the best soloution as well as the most complex ones. On the other hand, most things you need for a interpreter is alreday there i guess. GFA basic has had this function. Which indeed make it very powerful. Its also handy while testing code. A interpreter is way more error tolerant and you could change code during runtime as in vba. Which is really cool and time saving :). As a workaround you can currently pass a com object to VBS and maybe set some var to a callback function. (add the code you need interpreted as a string). This works fine already. With TABLACUS on github this will work also for 64 bit. But also true - not on linux. So the in the endgame Wayne will have finally provide some king of vb6 interpreter as well. Which is not the most worse things for the end users but for Wayne :) And if he is clever he made it in some way like it is with the script control where yo can choose between JS and VBS. So maybe some would be happy to call pyton or ruby code as well. But as also discussed. We can not yet clone Wayne :). So let the poor guy finalize the basic things first :).

[wqweto]

https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/expression-trees/

No compiled language brings a copy of its IDE interpreter just to support Eval but I leave this to Wayne to explain what other technical difficulties there are to the proposal.

[mburns08109]

@wqweto ...why not? Imagine if not only did tB drag along a tB interpreter (when needed), but it could drag in other interpreters as well?! Imagine the scripting possibilities for tB applications if your apps could support tB(Script), JS, Lisp/Scheme, Python, et. al. I'm not necessarily suggesting Wayne to DO all these, but perhaps putting in the plumbing to support others doing them...that could be something well worth doing (even if only after v2/v3...).

What made VBx great was its available hooks into so many other things. This idea could give tB that kind of reach, too.

[WaynePhillipsEA]

In vbWatchdog, I already solved similar problems. There we have an VBx IDE addin that provides automatic line numbering (that can be exactly matched to line numbers deterministically gained from VBx p-code at runtime). One of the problem areas is that the IDE addin has to know which lines of code are 'live' and which are 'dead' due to conditional compilation.

This is how it looks:

Note that the conditional compilation constant, and the If expression in the source code was evaluated in the addin in order for it to determine which line of code was live/dead which can be seen by the assigned line numbers to the left of the code.

The vbWatchdog Addin does not do any complicated codegen... it just generates simple expression trees and evaluates everything by passing values to the OS provided Variant functions. Sure, this means everything gets treated as a Variant, so performance is not great. But it's simple, and it works well. Performance is never a big concern for Eval() anyway, given that you're parsing sourcecode from text in the first place.

It will need some refinement, but re-using the expression evaluator from the vbWatchdog VBE Addin, I believe, is the best way forward for this.

[Greedquest]

@JBEJBE pointers are another topic - there are discussions on function pointers, pointer types and dereferencing them, delegates and lambdas. See other posts in this repo (sorry on mobile so can't easily link). I think this discussion should be limited to a function that evaluates tB code supplied as a string

[GaryMiller]

@mburns08109 So long as the compiler has a smart linker, the smart linker should know if the eval() function is not required called from the program so should be able to leave the portion of uncalled code out of the final executable.

[rexxitall]

Lightweight expression solver hm. There is one out. Usually just for math expressions. But as usually said better then nothing :) In some acad versions ive got throubles with the build in eval. And its nice to not always search a pocket calculator;) so as a quickfix for some : https://github.com/rnfc/VBA/blob/master/clsMathParser.cls

[JBEJBE]

@Greedquest
Good focus: "a function that evaluates tB code supplied as a string".

@rexxitall
Nice find and an additional keeper ;-)

[rexxitall]

Well, helps me out a few times. :) Also to be to consider is "scriptbasic". Also VB.net can be brought to compile and use things at runtime. I dropped both ways adding a sdditional interpreter as a sandbox to VBA cause it was to much afford. In my case VBS or any script control was fast enough for my purposes. For scriptbasic i add a dll wrapper with freebasic. But the syntax if scriptbasic was too far from what i need :) And freebasic has just one sample on how to made a com server dll. Which i never fully understand. Th author, a russian guy seems also how to say - a bit special :D If you has to deal heavy with genetic algorithm there is something else which looks intersting https://seed7.sourceforge.net/ Seed 7 is as language which can be extended by purpose. As sample the guy provides a basic interpreter with source.

[JBEJBE]

@rexxitall:
Thank you, your ideas propelled the discussion in the right direction. GFA Basic shows the technical possibility of the proposal.
The said alternate VBS solution would be too slow for 1000s of calls. And it would expose secrets of my software, that could be hacked in the communication between VB and VBS. Best solution would be an internal VB Eval.

I expect, it will take 6-12 months for Wayne to sort out all relevant issues regarding "classic VB", so if there would an Eval() after this period, I'm fine.

@wqweto:
Bingo, this is a very good candidate area.

@wayne:
Great, your proposed solution sounds very good to me. Performance is not an issue, hoping it will be faster than a VBS call. Datatype Variant is understood, everything else would be to complicated. If the reused expression evaluator would work dynamically with an external input file or an internal array, I'm happy. When performance would be an issue, mathmatical expressions could be solved in a specialised separate faster module (examples on planet source code), so that only complicated function calls need to be handled by the expression tree. Do you have a time frame for this "unclassical" side project?

[wqweto]

Do you have a time frame for this "unclassical" side project?

Hope not. . .

[Greedquest]

You seem keen on this feature, what usecases do you have for it? The only time I've reached for VBS is for multithreading, you write the VBA to a VBS file or just use VBS to call another Excel instance. All very messy and much better solved with tB's native multithreading. Is there any non-hacky usecase for eval, other than a command line calculator? Been getting by decently without it in VBA so far... (Nb I'm all in favour of new capabilities, but strong justification will help prioritise them and also may identify better alternatives)

[rexxitall]

Imagine you has a running VBA project but you want to let the user write own code. So VBS can be used as sandbox. If the VBS fails it will not prodce any errors messages in the main VBA. You are also able to edit the vbs during runtime. The mutithreading soulution with excel is to slow :) For that case i would curently rather implement a vbs server in VB.NET or TB active exe which holds all the times a few vbs objects in stock. The other way around is to abuse Apache for it. If i have the need to make some multithreading without TB in mind i use VB.net. Its quite simple cause its easy to let VB creaate com classes on the fly for you. You can even give that a twist if those com classes calls native compiled dlls. So you just uses the multithreading mechanism of VB.NET. VB.NET itself is slow like a snail. Usually last years i spice it up with freebasic. But that is all rather a hack then a nice handy thing. But at least it is possible until you do not have a better tool on the hands. :)

[GaryMiller]

The most common usage I have seen is in plotting programs where you let the user enter an algebraic or trigonometric function to generate a 2D or 3D graph. Also Calculus programs that calculate derivatives or due integration need the capability to enter functions.

But I use it heavily in AI to add logic to responses based upon the knowledge the AI has gained in the conversation.

To allow users to add their own patterns and responses with logic you don't want to release the full source code to them and have them change the code possibly breaking it. You want to have them enter the logic in English and generate the VB code .

I will use the code templates in conjunction with a graph database to store knowledge that the AGI can reason with and generate custom VB responses.

[rexxitall]

Just a note: I use the VBS soloution often in my projects since years. Also the 64 bit ones via TABLACUS. It is surprising fast. In one area i use even vbs as CGI on Apache (on localhost !). Sure you will not win every race with it. But it seems MS has speed up vbs during the years a lot - or intel :D A small trick is also to put the time consuming routines in a (COM) dll. So if you just use VBS or the VBS script object as "glue" you get already a lot out of it. Which also adress the problem to keep some application internals secret. Only problem is to debug vbs code. There are a few VBS editors out but as closed source. But if you has hands on VBA you can test it quite well if you put the new VBS routines temporary in a module. Best cheapest other soloution is to buy vbsedit. VS is also possible but this will always load a new instance of VS. Not really handy.

[JBEJBE]

@rexxitall:
Very useful tips in your answers. I'll keep your ideas as a promising alternative to a native TB Eval().

@Greedquest:
Yes I'm keen on a native Eval(), I admit that. I've built a big business software solution in Python over the years that is centered around Python Eval() as a dynamic function interpreter. There are parts with genetic algorithms, that can only be tackled dynamically.
Nearly all code could be transormed to VB/TB, except the dynamic function evaluation. If Python could be compiled, it would easily be THE winning programming language, but it fails in this regard. One can't sell closed software with Python. VB/TB has top quality in the areas of user interfaces, database access, business value, (your personal favourites here...), and beats all other languages. Thanks Microsoft, for throwing in tons of money, people and knowhow into classic VB, it's a world class software. A native Eval() would be the icing on the cake. I think, we're very good on track with Wayne's proposed recyled expression evaluator from the vbWatchdog VBE Addin.

@wqweto:
I agree with you, classic VB has top priority. But better discussing dynamic Eval() now, than maybe in a year, when the internal architecture of TB is set and it would be too late/too costly for a change in this direction.

[mansellan]

OK, I'm lost.

I'm a .Net developer, so I'll try to cope from there. .Net has had assembly metadata since v1. It supported reflection, by storing the shape of all code alongside the bytecode. Anything could be instantiated or called by accessing that metadata. You could even emit assemblies dynamically, by directly writing IL.

Recently, the Roslyn compiler has also made it possible to compile source code at runtime. IIUC, you have to incur significant cost to do so. Most recently, source generators allow you to emit code at compile-time (with some restrictions), and avoid the runtime cost.

TwinBasic compiles to native. It doesn't have metadata to support reflection. And it doesn't have the option to do on-the-fly compilation. Feel like I'm missing something here.

[Greedquest]

I should say IDispatch (which all VBA classes and many standard external objects implement) does give a basic level of reflection. The precursor to modern reflection. You can, given an object instance, obtain a list of all its members, call them, obtain more detailed information about parameters and types from the ITypeInfo and containing ITypeLib.

These features have no good native support since the interfaces are restricted in VBA, however in tB it is already possible to write a library to use these interfaces and provide ReadOnly reflection of many things, and calling members is also possible.

As VBA is interpreted, it's not difficult to dynamically write source code and execute it (although it does switch off the debugger). tB it'll be much more of a challenge as it is compiled. Instantiating objects by reflection is as far as I'm aware not possible universally as it requires the object to be registered publicly in a COM server (stretching my knowledge) so it can be instantiated through CreateObject - this registration is optional and VBA classes don't do it.

[wqweto]

I think augmenting TB's CreateObject to allow instantiating private classes is possible and cheap to implement.

In VB6 I always wondered why they couldn't make CreateObject("Project1.Class1") succeed when Project1 is an Std-EXE.

Checking and instantiating private classes before calling CoCreateInstance can be data-driven implemented (a single array) and is such a low-hanging fruit for enhancing the built-in CreateObject function to become an intrinsic.

[Greedquest]

@wqweto That would be useful and make object serialisation closer to possible. Would need an overhaul for tB constructor args

[JBEJBE]

At first, thanks to all participants for their high level input.

There's a reason, why dynamic languages like Python are immensely popular: they solve today's programming challenges like AI in a fitting dynamic manner.

VB is the perfect product for "classic" programming themes like business software with a certain restricted market appeal. I think TB, as VBs
successor, could be much more successfull in the future market, if it would give us additional ways to handle today's dynamically centered problems. Of course, after becoming "classic" ;-)

I'd like to clarify, what Eval() means in my proposal. If you have a source code line like

Call Func1(Parm1)

we define that Func1 will start at address 1000, Parm1 will be at address 2000. Compiler/Linker know both adresses and are satisfied.

I actually need a way in TB to formulate something like

Call *Adress(*Parm1)

meaning, I want to dynamically call known functions adresses with known parm adresses (classical pointers), but in a more user friendly way like defined above. I would be satisfied with an Access Eval(), only native in TB.

No need to generate whole new programs, which would have to be compiled on the fly. That's unsolvable, endless recursive generation could happen (generated programs generate new programs and so on).

Maybe Wayne could give a a hint, what his proposed solution would be be capable of?

[mburns08109]

Hmm... perhaps what we'd really need here, then are multiple EVAL() gateways.
JSEval()
VBSEval()
JSONEval()
LispEval()
PythonEval() (to name a quick few candidates)
...each with runtime/plug-in "extenders" for dynamic/interpreted execution.

An architecture like that would bring in the flexibility for those future AI-driven needs without burdening tB native-compiled .exe projects with the needed runtime support systems as a whole. They only get "hung on" only when needed.

So think, less of "an Eval() function" and more of designing an add-on interpretative ecosystem to work in harmony with tB.

[JBEJBE]

@mburns08109:
An interpretative ecosystem sounds like a huge project, whereas the initial proposal is a lot more lightweight (please see comments above). I think you're right in the sense, that reaching future AI capabilities will need a lot more "batteries" (tensorflow etc.) as Python calls these extensions/packages. That's community-driven, even in Python.

One could integrate existing C/C++-AI-Libraries into TB for a start. I would not expect the core language of TB to bring AI-Packages (Twinpacks etc.) to the table. Although I wouldn't say "no", if such a dream would become true ... ;-)

[mburns08109]

We could start with something like this: https://gist.github.com/pmachapman/661f0fff9814231fde48
and expand/retool it into a tb-equivalent interpreter for eval()

[JBEJBE]

Very good alternative starting point in addition to Wayne's proposal.