Reduce memory allocations

[MilesCranmer]

I think a lot of the remaining performance is due to garbage collection. This is a bit hard to benchmark in a "microbenchmarks" style approach since a different thread is handling garbage collection. So in this issue it would be useful to record what the biggest allocation source is, and try to reduce it. My guess is that it's from eval_tree_array - in which case, we should try to preallocate memory for it.