Introduction

Asynchronous Reliable Extensible Sleek RPC Server for Guile. First of all it’s a library providing a lot of tools for evaluating and exploring the code, from simple reflection API to interruptible asyncronous evaluation. In addition to that it provides an async RPC server implementation based on nREPL protocol and can be used for programmable interactions with a running guile processes, for implementing REPLs, IDEs, test runners or other tools. It also can be potentially used as a backend or library for LSP server.

WARNING: It’s in early stage of development, the API is a subject to change.

guile-ares-rs was previously known as guile-nrepl (because it started as just nREPL implementation in Guile Scheme). It was renamed to avoid any confusion with network/socket REPL, guile-ares-rs is not a usual REPL, it’s actually not a REPL at all, it’s an RPC Server, however it’s possible to implement REPL-like experience with it (and respective client).

It’s intended for use with Arei IDE, but you can use it with other generic nREPL clients.

Usage

The are multiple ways to utilize Ares RS, but most straightforward is to spawn a standalone server and connect to it from a client (Emacs Arei for example):

guix shell guile-next guile-ares-rs -- \
guile -c '((@ (ares server) run-nrepl-server))'

Don’t forget to add your project source code and other dependencies to load path.

Development

It depends, on fibers, and on guile-next (custom textual ports), tests are executed with gider.

Send patches to rde-devel mailing list in format: [PATCH guile-ares-rs 0/3] SUBJECT-HERE.

Send feedback to rde-discuss.

First Release Roadmap

Arei and Ares 1.0 milestone.

Open Questions

• How to work with fibers in nrepl? Maybe C-2 C-c C-e to entered to the clonned session with fibers spawned.
• Bootstrap from guile network repl? (Send guile-ares-rs implementation to remote network repl server to make it nrepl).
• Multiple guile languages support.
• How to bypass continuation barrier in evaluation thread, when using previously stored continuation.
• Translation level for shrothanded nrepl base operations?
• Do we need to support meta-commands? (Probably not, it just functions, which can be exported to user or repl module scope).

Other REPLs Quirks

One of the reasons this project started is unfortunate missbehave of current REPL soultions in different situation and here is a list of some of them:

• Text based REPL over socket is not interruptible.
• call/cc works when pasted into prompt of the repl, but fails when evaling from buffer with C-x C-e. (when saving continuation to symbol with set!)
• It’s not possible to see stdout until evaluation completed.
• Gider tests hangs repl on errors.
• CAPF function is injected, even if there is no connection to REPL and throws errors preventing other CAPFs to work.
• There is no way to provide stdin input to running expression.
• Go to definition doesn’t work for with-input-from-port.
• After using ,i in REPL, go to definition hangs up.
• Go to definition opens non-existing file.
• If expression being evaluated from buffer requires input from stdin it will be frozen.
• Rebinds M-. instead of using xref.
• https://spritely.institute/files/docs/guile-goblins/0.11.0/TEMPORARY-Live-hacking-_0060_0060fix_0027_0027.html
• https://www.reddit.com/r/scheme/comments/qyqxcd/geiser_sucks_is_there_a_way_to_make_it_suck_less/
• Comment by Olical, Conjure author on guile-ares-rs integration

Links

NREPL

Servers

https://github.com/nrepl/nrepl

default clojure implementation.

https://nrepl.org/nrepl/1.0/building_servers.html

some tips.

https://gitlab.com/technomancy/ogion/-/blob/master/main.rkt

very simple Racket nREPL.

https://gitlab.com/technomancy/jeejah

lua nREPL server.

https://github.com/babashka/babashka.nrepl

probably most advanced 3rd party nREPL server implementation, doesn’t support interrupts.

https://docs.cider.mx/cider-nrepl/nrepl-api/ops.html#info

extensions of nREPL for CIDER.

Debugging Tool

• http://www.flow-storm.org/
• https://github.com/clojure-emacs/sayid
• https://github.com/philoskim/debux
• https://github.com/jpmonettas/flow-storm-debugger
• https://docs.cider.mx/cider/debugging/debugger.html
• https://www.gnu.org/software/guile/manual/html_node/Interactive-Debugging.html
• https://www.gnu.org/software/guile/manual/html_node/Debugging.html

Tools

• https://github.com/lambdaisland/nrepl-proxy

Clients

• https://github.com/sanel/monroe/
• https://github.com/Sasanidas/Rail
• Potentially Conjure, when nREPL for guile will be implemented in it.

Bencode

• https://github.com/flatwhatson/grump/blob/master/grump-lib/bencode.scm
• https://github.com/caolan/chicken-bencode
• http://www.bittorrent.org/beps/bep_0003.html#bencoding
• https://wiki.theory.org/BitTorrentSpecification#Bencoding

FAQ

Q: Can I use guile-ares-rs with an existing network REPL?

A: Theoretically it’s possible to upgrade existing REPL to guile-ares-rs, see bootstrap and infect modules (they can be not implemented yet).

Acknowledgements

Thanks for inspiration, help, support and motivation to Jos´e Antonio Ortega Ruiz, Nikita Domnitskii, Maxime Devos, Andy Wingo, Bozhidar Batsov, Andrey Listopadov, Dmitrii Bogdanov, David Thompson, Dmitry Polyakov.

Architecture

@startuml
/'
 ' scale 350 width
 '/
state ForkState <<fork>>
[*] --> ForkState
ForkState --> EvaluationThread
ForkState --> EvaluationThreadManager

state EvaluationThread {
        [*] --> Idle
        Idle --> Running : Scheduled
        Running --> ReturningValue1 : Finished
        Running --> ReturningValue2 : Interrupted
        Idle --> ReturningValue3 : Interrupted

        /'
         ' ReturningValue -[dotted]-> EvaluationThreadManager : Returned Value
         '/

}
state EvaluationThreadManager {
        state "Waiting" as TWait
        TWait: Waiting for Command or Thread Value
        [*] --> TWait
        TWait --> RunEval : Eval Command Received
        TWait --> Interrupt : Interrupt Command Received
        state Interrupt {
                Try -[dotted]-> EvaluationThread : Interrupt

        }
        RunEval -[dotted]-> EvaluationThread : Schedule

}
/'
 ' state Configuring {
 '         [*] --> NewValueSelection
 '         NewValueSelection --> NewValuePreview : EvNewValue
 '         NewValuePreview --> NewValueSelection : EvNewValueRejected
 '         NewValuePreview --> NewValueSelection : EvNewValueSaved
 '
 '         state NewValuePreview {
 '                 State1 -> State2
 '         }
 '
 ' }
 '/
@enduml

@startuml
!theme reddress-lightblue
/' needed for &, but unfortunatelly breaks diagram '/
/'
 ' !pragma teoz true
 '/

title Ares-rs Architecture

actor "nREPL Client" as Client
participant "Ares Server" as Server
participant "Socket" as Socket
participant "Event Loop Fiber" as EventLoop
participant "Evaluation Thread" as EvaluationThread


group New Connection
        Client -> Server: Establish Connection
        Server -> Socket **: Create Socket
        Server -> EventLoop **: Setup Event Loop

        EventLoop -> Client: Connection Established
end

group Eval Processing
        Socket ->> EventLoop: Evaluation Request (eval op)
        EventLoop ->> EvaluationThread: Schedule Evaluation
        EvalutaionThread ->> Socket: Reply 1
        EvalutaionThread ->> Socket: Reply 2
        EvalutaionThread ->> Socket: Evaluation Value
end


/'
 ' participant "Evaluation Supervisor" as Supervisor
 ' participant "Evaluation Manager" as Manager
 ' actor "External Actor 2" as Actor2
 '/
/'
 ' group New Session
 '         Actor -> Supervisor **: Create
 '         activate Supervisor
 '         group Basic Evaluation with stdout and stderr
 '                 Actor ->> Supervisor: nREPL eval operation\nCommand: process-nrepl-message
 '
 '                 Supervisor -> Supervisor: Enqueue.\nCommand: evaluate
 '                 Supervisor ->> Manager **: Run Evaluation
 '                 activate Manager
 '                 Manager ->> Actor: Send Standard Output
 '                 Manager ->> Actor: Send Error Output
 '                 Actor2 ->> Supervisor: nREPL eval operation\nCommand: process-nrepl-message
 '                 Supervisor -> Supervisor: Enqueue Evaluation
 '                 Manager ->> Actor: Send Standard Output
 '                 Manager ->> Actor: Send Evaluation Value
 '                 return Done\nCommand: finished
 '                 destroy Manager
 '         end
 '         group Queued Evaluation with interrupt
 '
 '                 Supervisor -> Supervisor: Command: evaluate
 '
 '                 Supervisor ->> Manager **: Run Evaluation
 '                 activate Manager
 '                 Manager ->> Actor2: Send Standard Output
 '                 Actor ->> Supervisor: nREPL interrupt operation\nCommand: process-nrepl-message
 '                 Supervisor ->> Actor: interrupt-id-mismatch
 '
 '                 Actor2 ->> Supervisor: nREPL interrupt operation\nCommand: process-nrepl-message
 '                 Supervisor ->> Manager: Signal Interrupt
 '
 '                 Manager ->> Actor2: Succesful Interruption
 '                 return Done\nCommand: finished
 '                 destroy Manager
 '         end
 '         deactivate Supervisor
 ' end
 '/
@enduml

@startuml
!theme reddress-lightblue
/' needed for &, but unfortunatelly breaks diagram '/
/'
 ' !pragma teoz true
 '/

title Evaluation Supervisor

actor "External Actor" as Actor
participant "Evaluation Supervisor" as Supervisor
participant "Evaluation Manager" as Manager
actor "External Actor 2" as Actor2
group New Session
        Actor -> Supervisor **: Create
        activate Supervisor
        group Basic Evaluation with stdout and stderr
                Actor ->> Supervisor: nREPL eval operation\nCommand: process-nrepl-message

                Supervisor -> Supervisor: Enqueue.\nCommand: evaluate
                Supervisor ->> Manager **: Run Evaluation
                activate Manager
                Manager ->> Actor: Send Standard Output
                Manager ->> Actor: Send Error Output
                Actor2 ->> Supervisor: nREPL eval operation\nCommand: process-nrepl-message
                Supervisor -> Supervisor: Enqueue Evaluation
                Manager ->> Actor: Send Standard Output
                Manager ->> Actor: Send Evaluation Value
                return Done\nCommand: finished
                destroy Manager
        end
        group Queued Evaluation with interrupt

                Supervisor -> Supervisor: Command: evaluate

                Supervisor ->> Manager **: Run Evaluation
                activate Manager
                Manager ->> Actor2: Send Standard Output
                Actor ->> Supervisor: nREPL interrupt operation\nCommand: process-nrepl-message
                Supervisor ->> Actor: interrupt-id-mismatch

                Actor2 ->> Supervisor: nREPL interrupt operation\nCommand: process-nrepl-message
                Supervisor ->> Manager: Signal Interrupt

                Manager ->> Actor2: Succesful Interruption
                return Done\nCommand: finished
                destroy Manager
        end
        deactivate Supervisor
end
@enduml

@startuml
/'
 ' !theme cerulean-outline
 ' !theme plain
 ' !theme reddress-lightblue
 ' !theme sketchy-outline
 ' !theme vibrant
'/
!theme reddress-lightblue
title Evaluation Workflow

actor User
participant "NREPL Server" as Server
participant "Operation Dispatcher" as Dispatcher
participant "Evaluation Watcher" as Watcher
participant "Evaluation Thread" as Thread

/'
 ' group Connection
 '         User -> Server: Establish Connection
 '         Server -> Server: Create Socket
 '         Server -> User: Acknowledge
 ' end
 '
 '/
group New Session
        User ->> Server: Clone Request
        Server ->> Dispatcher: clone-op
        Dispatcher ->> Dispatcher: Create New Session
        Dispatcher -->> Server: New Session Object
        Server -->> User: New Session Response

        group Evaluation
                User ->> Server: Eval Request
                Server ->> Dispatcher: eval-op
                note right Dispatcher: session context
                Dispatcher ->> Watcher **: eval-op
                Watcher -> Watcher: Setup Channels and Conditions
                Watcher -> Printer **: Attach to Out Ports
                activate Printer
                Watcher -> Thread ** : Run Evaluation Thread
                Thread -> Thread ++: Setup dynamic-wind
                Thread -> Thread ++: Start evaluation

                loop Output
                        Thread ->> Printer: Flush Output
                        Printer ->> Dispatcher: Eval Object
                end

                opt evaluation interrupted
                        User ->> Server: Interrupt Request
                        Server ->> Dispatcher: interrupt-op
                        Dispatcher ->> Watcher: Interrupt Request
                        Watcher ->> Thread: cancel-thread
                        return interrupted
                end
                Thread ->> Watcher: Signal Finished
                Thread ->> Printer: Signal Finished
                return done
                opt evaluation finished successfully
                        Watcher -> Thread: join-thread
                        Thread --> Watcher: evaluation value
                end

                Printer -> Printer !!: Close Port
                deactivate Printer
                deactivate Thread
        end
end

@enduml