Improve the flexibility of respondd providers

[lemoer]

I really dislike the fact that writing a data for respondd now has to be done in C. This makes the whole process of monitoring a lot more static. It should be a lot easier and faster to add new data to respondd.

The common data sources in providers are:

• uci
• ubus call
• file
• constant

The idea is now to cover this data in a config file and add some post processing. Maybe we can do this with kind of a domain specific language (DSL) in lua:

The main functionality would be given by this functions:

provide(source, postprocessing_handler, cache_time)
emit(json_path, value)
as(json_path)(value, emit) -- higher order function, simply "emits" its value

Constant:

provide(const('1'), as('/node/software/status-page/apiverison'), 60000)

File:

provide(file('/sys/module/batman_adv/version'),
        as('/node/software/batman-adv/version'), infinity)

UCI value:

provide(uci_option('system.@system[0].hostname'), as('/node/hostname'), 60000)

Custom post processing handler:

provide(uci_option('autoupdater.settings'), function (result, emit)
  emit('/node/software/autoupdater/enabled', result['enabled'])
  emit('/node/software/autoupdater/branch', result['branch'])
end, 60000)

UBus call:

provide(ubus_call('system', 'info'), function (result, emit)
  emit('/statistics/loadavg', result['load'][0])
  emit('/statistics/uptime', result['uptime'])
  emit('/statistics/memory', result['memory'])
end, 60000)

I think the fact that the previous respondd implementation with lua providers caused a lot of memory consumption, was caused by the fact, that a lot of lua modules (uci, ubus, ...) were loaded by the provider modules. Maybe this could be solved by implementing the above mentioned functions in C using the native bindings.

What do you think about this idea?

[neocturne]

I think we should generally avoid using Lua in daemons, as a garbage-collected scripting language will always use more RAM than C.

Having a DSL in C might be a good idea though - and we could even try to avoid using json-c and thus keeping JSON objects in RAM at all and directly write JSON out as we generate it.

[lemoer]

I think we should generally avoid using Lua in daemons, as a garbage-collected scripting language will always use more RAM than C.

The main idea was to improve the flexibility of this daemon. Often it is very useful that one can modify (or add) a provider by simply modifying files on the router while is online. If you need C skills and a build ready LEDE/gluon development environment, you will often leave out monitoring because it's to complicated to write new modules. Decreasing the memory footprint is a minor aspect of this thread, but would be also nice.

I do not have a lot experience in garbage collection, but can't we reach a negligible difference if the lua functions are short and do not use external lua libs?

Having a DSL in C might be a good idea though

The concept to use lua was caused by the idea to add some easy to write postprocessing functionality for the different data sources. I think the postprocessing is useful because we want to do complex stuff like parsing output of ubus calls or batman neigh tables. How would you solve this postprocessing in the DSL? Or should we drop support for postprocessing of the data sources?

and we could even try to avoid using json-c and thus keeping JSON objects in RAM at all and directly write JSON out as we generate it.

I'm not sure whether I got it. Did I understand it correctly, that you want to generate the json on demand while serving the requests? I'm not really sure if we can decrease the memory footprint with this change, since caching would be still necessary. Saving the whole response body as a string would consume more memory than before.