Scripting interface to simulation workflow

[chapulina]

Desired behavior

Gazebo can be used as a library by other programs, like shown on the custom_server example. The Server C++ API can be used to initialize the simulation and step it using the Run function.

It would be interesting to expose that functionality through a C API so that it can be eventually wrapped by other scripting languages like Python.

We already have a C API in ign.cc which is used by the command line tool, but we don't expose that to users or keep it stable.

The new C API should be able to do things like starting a server, loading an SDF file, stepping the simulation and retrieving data (entities and components) from it.

Alternatives considered

Instead of adding a C API, we could extend the C++ API to add the desired functionality, and that can be converted to scripting languages using SWIG like we're doing on ign-math.

Implementation suggestion

Before creating the API, it would be interesting to write a concise design doc into https://github.com/ignitionrobotics/design

Additional context

The gym-ignition project is using SWIG to wrap our C++ API, with some sugar on top 🍬

[ahcorde]

I started a prototype of ign-gazebo based on the examples/standalone/custom_server. https://github.com/ignitionrobotics/ign-gazebo/tree/ahcorde/pybind11

I converted some few method to make it work:

• Gazebo
  • server (run, has_entity and running() methods)
  • ServerConfig (setSdfFile() method)
• Common
  • console -> (Setversosity static function)

You can try it with the following Python script:

from ignition_gazebo import ServerConfig, Server
from ignition_common import set_verbosity
import time

set_verbosity(4)
server_config = ServerConfig()
server_config.set_sdf_file('rolling_shapes.sdf')
server = Server(server_config)

print('Has entity ', server.has_entity('box', 0))

server.run(False, 10000, False)

while(server.is_running()):
    time.sleep(0.1)
 

Incremental example 1

But I was thinking about how to get some data from the simulation. I thought that I can subscribe to some ignition topic. Then I ported:

• Transport
  • Node (subscribe() method)
• Msgs
  • Pose_v msg

from ignition_gazebo import ServerConfig, Server
from ignition_common import set_verbosity
from ignition_transport import Node
from ignition_msgs import PoseV

import time

set_verbosity(4)
server_config = ServerConfig()
server_config.set_sdf_file('rolling_shapes.sdf')
server = Server(server_config)

print('Has entity ', server.has_entity('box', 0))

def square(msg):
    msg.show()

node = Node()
node.subscribe('/world/shapes/pose/info', square)

server.run(False, 10000, False)

while(server.is_running()):
    time.sleep(0.1)

Incremental example 2

Then I was thinking about to get values from the msgs in Python, not calling a method that prints everything. Here is where the magic start. I don't know if there is a simpler way to convert types between pybind11 and SWIG.

from ignition_gazebo import ServerConfig, Server
from ignition.math import Vector3d
from ignition_msgs import PoseV

p = PoseV()
print(p.get_size())
print(p.get_names())
print(p.get_poses())
print(p.pose())
print(p.pose().x())
p.set_pose(Vector3d(0, 1, 2))

The two key methods are:

• set_pose() (Setter): This method require to convert ignition::math::vector3 Swig object in to a ignition::math::vector3 in cpp. For this I used res1 = SWIG_ConvertPtr(h.ptr(), &argp1, SWIGTYPE_p_ignition__math__Vector3T_double_t, 0 | 0 );
• pose() (Getter): This method will return a ignition::math::vector3 Swig object in Python. For this I used PyObject * resultobj = SWIG_NewPointerObj((new ignition::math::Vector3< double >(static_cast< const ignition::math::Vector3< double >& >(result))), SWIGTYPE_p_ignition__math__Vector3T_double_t, SWIG_POINTER_OWN | 0 );

These thow functions are defined in pythonPYTHON_wrap.cxx which is automatically generated by SWIG when we create the wrapper for ign-math. I copied partially this file to make it work this example. I know this is a big hack and it's not the path to follow.

Suggestions are welcome

FYI @chapulina @azeey @scpeters

[ahcorde]

By the way this is branch https://github.com/ignitionrobotics/ign-gazebo/tree/ahcorde/pybind11

[traversaro]

I don't know if there is a simpler way to convert types between pybind11 and SWIG.

By the way, regarding the interoperation of Python bindings created by different systems a relevant discussion in Pinocchio is stack-of-tasks/pinocchio#1518 . There is a PR from @jmirabel in stack-of-tasks/pinocchio#1519 providing (as far as I understand) interoperability of pybind11 and boost-python, while in stack-of-tasks/pinocchio#1518 (comment) @jcarpent mention something related to SWIG and boost-python interoperability, that I guess was provided by this PR : https://github.com/stack-of-tasks/eigenpy/pull/219/files .

FYI @GiulioRomualdi @S-Dafarra @prashanthr05 @diegoferigo as I think this discussion is similar to the one in ami-iit/bipedal-locomotion-framework#214 .

[diegoferigo]

But I was thinking about how to get some data from the simulation. I thought that I can subscribe to some ignition topic.

While this is a pretty straightforward path to extract data from the simulation as first attempt, I would argue about its efficiency and precision. In your example, the Server object runs within the same process of the Python interpreter, this also means that the ECM could be accessed directly from memory. Even if it is surely too early to comment, I want to provide my insights:

1. You are run-ning the server asynchronously, hence the sleep calls below
2. In a tight loop and in many cases, you cannot be sure you received the most updated piece of data from a topic (from a synchronous service maybe yes)

My suggestion for the first comment would be to run the server synchronously so that code that caller doesn't have to poll the simulation status. For the second, instead, you might want to exposing to some extent the ECM to Python. If the whole ECM is too much, access can be limited by only exposing the World, Model, etc classes that abstract the ECM. Remember also that you can issue paused steps to trigger the ECM to process (almost) all simulation without advancing the Physics.

These two comments are also part of how the ScenarIO project is based. You can find the wrapper class of Server we are using in GazeboSimulator. After few iterations, we converged to use a custom system for extracting the ECM pointer and, with that, expose our own World, Model, etc classes that can be seen as extensions of those present in upstream (we also have an additional abstraction layer here, but it doesn't matter much for this discussion). Maybe this overview could help in this initial design phase, it's already few years we are working on a similar activity and we can share some experience.

What's still challenging is how to expose sensors (robotology-legacy/gym-ignition#199), it's something not yet possible and I think it could become a bottleneck in the future, especially for all the visual data-rich sensors. Passing through the many copies using transport might not be efficient. In any case, exposing sensors is an even wider discussion, I just wanted to mention it here since it could affect design choices.

[ahcorde]

Thank you for your feedback @diegoferigo and @traversaro,

I has been prototyping more ideas here, in particular

• Be able to add some callback to a system though Python
• Replicated the same demo with SWIG (to compare complexity)

SWIG Example

from ignition.gazebo import Console, ServerConfig, Server, HelperFixture, World, Callback, EntityComponentManager
from ignition.math import Vector3d

import time
Console.set_verbosity(4)
helper = HelperFixture('gravity.sdf')

post_iterations = 0
iterations = 0
pre_iterations = 0

class CallPostUpdate(Callback):
    def __init__(self):
        super(CallPostUpdate, self).__init__()

    def call_update(self, _info, _ecm):
        global post_iterations
        post_iterations += 1


class CallUpdate(Callback):
    def __init__(self):
        super(CallUpdate, self).__init__()

    def call_update_no_const(self, _info, _ecm):
        global iterations
        iterations += 1


class CallPreUpdate(Callback):
    def __init__(self):
        super(CallPreUpdate, self).__init__()

    def call_update_no_const(self, _info, _ecm):
        global pre_iterations
        pre_iterations += 1


class CallConfigure(Callback):
    def __init__(self):
        super(CallConfigure, self).__init__()

    def call_configure_callback(self, _entity, _ecm):
        world = World(_entity)
        gravity = world.gravity(_ecm)
        modelEntity = world.model_by_name(_ecm, "falling")
        print("modelEntity ", modelEntity)
        # print("gravity ", gravity)


postupdate = CallPostUpdate()
update = CallUpdate()
preupdate = CallPreUpdate()
configure = CallConfigure()

helper.on_post_update(postupdate)
helper.on_update(update)
helper.on_pre_update(preupdate)
helper.on_configure(configure)

helper.finalize()

server = helper.server()

server.run(False, 1000, False)

while(server.running()):
    time.sleep(0.1)

print("iterations ", iterations)
print("post_iterations ", post_iterations)
print("pre_iterations ", pre_iterations)

Example

from ignition_gazebo import ServerConfig, Server, HelperFixture, World
from ignition_common import set_verbosity
from ignition_transport import Node
from ignition_msgs import PoseV
from ignition.math import Vector3d
import time

set_verbosity(4)

helper = HelperFixture('gravity.sdf')

post_iterations = 0
iterations = 0
pre_iterations = 0

def on_configure_cb(worldEntity, _ecm):
    print("worldEntity ", worldEntity)
    w = World(worldEntity)
    # print("Gravety ", str(w.gravity()))
    modelEntity = w.model_by_name(_ecm, "falling")
    print("modelEntity ", modelEntity)


def on_post_udpate_cb(_info, _ecm):
    global post_iterations
    post_iterations += 1
    # print(_info.sim_time)


def on_pre_udpate_cb(_info, _ecm):
    global pre_iterations
    pre_iterations += 1
    # print(_info.sim_time)


def on_udpate_cb(_info, _ecm):
    global iterations
    iterations += 1
    # print(_info.sim_time)


helper = helper.on_post_update(on_post_udpate_cb)
helper = helper.on_update(on_udpate_cb)
helper = helper.on_pre_update(on_pre_udpate_cb)
helper = helper.on_configure(on_configure_cb)

helper = helper.finalize()

server = helper.server()
server.run(False, 1000, False)

while(server.is_running()):
    time.sleep(0.1)

print("iterations ", iterations)
print("post_iterations ", post_iterations)
print("pre_iterations ", pre_iterations)

I have some blockers:

• I'm able to cast inside pybind11 (c++) a SWIG object (copying some of the utils by SWIG in the pythonPYTHON_wrap.cxx), but I'm not able to do the opposite, generate a SWIG object inside the C++ code.
• Something similar is happening with the SWIG prototype. SWIG is generating pythonPYTHON_wrap.cxx with all the stuff to create SWIG objects but it's not providing any header which means if we generate an different SWIG module (ign-gazebo in this case) we don't have access to create object from the ign-math SWIG module.

[ahcorde]

Ey @diegoferigo and @traversaro, I saw that you have read the last comment (because of the thumbs up).

Do you have any idea about how to create SWIG object inside pybind11 ? or do you an an opinion about mixing these two libraries ? it's SWIG or pybin11 the way to go ?.

We appreciate any feddback here.

[diegoferigo]

Ey @diegoferigo and @traversaro, I saw that you have read the last comment (because of the thumbs up).

Do you have any idea about how to create SWIG object inside pybind11 ? or do you an an opinion about mixing these two libraries ? it's SWIG or pybin11 the way to go ?.

We appreciate any feddback here.

Hi @ahcorde, I was hoping someone to chime in, I have experience with both pybind11 and SWIG used independently, but I've never tried to make bound objects compatible from one system to the other. For what regards pybind11 vs SWIG, I can only repeat what I've already wrote in gazebosim/gz-math#210 (comment).

In my experiments, I've only used SWIG within the same project. This means that I didn't have the chance to explore how Python objects from other projects can be passed as arguments. I guess that, under the assumption you control all the .i files, you can install in the same location all the files from all Ignition libraries, and import them where needed. It might mean that a lot of glue code gets duplicated (maybe there's a smarter way) but it should work.

Staying instead in a pybind11 setup, the interoperability between different projects seems easier. We successfully managed to use the official manif bindings to create objects and pass them to our bipedal-locomotion-framework. Things have worked magically (modulo some edge case).

In light of my gazebosim/gz-math#210 (comment), considering the stability of your APIs, the standard you're using in the public headers, the willing to make bindings coming from different projects interoperate, and -perhaps- easy numpy support, I think that the task would be easier with pybind11, but I also imagine that only supporting Python could be a blocker.

[traversaro]

Ey @diegoferigo and @traversaro, I saw that you have read the last comment (because of the thumbs up).

I must confess that it was a "thumb up" for "great that someone is working on this", without reading in detail your comment, not sure if this violates any code of conduct or similar. : )
I will read more in detail about your experiments and let you know.

[traversaro]

I'm able to cast inside pybind11 (c++) a SWIG object (copying some of the utils by SWIG in the pythonPYTHON_wrap.cxx), but I'm not able to do the opposite, generate a SWIG object inside the C++ code.

Can you make an example of what you would like to do? Anyhow, in general my general instinct is that mixing pybind11 and swig bindings is a bit of dangerous/effort intensive path.

Something similar is happening with the SWIG prototype. SWIG is generating pythonPYTHON_wrap.cxx with all the stuff to create SWIG objects but it's not providing any header which means if we generate an different SWIG module (ign-gazebo in this case) we don't have access to create object from the ign-math SWIG module.

Why we would need to create manually ign-math objects if all is handled by SWIG? If I remember correctly how these things works in SWIG, you should install the <>.i file of ign-math and then import it in the <>.i of ign-gazebo. For an example of such workflow, see https://github.com/casadi/casadi/tree/3.5.5/swig/extending_casadi and casadi/casadi#1559, in which casadi (a C++ project with SWIG bindings) is extended by another C++ project (in this case extending_casadi) and both have Python bindings. Another related CasADi issue is casadi/casadi#1613 .

[traversaro]

xref related issue: pybind/pybind11#1706 .

[srmainwaring]

@ahcorde thanks for posting your branch with the experimental pybind11 Python bindings, it's been very helpful. My immediate interest is to obtain bindings to the ignition-msgs and ignition-transport libraries so I can subscribe to ignition topics in Python for further processing.

Have you looked into using https://github.com/pybind/pybind11_protobuf to manage the conversion between the Python bindings generated by protoc and the interfaces into ignition-transport etc. generated by pybind11? On the surface it looks like it might help avoid having to provide manual bindings for each message type but I have not yet managed to get an example running. Unfortunately the pybind11_protobuf project is missing CMake files so there is a bit of upfront work to get going - if you've looked at it and ruled it out for a reason that would be good to know.

[chapulina]

Hi @srmainwaring , the scope of our current scripting efforts is limited to ign-math and ign-gazebo. The goal is to release a proof-of-concept with these libraries and expand to others from there.

Thank you for pointing us at pybind11_protobuf, I don't think it was in our radar. For ign-msgs / protobuf, we've long wanted to stop packaging the generated code and instead ship just proto files, see gazebosim/gz-msgs#113. We may need to remove all generated classes from our public APIs for that, which is something we're not sure we want to do. If that route is taken, maybe there's no need to wrap C++ into Python. But we have no plans to work on it in the near-future, so wrapping C++ may be the quickest thing to do.

[srmainwaring]

@chapulina thanks for the outline. As you're not planning to work on ign-transport at present I've taken a look into what would be involved in getting ign-transport to work almost natively in Python.

I've put together a project here https://github.com/srmainwaring/python-ignition which generates bindings for the messages I'm interested in and exposes a mock-up of ign-transport which is then exposed to Python with pybind11. I think the only interface that will need a bit of thought is subscribe as it's templated and ideally I'd like the Python side callbacks to use the derived types. Aside from that it all looks feasible.

It's all built with Bazel as that is what the protobuf and pybind11 projects seem to be using. I've started to look at the ign-bazel project to see how it can be adapted as an external dependency and what's needed to get a garden version of ignition transport working.

If there's any interest or overlap with your team's work - happy to collaborate, otherwise I'll post something on community if I manage to get it all working.

[chapulina]

That's great to hear, @srmainwaring ! We'd be willing to maintain and package the pybind11 interfaces within ign-transport and ign-msgs similarly to how we're doing for ign-math right now.

We'd need to get it to work with CMake though. Our Bazel support is experimental and meant to help downstream users who want to use it, but our official build tool is CMake.

In any case, you should advertise your work to the community, no matter where it lands 😉

[srmainwaring]

We'd be willing to maintain and package the pybind11 interfaces within ign-transport and ign-msgs similarly to how we're doing for ign-math right now.

@chapulina that would be great thanks. The plan is to get it working with CMake but easiest path initially is using Bazel as some of the dependent libraries are missing CMake builds (namely pybind11_protobuf). In the meanwhile I'll track the Bazel version in this issue gazebosim/gz-bazel#40 (and the publishing bindings are working!!)

[chapulina]

The proof-of-concept has been merged, so I'm closing this issue. If bugs come up and more features are desired, please open new tickets for them. We'd love to hear what the community thinks of the proof-of-concept before adding more features to the current implementation.