Turtlebot CI job

[dhood]

Here's what I have in mind for creating a CI job for the turtlebot demo. I am looking to get the direction approved before speaking with @nuclearsandwich about how to implement it.

As I imagine it, this will be a manually-triggered CI job while we're developing the demo and then can switch to a nightly once it's stable.

1. Add the supplementary repos file as a jenkins parameter
   1. Modify the ci_job.xml template to include this parameter if something like supplementary_repos_file is set when evaluating the template.
2. Add the ci_linux-turtlebot_demo job to the farm
   1. Add it to the create_jenkins_job.py script
   2. Set the default supplementary repos file to https://raw.githubusercontent.com/ros2/turtlebot2_demo/master/turtlebot2_demo.repos or something (using a similar strategy to 38f57f3#diff-10b89ee3f1ac5a693c0ff698ad83c16dR1)
3. Pull the supplementary repos file into the source space
   1. Put a for-loop around this section that calls vcs
4. For turtlebot demo dependencies available as binaries, only add them to the dockerfile if necessary
   1. Inject the installation of the dependencies into the dockerfile using sed as is done to modify the FROM command for arm jobs -- OR --
   2. Use the create_dockerfile method from ros_buildfarm to generate the Dockerfile from a template, only including turtlebot dependency snippet if appropriate
5. Ask turtlebot demo developers to target PRs at the turtlebot2_demo.repos file, and PRs for additional binary dependencies at the turtlebot demo extra dependencies list on ros2/ci

I understand point 4 as just a "nice to have".

[dhood]

The above was written with the understanding that the turtlebot demo repos file would be pulled in on top of the ros2 repos file.

Certainly we can just have the turtlebot demo repos file written such that it also includes the contents of the standard ros2 repos and in that case the task is simplified to:

1. Add the ci_linux-turtlebot_demo job to the farm
   1. Add it to the create_jenkins_job.py script
   2. Set the default repos file to https://raw.githubusercontent.com/ros2/turtlebot2_demo/master/turtlebot2_demo.repos or something (instead of using 38f57f3#diff-10b89ee3f1ac5a693c0ff698ad83c16dR1)
2. For turtlebot demo dependencies available as binaries, only add them to the dockerfile if necessary
   1. Inject the installation of the dependencies into the dockerfile using sed as is done to modify the FROM command for arm jobs -- OR --
   2. Use the create_dockerfile method from ros_buildfarm to generate the Dockerfile from a template, only including turtlebot dependency snippet if appropriate
   3. Add an argument to the Dockerfile to only install the dependencies if appropriate (as is done for PLATFORM)
3. Ask turtlebot demo developers to:
   1. target PRs at the turtlebot2_demo.repos file, and PRs for additional binary dependencies at the turtlebot demo extra dependencies list on ros2/ci
   2. keep the repos file "in sync" with the ros2.repos file

Again point 2 is just a "nice to have".

Unless anyone objects I will go with this approach as the first iteration and then we can improve it in another iteration. I would appreciate an approval on this direction before starting work.

[mikaelarguedas]

Thanks for iterating on this, that matches what I understood of the offline discussion and allow us not to have to deal with all the edge cases of having/merging two repos files (as discussed in #17)

[dhood]

Thanks @mikaelarguedas

inspired by #33 I updated the approach to pass an argument to the dockerfile for point 2 (thanks for the idea @dirk-thomas!)

[nuclearsandwich]

I started working on updating create_jenkins_job.py script for the new CI job and ran into issues with my local dev environment not being able to communicate with Jenkins.

Stacktrace in this gist.

[dhood]

The script will look for a ~/.buildfarm/jenkins.ini file. In it should be an entry like:

[http://ci.ros2.org]
username=<github_id>
password=<token>

where the token is generated from https://github.com/settings/tokens

Does that match what you have in your setup?

[dhood]

(there are a couple of readme links you have to follow to get that information. here is the final section of relevance: https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/environment.rst#provide-credentials-for-jenkins-master)

[nuclearsandwich]

Does that match what you have in your setup?

yes, mine currently defines a [DEFAULT] entry as well since I spent some time Friday troubleshooting.

[nuclearsandwich]

With @dhood's help we sorted it out. The token needs some scope beyond empty. We got success with a token that had repo and user scopes. I'll do some checks to figure out what the minimum scope required is and update the docs.

[mikaelarguedas]

@dhood @nuclearsandwich should this be closed now that #35 has been merged and deployed?

[nuclearsandwich]

I think it would be fine to close it and iterate on the configuration (system dependencies, repos file in https://github.com/ros2/turtlebot2_demo/blob/master/turtlebot2_demo.repos) or if we wanted to set some more concrete goals for testing those before closing this, I'd be up for trying to get it across that finish line as well.

When I tried optimistically to just toss the two repos mentioned in the tutorial into a repos file with the rest of the ros2 deps I got a missing kobuki driver package.

http://ci.ros2.org/job/ci_turtlebot-demo/5/console

There's no kobuki repo that I can find in the ros2 org. Should that be a system dependency, or a source build of a ros 1 package?

[mikaelarguedas]

Ah yeah that's a bit misleading. It looks like the parts of interest of kobuki_driver have been carved out and put into the turtlebot2_drivers package via ros2/turtlebot2_demo#7. So I think it should be neither a system dependency or a ros1 package built from source in this case.

[mikaelarguedas]

Follow-up on this: my bad, it looks like this package still depends on several ros1 packages (according to this comment, thanks @clalancette for pointing it out)

[dhood]

we mentioned that it is left to turtlebot demo developers to iterate on the turtlebot demo repos file so I'll close this for now.