Skip to content

Latest commit

 

History

History
91 lines (78 loc) · 11 KB

2023-02-01.md

File metadata and controls

91 lines (78 loc) · 11 KB
title parent
2023-02-01
Meetings

ASWF CI Working Group

Meeting: 01 February 2023

https://zoom-lfx.platform.linuxfoundation.org/meeting/97730566101

Attendees

  • Jean-Francois Panisset (VES Technology Committee)
  • Jean-Christophe Morin
  • Larry Gritz (Sony Imageworks)
  • Andrew Grimberg, LF Release Engineering
  • Christina Tempelaar-Lietz, ILM
  • Aloys Baillet, NVIDIA
  • Jeff Bradley, Dreamworks
  • Kerby Geffrard, OpenRV
  • Alex Mohr, Pixar
  • Deepanshi Sharma, Pixar
  • John Mertic, Linux Foundation

Apologies

New items

  • OpenRV CI requirements: CI builds, test suite, installers...
    • Kerby Geffrard joining
    • The build guy for RV, know how it was before, manage the build of the commercial RV pipeline, doing the same for other Autodesk products and services.
    • Build / infrastructure / backend
    • JF posted to open review initiative our current resources, Kirby saw it, also with Alain Compagnat, who manages the team. Wanted to review before making asks. We know more, and collecting information. Mostly here as a "fly on the wall", we don't even know if we can publish pre-compiled binaries due to legal requirements. We are far from making asks of the CI WG.
    • We have a pull request for Windows containers? Is that abandoned? JF: definitely interested, but have stopped due to resources. Kerby: most of our issues are building on Windows, so it would be useful.
  • TBB to OneAPI transition
    • DCC vendors reporting that libraries still using TBB are blocking their transition to OneAPI
    • Real world experience on dealing with deprecated APIs
    • Mixing TBB and OneAPI (too many threads problem): https://www.intel.com/content/www/us/en/develop/documentation/onetbb-documentation/top/onetbb-developer-guide/migrating-from-threading-building-blocks-tbb/mixing-two-runtimes.html
    • Any guidance we can offer projects?
    • Larry: I have done it, for processes that use TBB not too sophisticated, thread pools and mutexes for instance. Haven't tried using both at once. More sophisticated stuff like graphs have changed or may not be available anymore. Might have to write some of those components yourself. Haven't needed that for any of our projects. Not sure that CI WG can offer much, it's a project per project? Libraries typically not using those higher level APIs, probably more the apps.
    • Didn't have to change code / use #ifdefs, possibly just in CMake find to find the right library. But names of the classes or includes have not changed.
    • JF: would a FindOneAPI cmake module be helpful? Larry: the oneAPI stuff comes with a CMake config, work is already done. Looked at my includes, the oneAPI stuff is still #include<tbb/...>
    • Reach back to DCC vendors to ask?
  • GitHub Projects disabled at Organization level, detected by OTIO
    • Other ASWF projects using those?
    • Any fallout?
    • JC: CI checks for dead links in documentation, and documentation linked to dead projects, so that's how it was detected.
    • JC: It was re-enabled, and it seems to work now. Back in OTIO, Rez.
    • Andrew: assuming it was a mistake, I didn't do it, didn't dig through audit logs. But not going away, GitHub adding new features to Project system. John: probably a mistake, we want to be conscientious from a coordination aspect. All of our projects sharing an org can result in collisions. We don't a lot of projects that use the same GitHub Org? Andrew: correct, in general most umbrella orgs have major projects in their own org. But ASWF is mostly mono repos, whereas other big LF projects are multiple repos under an organization. We try to keep things consistent. JF: don't need to change the way our GitHub repos are organized. John: OpenAssetIO may be thinking of its own organization, also OTIO that have multiple repos. OTIO has an org just for OTIO for adapters and language bindings. https://github.com/OpenTimelineIO

Follow Ups

  • Need to produce some kind of deliverable from our CII badging discussions
    • Action item for JF
    • John: did we want to do a write up on some of the items we discussed? Larry: there's still a craving for a bit of a centralized approach to dealing with security. Some of the requirements to have a security audit every X year. John: that one, the foundation should fund a sponsor. We tagged everything where we had questions, I'll put together a document to review. There's also a security WG that's forming, so some overlap there. Larry: may help to have a Wiki page where we can have the feedback for every item, have a place where it's written down, and as every project has issues, they can document what they did. Allowing new people to access that wisdom. John: makes sense, would want to upstream a lot of this back up to the project. So of our feedback is ASWF specific, but some broader feedback can go back to the process. JC: we talked at the last meeting of building a list, point by point with the editorial notes. Larry: there are some explanations getting better in the CII documentation, so if we can upstream, that might help everyone. CII is an LF project, so we can talk to them directly? John: yes, we do, and PRs are also welcomed. Will put together a document of what we captured, we can review it and see what we can push upstream.
  • Update on GitHub Actions for pay runners (Andrew)
    • Updates from GitHub (GPU runners?)
      • No update, missed last update from GitHub team
    • Updates on Apple Silicon Runners?
    • ARM Linux / Windows runners?
      • Andrew: GitHub is aware of interest, but no timeline.
      • Larry: Travis may have some Linux ARM machines
      • VEXXHost is still available
      • Kerby (chat): is there a requirement for ARM other than Apple Silicon? Larry (chat): speaking from OIIO, I do see occasional users who care, and I definitely see distros which prefer that anything they carry on one platforms builds if at all possible on all the platforms the distro supports. JC (chat): we do have libraries that are expected to work on ARM Linux distros if I'm not mistaken.
    • Andrew: Cost usage: we slightly exceeded our cost settings for January but $31, had to scramble to getting that fixed. GitHub gave us a coupon to reduce it to $1500. If we are very close to our limit and our job starts, it will complete and we will be over. So lowered our limit to $1450 to take that into consideration. So we probably need to reforecast budgeting and potentially update our limit.
    • Kirby: right now we have VMs, we don't try to keep them small, so we don't know the minimal requirements. Most painful part of building OpenRV is spawning off the dependencies, so if you are building OpenRV and ffmpeg at the same time, that can bump up requirements. JF: what if you had a build container with all your dependencies? Kirby: would need to refactor the build process, right now we build everything. JC: caching could also help, GitHub has caches. Kirby: the build support keeping third party builds in a folder. JF: could also leverage Conan repo.
    • Andrew: have raised with GitHub about being able to do custom runners and have fallback to free runners if some conditions apply, haven't heard back from GitHub on that.
    • Aloys: still building aswf-docker on the free runners since didn't need it for now, but seemed difficult to switch between free and non-free runners, so only used the paid runners a couple of times to build the larger components.
    • Andrew: your use of those paid runners is a very good one. How do we get builds to work the most optimal way without exceeding budget.
    • Andrew: paid runners will be the only route to GPU builds. We're not seeing all the costs yet. The AWS CodeBuild costs are not part of that $1500, but was anticipating they would shift to the GitHub builders when the GPU runners are available, so we can increase the GHA cost limit when that happens.
  • Aswf-docker update (Aloys)
    • Aloys: not since 2 weeks ago, released last batch of 2022 updates, changes were mostly material MaterialX in vfx-all, release notes are extensive.
    • No progress since then, we might try to go into 2023, using a RHEL 8 based distro. We haven't started that process.
    • There is a request to change MaterialX from a static to a dynamic build (DSO), happy to do that, didn't realize the default was a static library, might be only change in there.
    • JF: are there other projects that would need build containers? Aloys: no idea, would be good to look at list of dependencies for OpenRV, and if it would be useful to have pre-built dependencies (ffmpeg).
    • JF: would raw2aces need a container? Is the mandate to have containers for every project, or wait for request. JC: maybe best to wait. OTIO now has a C++ API, we have dependency on a small external project, we don't need a full container for that. JF: but you get VFX Platform compliance from container. JC: we had planned to test for VFX platform compliance, but we can use the generic base image. OTIO is a bit special, it was Python before, now it's C++, but most of our users are using Python API. We are shipping Python binaries, and they work all the way back to CentOS 6.
  • Proposed "small" projects from Robin Rowe at TAC meeting, is this something CI WG could adopt?
    • Libunistd : an abstraction layer allowing the use of POSIX APIs when porting to Windows
    • Cmaker : a Cmake-based project template generator
    • Also file I/O abstraction layer from OIIO

Tools and Links

  • GitHub CoPilot usage for ASWF projects, any stories / tips / tricks that can be shared?
    • Larry: an ongoing issue, we don't have the tools (promised for the future) that will allow you to ask about the suggestions it makes, where it came from, and filter based on licensing. When I used it, I don't find it all that hard when its suggestion is auto-complete of a line, customized to my code, vs a block of a whole function, "where did that come from". The bigger a chunk it gives you, the most likely it is to have "wrong" things in it.
    • Aloys: a very clever auto complete. Larry: it's very handy for that, sometimes it suggests exactly what I would have typed. And sometimes it's really clever, it's folded in my own variable naming convention, it's not just cut-n-paste. But when it gives you a whole function, you may not know where that came from.
    • Jeff: looking at the cost. Larry: not paying for it, using it for free as an open source maintainer. Not sure I would pay for it out of my pocket. Aloys; started paying for it, $100/year. Would be nice if ASWF could contribute it to some contributors. Larry: what is a contributor? Hard to determine.
    • Aloys: onus is on the developer, if you get a contribution from CoPilot that looks "strange", it's on you to reject it. Larry: we face the same issue with anyone who makes a PR, even with the DCO, it's hard to know where the code comes from.