| title | parent |
|---|---|
2024-06-19 |
Meetings |
Meeting: 19 June 2024
- Jean-Francois Panisset, VES Technology Committee
- Larry Gritz, Sony Imageworks / OIIO / OSL
- Eric Strauss, Open Review Initiative
- Michael Kessler, Open Review Initiative
- Jean-Christophe Morin, Rez
- Open Review Initiative CI Requirements
- JF: Lots of interest in building ORI tools expressed at FMX, especially on Windows
- Micheal: Epic (through myself) contributing to xStudio project, specifically to get it to build on Windows. Contributing CMake + vcpkg, also making code more OS agnostic. As of today, the branch we have has Windows support PR, can take a vanilla install with 3 manual dependencies:
- git
- Visual Studio
- Qt
- Ninja
- Will give you a portable install, but will take a while. vcpkg doesn't provide binaries by default unless you are a "trusted partner".
- Around SIGGRAPH timeframe, hoping to have it merged, talking to DNeg about it, merging into xStudio 2.0, will join mainline.
- Ideal case would be to have CI for both validations (build + test), currently all testing is manual on Windows
- Ideal case: provide a pre-compiled binary. Highly sought after on Windows platform.
- Would like to have CI on a push basis to validate PRs, not a CI expert nor a Windows build expert. JF: lots of examples in other ASWF projects. Don't hesitate to ask questions on Slack.
- Michael: we're just starting to think about it.
- JF: does xStudio do unit testing? Michael: yes, think it's mostly unit testing, not integration testing (no GUI testing?). Some issues with the Actor framework they are using. Might take someone who is more aware of testing for automated integration testing.
- JF: if you need a licensed product for UI testing, that could be possible.
- Michael: aiming for unit testing of base components, since DNeg uses this internally, we will want to align to whatever testing they want to use. There's a significant cost to keeping these tests up to date.
- JF: several project use CTest, but not explicitly.
- Michael: has anybody produced any thoughts on UI testing? JF: look at previous CI WG meetings history. JC: OpenCue has UI testing, testing PySide, and they test it in GHA CI, not sure if they run the tests there. Michael: UI is Qt 5.15. JC: so shouldn't be too much different from PySide. Michael: we should be able to use that.
- JF: is there a script to install the dependencies? Michael: we had a contractor before and had something for that, and you need admin permissions to install from the Internet. Anyone who is doing any development, it didn't seem like the right thing to do? I've tried to find a reliable way to automate Qt install, only reliable way for Windows seems to be to get it from Qt-online installer. If anyone has a better way, I'm open. The vcpkg distribution often introduces breakage. Haven't tried Conan, could be an avenue. JC: it has all the versions. And it's pre-compiled. Michael: Conan might be the solution, we didn't want to mix vcpkg and Conan. Visual Studio is pretty easy to automate, could give a short script. JC: buildtools installer has a command line, pass it the components you want to install. Michael: I haven't done that personally, maybe we can bring it down to zero.
- JF: don't underestimate the gulf between a Python-focussed developer and a full C++ environment. Michael: we could automate these, but not sure if that's really worth it compared to providing binaries.
- JF: ASWF Artifactory repository is also available. Michael: we could use a NuGet repo? JC: yes, it supports it, not sure it's enabled. Michael: that would allows us to avoid rebuilding all our dependencies.
- JC: didn't they say there may be issue with licensing?
- JF: docker container images on Windows would also be a nice long term solution
- Larry: I have builds against "top of tree" of all my dependencies, that frequently breaks since people check in broken code to top of tree of their projects. Unfortunately not easy to avoid a whole run breaking because of that build. More than once I'll let the upstream dependency know that they've checked in a broken change. I don't build all dependencies like LLVM or Qt, but something that can be built in a reasonable time, I build. That way I'm not surprised by a "real" release. JF: is there a way to "build the latest release tag"? Larry: not sure, but I do have a "build the latest release", but have to manage those manually. An automated way to do this would be useful. Would also love to know how to automatically trigger a CI build of my project when a certain dependency releases. JC: dependabot is not flexible enough, you have to work with the formats it supports, and it's not very flexible. You would want to use renovate which is much more powerful, can have custom filters, custom matchers, at least with on-prem version, not sure about open source version. The issue is that C/C++ is poorly supported by these dependency management tools. Some support Conan files and Conan lock files, but not easy to implement. Python / Java / Go / Rust / Javascript first. There's no central index server for C/C++, some projects track their project versions in GitHub, some don't care about tags, some projects just release from top of tree...
- Erik: at next meeting will bring someone from OpenRV, struggling with Windows build, will send someone to a follow up meeting. Bernard should be joining. Commercial RV already building from open source repo, so should be simpler. Michael: also different product cycles, xStudio is brand new. Resource constraints to making the open source repo the main one.
- Larry: I recently merged on OpenImageIO, seed of how to build dependencies, make it a bit more fail-safe to help people who don't know how to install dependencies. Michael: what are you using? Larry: I rolled my own since I tried every one out there and couldn't make anything work to my satisfaction, including CMake external library support, CPM, Conan, nothing specifically wrong, but not quite scoped to the "simplest solution I needed for my problem". Each one had some shortcoming that made me uncomfortable and had complexity I didn't need. I built a two tiered system, one layer when "find" doesn't find a dependency. For the 4 dependencies I've done so far, it seems to work fine. Or maybe I could use one of the other systems. Michael: our department has something similar, so understand the justification. Larry: it's a huge time sink, had an edge case where it found something but wasn't in the version range I specified, don't want to pick up artifacts from wrong version installed.
- JC: eventually people will upgrade to newer CMake versions, can use dependency finder, have a function you can hook into the Find_Package function. Larry: doesn't totally solve it, that solves the find problem, but the other problem is that the only thing you give the compiler is a list of directories to look in, but no way to tell it "don't get the OpenEXR headers from this directory", so if OpenEXR headers you don't want are in /usr/local/include yet something else you need is in there... JC: that's why package managers exist! Ideally want to all packages one way. Larry: yes, if you use all Rez or Spk in a controlled studio environment.
- Larry: will find the PR that first introduces the logic. If someone finds it useful, they are welcome to take it / provide improvements. First step on OpenImageIO, it is by no means a substitute for a real dependency management outside our builds, but meant to handle builds without presence of such a system. JF: eventually a CI Wiki article? Larry: yes, when it has settled down. Michael: getting your dependency report for every project would be fantastic.
- CI WG presentation to TAC
- Slide Deck
- WG renewed by unanimous vote of the TAC
- TAC Proposal for Language Interop WG
- Slide Deck
- "Replaces" Rust WG, extends scope to additional languages: Python, C, Swift
- Language specific packaging is in scope
- Opportunities to collaborate with CI WG?
- Congrats to SPK team who will be presenting at DigiPro!
- Larry: working hard on the documentation, we've been using it for a few years. We're past the point of whether the explanations make sense if you don't have all the background.
- JC: last time I looked at the documentation, it made sense, but I may not be the target audience given how much package management I do.
- OpenVDB GPU CI for NanoVDB
- Nvidia driver 535 installed on our GPU runners, supports CUDA 12.x
- 2024 aswf-docker images based on nvidia/cuda base image, includes CUDA 12.3 libs
- Does not appear to be in production yet
- Current nanovdb CI run spends 2m compiling, 1m running (non CUDA) test suite, may not be worth splitting compile vs test to only run test on GPU runners?
- Larry: OSL and OIIO are in the same position: building against CUDA, but not running yet.
- Larry: for OSL build times may be much longer. Is there an example of how to pass results from one builder to another? JC: some of the Python projects do it, upload a GitHub artefact, create a dependency from the build job to the test job, and download the artifact. I can provide an example. Larry: will probably prototype it. But not ready yet.
- JC: The sdist job produces an artifact and the wheel job downloads that artifact
- GHA ARM builders for Windows and Linux
- Do we have access yet?
- (slightly off topic): Anyone familiar with PyMel vs Maya 2025 issue? Anything we can do to help?
- Issue with Ubuntu 24.04 GHA runners killing the GHA daemon when doing a
apt-get update(JF's theory)- Did that turn out to be the case?
- TAC repo now using GitHub Sites / Jekyll rather than Netlify to publish documents
- Interesting tooling, worth documenting in CI Wiki?
- GitHub GHA Updates? (Andrew)
- Vote to take down ASWF JIRA instance passed
- Any remaining issues with transition to GH Enterprise account / paid runners?
- Any remaining issues with DCO bot?
- libtree
ldd-like tool to display DSO dependencies as a tree