Skip to content

Latest commit

 

History

History
113 lines (102 loc) · 13.2 KB

2022-05-25.md

File metadata and controls

113 lines (102 loc) · 13.2 KB
title parent
2022-05-25
Meetings

ASWF CI Working Group

Meeting: 25 May 2022

https://zoom.us/j/757849640?pwd=QzE1K2hrL2FHSFhKK3h5Z3BWTFJsZz09

Attendees

  • Jean-Francois Panisset (VES Technology Committee)
  • Andrew Grimberg (LF RelEng)
  • Aloys Baillet (NVIDIA)
  • Jean-Christophe Morin
  • Larry Gritz (Sony Imageworks)

Apologies

New items

  • USD Working Group overlap: that meeting is every 2 weeks, we may be losing attendance to that. JF to reach out to USD WG to figure out their scheduling plans going forward.
  • Impact of VFX Platform 2023 on ASWF projects and CI
    • glibc bumped from 2.17 to 2.28, effectively EOLs CentOS 7
      • What's new in glibc? Microsoft Teams and VS Code may no longer support glibc 2.17, same as Chrome (actually that was a libstdc++). There are some modern UI platforms that require a newer glibc. There are new C APIs.
      • Get this by default when compiling in a RHEL 8 / Rocky 8 environment.
    • New C++ ABI: Since gcc 5.1, libstdc++ introduced a new library ABI that includes new implementations of std::string and std::list. In order to maintain backwards compatibility the old implementations are still supported in parallel with the new ones. The choice of implementation is made using the _GLIBCXX_USE_CXX11_ABI macro, and the VFX Reference Platform up to CY2022 used the older option so the compiler setting for those Platforms should be _GLIBCXX_USE_CXX11_ABI=0, although that setting may not be supported by some older Linux distros. From CY2023, the Platform has moved to the newer implementation since all the major Linux distributions have made the transition. This is a major change for CY2023 that may require a rebuild of all software against the new ABI for those moving from earlier Platforms.
      • You get the new ABI by default when compiling in a RHEL8 / Rocky 8 environment by default.
      • libstdc++ on the system will be compiled with the new ABI, would have to ship your own libstdc++ if you don't compile to the new ABI.
    • What are specific tasks that we can tackle as a group to help with the work required to produce CY2023 containers?
      • Can it be done as partial PRs, or does it have to be one big PR?
      • Aloys: want to finish updates to CY2022, but will start soon. Create a new version of the common layer for RHEL8 should be fairly standard.
      • Base image is available as a Rocky 8 based image from NVIDIA, not in the same group. But NVIDIA says CentOS 7 and 8 no longer supported, it's Rocky 8.
      • Should be fairly trivial to update to Rocky Linux 8 as the base image for version 3 of the common layer, see what breaks. Find the yum packages that don't work anymore.
      • First challenge is to find Dev Toolset, was always a challenge on CentOS 7, if someone know how to find this on Rocky 8, that would be helpful. JF: part of base Rocky repos. JC: seems to be readily available.
      • Aloys: if these stars align, shouldn't be too complicated.
      • Ref Platform says gcc 11.2.1, so DevToolset 11 should do.
      • DTS now called gcc-toolset, version 9, 10 and 11 are available in the AppStream repository: Rocky Linux 8.6 AppStream Repo Mirror
      • JF to repost to Slack what was posted before on that topic.
      • Aloys: anyone can clone the aswf-docker repo, create a version 3 and try to build it. Tried to document everything, following my own documentation for the 2022 updates. May be some implicit knowledge that isn't documented. So would be good if someone else tried it. JC: will try to do this.
      • Have been running 2022 update on Windows update via WSL2, working quite well.
    • Status of Rocky 8 based NVIDIA base container images (is NVIDIA moving to Gitlab?): NVIDIA Rocky 8 Base Docker Container
      • Images pushed to Docker Hub from Gitlab. There might be a read-only mirror on GitHub.
      • Building on Rocky itself, also Ubuntu.
    • Don't hard key on /etc/redhat-release containing "Rocky", could be RHEL, Alma...
    • Is Conan support affected?
      • Based on Python, so supports RHEL 8 and derivatives
      • Not clear if Conan can support multiple build versions? But presentation from Foundry would seem to indicate you can build different variants (compilers, compiler flags...), so should be possible (JC)
      • Aloys: Conan uses a settings file that has all the different versions of everything, when you build a package you tell Conan which of these settings are important for that package, stores a list of key/value pairs, if it doesn't find a match, it will propose to build a version (including upstream versions). Have to tell it with settings are important, that's how Conan differentiates between build variants.
      • Reproducible builds: a big thing for supply chains. Andrew: the Eclipse project wrote a Maven plugin to get reproducible builds, they can rebuild anything, and outside of timestamps, they can reproduce any build.
    • Should we "freeze" the base image for the CY, or publish updates corresponding to underlying distro updates? or is that counter productive?
      • RHEL 8 on a 6 month release cycle?
      • Our policy should be documented. Aloys: from minor version to the next, can't run anything anymore, something like libtiff can change, one of the hundreds of packages we implicitly rely on, so if you build on CentOS 7.9, it might not run on CentOS 7.8.
      • JC: it will work from lower to newer version.
      • JF: difference between using the containers just to compile, or also using it as runtime.
      • JC: issue if you statically link anything.
      • Aloys: what we've done so far is use the most recent version available at the beginning of the year when we build the images, and stick with that, and don't update to newer versions. We may have updated mid year because of rebuilds. Once the common version is released, try to leave it alone. Should be OK to state that.
    • Should we also try to have RHEL9 derived containers now that it has been released? Could we use the free developer license to have RHEL-based containers, or does that prevent the use of NVIDIA base images?
      • Experimental versions?
    • macOS base version moved to 11 / Big Sur
    • Visual Studio base version moved to 2022
  • Updates to 2022 containers: AcademySoftwareFoundation/aswf-docker#152
    • USD 22.05
    • OpenEXR 3.1.5, all the latest CY2022 compliant ASWF releases (OSL,imath, OTIO, OCIO...)
    • MaterialX
    • Python 3.9.11, Conan 1.47.0
    • Clang 14
    • New CUDA version 11.4
    • Based on latest (final?) CentOS 7.9 base image
    • Added a "os test" to test the runtime environment on GHA (testing for 7.9). Nothing else assumes the OS after that, the packages are keyed on the Linux version.
    • Haven't looked at Black issue, may be locked somewhere
    • In the process of releasing the aswf-testing versions, until that's released, the PR will be failing. Should possibly mark the PR as draft. Will keep fixing small things before release.
    • Larry: did see the alert, will take a look at it. Aloys: would be good if you could validate that the versions being used are sensible. Once the images are available, will finalize the Changelog with new image versions. There will be a Clang 14 for you to try.
    • Larry: CUDA is good at being forward and backward compatible within a major version. Aloys: not worth adding CUDA to VFX Platform. Older versions become no longer available. Also puts a limit on the NVIDIA driver version, now it's a bit easier, some dependencies between CUDA driver. Also sometimes end-of-lifes GPU generations.
  • The perils of dependabot: it tried to update pyjwt to 2.4.0, but Conan doesn't want 2.0.x. Caught by CI: https://github.com/AcademySoftwareFoundation/aswf-docker/runs/6586112729?check_suite_focus=true#step:5:403
    What are good strategies to use dependabot without getting broken by it? How do you remember to undecline an update once your other dependencies no longer preclude it?
    • Aloys: ASWF Python dependencies are dependencies in setup.py, but the source of truth isn't in there, it's in a pip depends file. Dependabot may not know about pip files, so not useful in this repo since we use pipenv.
    • JC: there are ways to manage dependabot notifications, and it tells you about it. So you can ignore minor versions, patch versions... There are a few different commands you can send it. But if you tell it to ignore a dependency, you have to "remember" to tell it to notify you again in the future, may have to keep the dependabot PR open.
    • Aloys: may try to update to latest of everything, lots of dependencies on Python side, some lock things up, haven't tried to run an update recently. Will end up with a new pipfile.lock that might have the right versions. The dependabot mechanism is helpful to know about new versions, just leave the PR open until run a pipfile update.

Tools

Follow Ups

  • Updates on GHA custom / for pay instances (Andrew)
    • Had meeting with GitHub last week, still coming, but no ETA
    • Also no update on M1 silicon