Skip to content

Latest commit

 

History

History
143 lines (126 loc) · 9.86 KB

2024-07-17.md

File metadata and controls

143 lines (126 loc) · 9.86 KB
title parent
2024-07-17
Meetings

ASWF CI Working Group

Meeting: 17 July 2024

https://zoom-lfx.platform.linuxfoundation.org/meeting/97730566101?password=cb28b3b9-f744-46d0-ab69-d9f75f1b8668

Attendees

  • Jean-Francois Panisset, VES Technology Committee
  • Andrew Grimberg, LF RelEng
  • Jeff Bradley, DreamWorks Animation
  • Ibrahim Sani Kache, DreamWorks Animation
  • Jean-Christophe Morin, Rez

Apologies

  • Larry Gritz, Sony Imageworks / OIIO / OSL

New items

  • ASWF Confluence instance moving to hosted version

    • Presentation to TAC
    • Overhead and cost of maintaining self hosted version (recent issues requiring re-indexing)
    • Plugins may be different between self hosted and cloud version
    • Will require Atlassian ID (cannot use LF ID for SSO)
    • Current transition slowed down by large amounts of unindex spam, required cleaning up the database directly. There may still be 3 million pages in a Space which only has 2 pages.
    • Looking for alternate approach of how to locate these missing pages.
    • TAC voted to accept this transition
    • Will do a test transition, migration tools provided by Atlassian report that it will be a 13 hour job for test migration. Just took a backup of the space that will cause the most problems, it will take 20 minutes do the backup, so migration may be faster. Still a lot of data to be moved. Don't want to take the system offline for 13 offline. All our other test migrations we can do in the middle of the day, take 10-30 minutes.
    • JC: are there large assets uploaded? Andrew: tools says it will move 3.5m pages, vast majority against a space that only has 2 "real" pages, don't know where all these pages come from. Data sums up to only 300MB, attachments are a drop in the bucket compared to database records. Backup is 250MB, DB records is a 5GB XML file.
    • Andrew: trying to get XML file opened in vim.
    • Re-indexing is finished (took 5.5 days), finally reopened the system yesterday, haven't seen major stability issues, seems more stable than it was. Will get migrated to the cloud as soon as we can.

  • aswf-docker 2024.1 release (almost)

    • All containers released except ci-vfxall:clang-17.1
      • PySide Conan package builds with Clang
      • Don't really want to build both Clang 16 and 17 versions
      • Can you mix? Can we build PySide with gcc instead?
        • JC: it can be tricky to mix Clang 16 vs 17, should be OK in most cases, but usually want to compile with a single compiler. Can be tricky. Can try it and see if it works. But if it doesn't work, it will crash at runtime rather than at build time. PySide / Shiboken might qualify as "tricky"? JC: if libc++ changes, it gets tricky. Shiboken creates C bindings? JF: might be simplest to try to build it with gcc. JC: are you already mixing compilers? JF: there is already mixing.
        • Ibrahim: we do mix and don't run into problems, we build OpenVDB with both gcc and clang, and our client packages may pick up either version.
        • JC: in most cases it should work, but there may be corner cases. Linux distros don't mix and match, either all gcc or not. At Anaconda, all the packages we build with the same compiler, in the past they've hit corner cases. ABI corner cases.
    • ChangeLog
      • blosc moved to base1
      • rework Conan packages to use CMakeToolChain, conandata.yml (in progress)
      • Conan packages mostly install to lib64 instead of lib (partial #120)
      • libdeflate in base image for OpenEXR
      • oiio build container (#173)
      • Conan packages preserve DSO symlinks when installed (#194)
      • Conan packages don't overwrite each other's installed license files
      • Break circular dependency between OCIO and OIIO by compiling OCIO utils without OIIO (#54)
      • OpenEXR and Imath built as Conan packages only
      • MaterialX 1.38.10 (was 1.38.8)
      • Imath 3.1.11 (was 3.1.10)
      • OpenEXR 3.2.4 (was 3.2.2)
      • OpenImageIO 2.5.12.0 (was 2.5.8.0)
      • OpenShadingLanguage 1.13.10.0 (was 1.13.6.1)
      • OpenTimelineIO 0.17.0 (was 0.15)
      • Conan 1.63 (was 1.62)
      • CMake 3.27.9 (was 3.27.8)
      • Clang 16.0.6 (was 16.0.4), 17.0.6 (was 17.0.1)
      • pybind11 2.12.0 (was 2.11.1)
      • Python 3.11.9 (was 3.11.8)
      • USD 24.05 (was 23.11)

  • Starting work on VFX Platform 2025

    • Anyone has experience moving from TBB to oneTBB? Will ASWF projects be ready?
      • Jeff: haven't done any work on oneTBB yet, so no experience to share yet. Curious if anyone else has tried.
      • JF: will likely need to continue including "traditional" TBB
    • oneMKL?
      • Not currently used by ASWF packages, but part of VFX Platform
      • JC: are we allowed? For Anaconda, we had an agreement with Intel, don't know current status. JF: good point, will need to check. Maybe why Aloys didn't include it in the first place.
      • oneMKL License

      • Can I redistribute oneMKL? Answer: Yes, redistribution is allowed per the terms of the ISSL.

      • Yes, you can include oneMKL in a public container.

      • JC: oneMKL may be open source, but the binaries are provided by Intel (Intel MKL)
    • OpenRV / xStudio / OpenFX / OpenAssetIO build containers
    • Continue modernizing Conan packages
      • Potentially move to Conan 2
    • New Conan "base" packages?
      • pugixml
      • libtiff / libpng instead of system installed versions?
    • Restore testing infrastructure
    • Better capture dependencies between packages / better automated full builds
    • Any other ASWF-adjacent projects which should be included?
    • Update Python infrastructure / dependencies PR 198

  • Meta app to test all ASWF libraries (Larry)

    • Test static vs dynamic linkage
    • Test CMake modules
    • Test dependent libraries
    • Look for symbol conflicts
    • Stand-alone project (independent of VFX platform)? Inside aswf-docker?
    • Larry on vacation this week: postpone discussion

  • Conventional Commits and automated ChangeLog generation

    • Conventional Commits Spec 1.0.0
    • git-cliff changelog generator
    • OIIO / OSL experience shared in Slack / Wiki
    • JC: may be too much for open source projects in general? Works if the contributors are maintainers, but it's not user friendly for outside contributors. If I need to follow a specific commit naming convention.
    • Andrew: RelEng enforces Conventional Commits for our internal repositories. Forcing a team to start adopting it is painful for the first 3 weeks, then commit messages get very good.
    • Ibrahim: I agree
    • Andrew: also good for automated tooling, including GHA tooling.
    • JF: always some manual editing required? Andrew: yes, on personal projects I generate ChangeLogs from Conventional Commits / labels, but still end up having to write "stuff".
    • JC: I think it works internally, without a lot of external contributors, but wouldn't force this on Rez contributors for instance, when people are just trying to "make things better". It turns me off when I create a PR for an open source project and there are strenuous requirements on formatting and naming.
    • Andrew: should have a contribution guide in your repo where you can detail preferences. We hard enforce Conventional Commits in CI, engineers can't even make local commits. JC: not everyone reads this, unless you put it in PR templates. Andrew: put it in pre-commit hooks.

Follow Ups

  • NuGet support in Artifactory for xStudio builds
    • Anything required on RelEng side, or "should just work"?
    • xStudio repo should inherit ARTIFACTORY_TOKEN from org-level? Andrew: yes, they should inherit it. JF: aswf-docker gets it this way.
    • Andrew: not aware of any projects using NuGet packages with Artifactory
    • JF: should xStudio open a ticket? Andrew: yes
  • GitHub GHA Updates? (Andrew)
    • Access to GHA ARM builders for Windows and Linux?
    • Andrew: I believe these have been enabled.
    • JC: are they GA? Andrew: I believe they are, enabled at Enterprise account level.
    • Andrew: thought I had them created (since they are still beta), but don't seem them.
    • Andrew: they are all premium runners
    • Andrew: can you open a ticket? JF: will do
    • LF ticket IT-26991
  • PyMel vs Maya 2025 issue?
  • CI builds failing in CentOS 7 based containers
    • Default checkout moving to NodeJS 20 which requires newer glibc
    • Environment variables can be used for now to revert to NodeJS 16
    • How long can CentOS 7 / VFX 2022 builds be supported?
    • Wiki Documentation
    • Good example of what WG CI can do
    • JC: not the first time we've had issues with glibc compatibility, have had issues with setuppython before. Not a great experience. Can that feedback be passed along? Andrew: can pass that along. JC: there's not much you can do when the actions are no longer compatible, so at the mercy.
    • Ibrahim: ran into this problem this week, ended up using the env var this week. This was for internal projects using GitHub Actions

Tools and Links