Skip to content

Latest commit

 

History

History
109 lines (88 loc) · 9.13 KB

2024-08-14.md

File metadata and controls

109 lines (88 loc) · 9.13 KB
title parent
2024-08-14
Meetings

ASWF CI Working Group

Meeting: 14 August 2024

https://zoom-lfx.platform.linuxfoundation.org/meeting/97730566101?password=cb28b3b9-f744-46d0-ab69-d9f75f1b8668

Attendees

  • Jean-Francois Panisset, VES Technology Committee
  • Andrew Grimberg, LF RelEng
  • Stephen Mackenzie, NVIDIA / Rez
  • Jeff Bradley, DreamWorks Animation
  • Christina Tempelaar-Lietz, ILM
  • Ben Giles, Caligra
  • Larry Gritz, Sony Imageworks / OSL / OIIO

Apologies

New items

  • VFX/Animation Studio Linux Survey

    • Please fill before August 23rd
    • Stephen: feel free to cross post to Rez channel
    • JF to follow up with Nick

  • aswf-docker 2024.1 release (almost)

    • No progress since last meeting, will try to have this done before next one
      • Want to avoid having to build PySide for both Clang 16 and 17
      • Qt build missing QtWebEngine (missing dependencies)
        • Someone is working on this
        • Stephen: I thought QtWebEngine was deprecated in 6? JF: this was in contact of 6.5.3, so still seems to be there
      • Also some people getting caught by incomplete 2024.1 set images

  • Starting work on VFX Platform 2025

    • No real work since last meeting
    • OpenRV / xStudio / OpenFX / OpenAssetIO build containers
      • OpenRV work happening in 2024
    • Availability of Docker BuiltKit on Windows
    • ARM containers?
      • nvidia/cuda base image available for both amd64 and arm64 architectures
      • Stephen: most of the high performance processors are x86_64 still, but ARM processors getting traction in consumer markets. But are studios using them? Likely not desktop workstations.
      • Andrew: also no GPU-based ARM runners (yet)
      • Stephen: Windows on ARM machines are becoming more common, with the new Qualcom processor. Answer may change drastically in a year.
      • Christina: how would we currently do ARM builds? Install with scripts? JF: yes, until we container.
      • JF: NVIDIA Grace Hopper combined CPU / GPU is ARM on the CPU side. Stephen: NVIDIA cares a fair amount about ARM, there will be traction in the future.
      • Ben: NVIDIA Jetson AGX Origin 64Gb is a good platform for ARM + CUDA Caligra Ltd may be interested in driving that Stephen.
      • Stephen: "mini PC" form factor.
      • Ben: I would love to know which DCC tools are going to support ARM. We've given up for now. The Jetson is enough for, say, animation.

  • Meta app to test all ASWF libraries (Larry)

    • Test static vs dynamic linkage
    • Test CMake modules
    • Test dependent libraries
    • Look for symbol conflicts
    • Stand-alone project (independent of VFX platform)? Inside aswf-docker?
    • Larry on vacation this week: postpone discussion
    • Stephen: thought it would be interesting if the ASWF projects got together just enough to make a "reference application" like the PBRT ray tracer book. Could use OIIO to load a texture, OCIO to color manage it, OSL... Like an "ultimate CI test tool". But might need a lot of resources. "Shim togehter" enough of a renderer to be an "end to end aswf test". Of course some projects may not make sense to "link in", may have to stretch the definition.
    • Larry: haven't pursued it further. One of the chats had a discussion about a larger scale of integration testing. A weak part of many of the projects is that they do their own CI, run the tests within their build systems. None of these tests verify that something outside their build system can consume the build artifacts including the CMake configs. Would be good to do this in the context of the aswf-docker containers. We may build containers that have build of projects, but we don't know if ci-vfxall contains "compatible" builds. Maybe it would be interesting to make a shell project that pulls up the ci-vfxall container and makes a small app that links every project applicable. At least we know the versions we have in the containers play well together. That's as far as I've thought about it so far.
    • JF: USD build already pulls in several projects. Larry: USD builds depend on everything!
    • Christina: could go stale quickly, whereas USD is a "real world" application which might be a better test.
    • Larry: if you wanted to catch all the cases, would want the CI to also have a test case to build against top of tree of everything. A long, onerous build that might not fit in our resources. OIIO and OSL have one top of tree build of the most important dependencies to catch what the next release of those dependencies might break. There aren't enough CI minutes to build every dependency, tried to pick the ones that are most important, likely to break in the future, and that we have enough time to build.
    • Ben: How close is Omniverse to this test! Or O3DE? If it doesn't exist and its considered useful, we would do it. Larry: I don't know the answers.
    • JF: at least running the USD command line utilities as part of a test would be a first step.

  • Recurring issues with growing size of build containers and very limited disk space on free GHA runners

    • Any chance we might see runners with larger disk space or less pre-installed software (for in-container builds)?
    • Andrew: not aware of anything currently planned, but doesn't mean it won't happen. Monthly meeting is next week. I know that they have been investigating methods to allow GHA consumers to define the base image used by GHA. Could get rid of stuff you don't care about in the base image. So we provide a container that has the GHA agent in it, and they would run that. That would avoid all the "cruft" installed in their image. But don't know the status yet. Only alternative is to use the premium runners.
    • Andrew: lots of folks would be interested in being to provide their own build environment.
    • Stephen: how does limitations on container size work? What would something look like if instead of a container based way, what if we used Rez instead? It could all be done as a single container and then pull Rez containers. Would we hit some of those limits? How would we layer? Is there a logic to how the current containers are layered together? As opposed to a full build end to end? May have more to do how things are installed at the "system" level, whereas with Rez with build in a package path at a different prefix, with different packages in different prefixes.
    • Larry: projects that are "lower" in the dependency order, they save a lot of minutes consuming only the layers they need for their CI, saves time / space to download smaller container. A project like OpenEXR has pretty minimal dependencies, so the containers it pulls down are pretty fast. Those CI jobs go really fast, something higher up the hierarchy could take several minutes just to get the container before it can start running the test. So we appreciate that we have containers with only the dependencies for the project.
    • JF: a Rez plugin for Artifactory would allow using Rez packages as an alternative to Conan packages. Stephen: we have an epic on using cloud storage for Rez packages.
    • JF: would be great long term to be "packaging system independent". Stephen: yes that would be great.

  • Larry: at VFX Platform BoF, several people in the audience independently "name checked" the aswf-docker containers as very helpful, and people using those containers outside of CI to get things running. It was called out as something helpful we do.

  • Stephen: there is a GitHub issue to repro virtual environment for the upgrade to Conan 2. JF: can merge on top of 2024.1 and start working towards 2025. Stephen: some pip dependency is causing odd behavior with auth behavior for containers and breaks CI.

Follow Ups

  • NuGet support in Artifactory for xStudio builds
    • Did xStudio open a ticket?
    • Andrew: do not remember seeing a request for that.
    • JF to follow up with xStudio team
  • GitHub GHA Updates
    • ARM, latest distro runners documented in Wiki under GitHub Actions Runners
      • Andrew: haven't pull usage report lately, we've been under 50% for the last few months
      • VFX Platform 2025 deadline coming up
        • Projects racing towards their VFX Platform 2025 releases

Tools and Links