Skip to content

Latest commit

 

History

History
148 lines (136 loc) · 10.7 KB

2023-12-06.md

File metadata and controls

148 lines (136 loc) · 10.7 KB
title parent
2023-12-06
Meetings

ASWF CI Working Group

Meeting: 06 December 2023

https://zoom-lfx.platform.linuxfoundation.org/meeting/97730566101

Attendees

  • Jean-Francois Panisset, VES Technology Committee
  • Kerby Geffrard, OpenRV
  • Andrew Grimberg, LF Release Engineering
  • Aloys Baillet, NVIDIA
  • Jeff Bradley, Dreamworks
  • Jean-Christophe Morin, Rez

Apologies

New items

  • aswf-docker updates
    • 2023.2 released
      • Local Conan builds mostly fixed, leverages BuildKit caching
      • OpenEXR and OpenVDB containers now include PyBind11
      • Regression with GLeW CMake not yet addressed for OpenVDB
      • Will address Python comments in 2024
    • 2024 progress
      • New stuff
        • VFX Platform 2024
          • Python 3.10.11 -> 3.11.6
          • Qt 5.15.9 -> 6.5.3
          • PySide 5.5.19 -> 6.5.3
          • NumPy 1.23.5 -> 1.24.3
          • OpenEXR 3.1.11 -> 3.2.1
          • OpenSubDiv 3_5_0 -> 3_6_0
          • OpenVDB 10.0.1 -> 11.0.0
          • Alembic 1.8.5 -> 1.8.6
          • OpenColorIO 2,2,1 -> 2.3.0
          • Boost 1.80.0 -> 1.82.0
        • Non VFX Platform
          • clang 16.0.4
          • Conan 1.58.0 -> 1.62.0
          • CUDA 11.8.0 -> 12.3.0
          • GLVND 1.6.0 -> 1.7.0
          • CCache 4.7.4 -> 4.8.3
          • Sonar Scanner 4.8.0.2856 -> 5..0.1.3006
          • blosc 1.17.0 -> 1.21.5
          • cmake 3.27.2 -> 3.27.8
          • glfw 3.1.0 -> 3.3.8
          • gtest 1.11.0 -> 1.14.0
          • log4cplus 1.1.2 -> 2.1.1
          • materialx 1.38.7 -> 1.38.8
          • oiio 2.4.13.0 -> 2.5.5.0
          • osl 1.2.13.0 -> 1.12.14.0
          • pybind11 2.9.2 -> 2.11.1
          • usd 23.05 -> 23.11
      • Current Status
        • Made it past building Qt6 by disabling test suite
        • Working on building PySide 6
        • Changes may no longer support building previous years
      • Blockers
        • Optix 8 header download URL: anyone can help at NVIDIA?
          • Aloys: can try to reach internally
        • Conan package tests: Pybind11, Qt
          • Aloys: not sure if Conan 2 helps
          • Should be possible to generate CMake files that don't depend on Conan, it may be a choice that some packages take to not be CMake compliant, and require Conan awareness.
          • Normal CMake generator are pretty generic. There's been quite a few versions, not sure what's the latest recommendation for CMake generation. Different ways of generating CMake files, and CMake itself has gone through several iterations.
        • Incomplete CMake exports from Conan packages (GLew)
      • Nice to have
        • OIIO, OpenFX, OpenRV containers
        • Blosc, Alembic as Conan packages
      • Looking Forward
        • Can we consume existing binary builds directly from Conan Center
          • Or do we want to guarantee everything compiled with VFX Platform compilers?
          • Kerby: for Qt, the object model is mostly C / their own Qt object model, so less sensitive to compiler versions.
          • PySide is Shiboken that parses Qt headers to be a thin layer, a copy of the Qt shared libraries. Should not be an issue.
          • No PySide standard Conan recipe. Hard to provide PySide binaries. ADSK has some internal artifacts for PySide, tied to Python version. May not make sense to have a Conan recipe for that.
          • It alters your Python, installs something inside your Python folder.
          • JC: PySide isn't special, it's a normal Python package / CPython extension. Makes sense in the context of the Python ecosystem, can install with pip install. Meant to go into site-packages.
          • Kerby: there are Python Conan recipes in the Foundry recipes.
        • Slimmer Docker images
          • Demonstrate how a project could pull a generic base image and install Conan packages on demand?
          • Tradeoffs in startup time vs being able to iterate faster / not having to release full set of Docker images
        • Issue with LF Artifactory
          • Do we need a commercial instance to host binary components?
          • Andrew: as long as we don't "push" availability of these binaries shouldn't be a problem. Our usage is a drop in the bucket compared to other LF users, if we are using in our build process, we should be OK. We just have to be careful about how we advertise it.
          • Andrew: if there are concerns from the vendor, we will let us know. But it has to be open source, can't be closed source. JF: only CUDA is closed source, and that just goes to Docker Hub.
          • JC: of course Qt is borderline.
          • Aloys: do we need to provide the source? Andrew: as long as we don't make modifications to the sources. Andrew: if provide binaries with modifications to the base source, we have to provide those modifications (which we do: Conan patches directory). With the Linux kernel, you can get it from a number of places, if your build says 'it was from here', you don't have to provide that source. JF: part of our build scripts or Conan recipes.
  • OpenSFF Badging Requirements
    • TAC discussions on the requirement for Gold level for (new) projects to reach Accepted Stage
    • OpenEXR, OpenColorIO, MaterialX, OpenVDB, OpenImageIO, OpenCue, OSL, OpenAssetIO have achieved Passing
    • OTIO is close to Passing, can we help them get there?
    • No project has reached Silver, MaterialX and OpenEXR are closest
    • Current project status
    • Should CI WG present an opinion to the TAC?
    • JC: the TAC is mainly composed by the TSCs, some of them have frustrations with respect with the badges, we shouldn't care about these things. They may not know about security, they are trying to push away the important points. If anything, when we had discussion about a Security WG, it was decided by the TAC that it wasn't required since there would be overlap with CI WG. Also need to support the projects. So we need to figure out how projects can make progress, including the harder points.
    • JF: has Rez started the project? JC: not yet. Currently the only one actively contributing, and will be quite busy in the next couple of months, but will start the process. When we had a look the last time, most of the requirements are pretty staightforward. A lot of them are easy, and important. You cannot have the Silver badge before the Passing badge, but in practice, you can address some Silver issues before Passing issues.
    • JF: could we help OTIO? JC: we could provide the Security reporting Markdown file to have a consistent reporting infrastructure between projects.
    • Andrew: some of these can be generated as templates at the ASWF GitHub organization level, we could put these files in the template, so as long as the repository is in the organization, they will get. Read the changes in the PR, for now they are still requiring Gold, I believe that Silver should be sufficient for the Accepted stage, with statement of "Moving towards Gold" in the first year, and "Achieved Gold" by the second year.
    • JC: not sure if that will be liked, but I agree this is how it should be. Security shouldn't be an option. If I was part of a member company, I would strongly require this level of security.
    • JF: example of BIOS vulnerability in BMP / PNG parser
    • JC: a lot of people may not have worked outside of VFX / Animation, yet our projects can be used outside the industry. Projects can be incorporated in browsers, web services... There may be reluctance to run fuzzers, "we know what our inputs are", which is true internally, but not for outside use cases.
    • JF: OpenEXR seems to be the most aware, part of OSes, use fuzzing
    • JC: even in corporate environments, it's hard to get access to a security expert. I was lucky to follow a training, they teach us that you have to do the work yourself, you have to read about security and educate yourself.
    • JF: will try write a paragraph as our position, we can discuss on Slack, and will present to TAC.

Follow Ups

  • GHA GPU runners now in paid beta
  • Andrew: GHA Team has asked if we wanted to have a meeting to ask questions. 30 minutes / 1 hour to go over upcoming GitHub functionality / issues we are running into.
    • Scheduling
    • Come up with an agenda ahead of time
    • JF: will bring up on Slack channel
    • JF: could try to use meeting slot / 30 minutes
    • Andrew: questions would determine who would come, for instance the person in charge of GPUs and ARM runners
    • JC: there was an apt cache issue? Andrew: they had died on the backend, we deleted and recreated them and they started working again.
    • Andrew: they are all working now, including the Windows one. It was a bug on the GitHub side that failed to show that they weren't working
    • Andrew: I've given the info as to what's in those images, hopefully enough to run our workloads. Linux image is published by NVIDIA, Windows one is published by Microsoft.
    • JF: need to publicize those to the various projects, perhaps work with OCIO to transition to those runners.

Tools and Links