Skip to content

Latest commit

 

History

History
188 lines (101 loc) · 14 KB

2021-11-10.md

File metadata and controls

188 lines (101 loc) · 14 KB
parent title
Meetings
2021-11-10

ASWF CI Working Group

Meeting: 10 November 2021

https://zoom.us/j/757849640?pwd=QzE1K2hrL2FHSFhKK3h5Z3BWTFJsZz09

Attendees

  • Jean-Francois Panisset (VES Technology Committee)
  • Aloys Baillet (NVIDIA)
  • Larry Gritz (Sony Imageworks)
  • Sergio Rojas (Arena World)
  • Andrew Grimberg (LF RelEng)
  • Patrick Hodoul (OCIO / Autodesk)
  • Ryan Botriell (spk / ILM)
  • Trevor Thomson (Intel)
  • Jeff Bradley (DreamWorks)
  • Brent Villalobos (DreamWorks)

Apologies

New items

  • Conan builds in aswf-docker!

    • AcademySoftwareFoundation/aswf-docker#136 and follow on PRs

      • All the builds go through a release workflows, had some issues with GLEW and GLFW, got resolved by adding the Conan Central repository to get header only packages

      • TBB, Boost, clang, cppunit, Python...

      • Was going to add clang12 and clang13 packages to make them available as well via Conan, best place to start to use Conan packages, not there yet. Looks like clang12 and clang13 need Python 3 to build, creates a dependency between packages that needs to be sorted out.

      • packages/conan/recipes had a README and license files from Conan Central repository, no reason not to use these packages, except that the binaries aren't built on CentOS 7, so may not run fine in the environments we need. So made local copies to be able to add settings needed, Boost is a complex example of what's needed. "Settings" is like "variants" in Rez, creates a unique variant amongst the builds. Defines compiler, architecture...

      • Packages are defined based on VFX Platform years

      • Need documentation, and a good first use case. In order to use these packages, need Conan and the right settings, defined in a settings folder (packages/conan/settings). Can copy into your own Conan settings folder, the "remote" is important, points to Linux Foundation artifactory, you can create a personal testing repo, they expire after a few weeks if you don't use them, not sure what's the storage limit (a few GB). Settings file is important, defines all the possible variants for your build. Added the Python version (not in the default one), devtoolset version, VFX Platform version. If you try to use any of the recipes from a vanilla Conan install, it won't recognize those variants.

      • For now no strong incentive to use these packages except for curiosity

      • In the ci-common Docker image, there is now an install of Conan, will add the Conan executable. All the Docker images now have a (commented out for now) example of how you can install packages via Docker. Creates a text file (could use new support for heredoc in Docker) to list what Conan needs to install ("conan install ."), will install all dependencies into a folder.

      • Will keep these commented out for a while, once images are released, any maintainer in a repo that needs a specific version of clang / qt/ ... will be able to do something similar. Will be documented.

      • Goal for the next meeting is to have a solution to add clang versions to a build, supporting clang 12 and 13.

      • Need to update OpenVDB 9, latest OpenEXR patch, request for Google test

      • May make sense to go the "conan way".

      • Boost builds just went through a few minutes ago.

      • Qt recipes are building locally, but fail in GHA due to timeout

    • Work necessary to create Conan package for ASWF, ASWF adjacent or basic dependency packages

    • What projects could really use Conan package descriptions?

      • Designate volunteers to help with some packages?

      • Want to make sure it's usable first before we go into large scale effort

      • Some recipes are just copy / paste from the base version, usually the issue is that we maintain more versions than what Conan Central maintains, they typically only maintain one or two versions. For each package we use, need to make sure they work with VFX Platform 2019-2022, annoying issues working around CMake issues... Requires workarounds. Maybe we only need older versions for the base packages, and don't need to go back and build older versions of packages. Probably better to document what we have, and make sure added flexibility is used. If people want to contribute Windows version, that would be great. Will start playing with this in the next few months. On Windows some settings will look a bit different, not clear what they should look like yet. But would definitely welcome contributions for Windows builds. No TODO list yet of what packages are required.

      • Need for different versions of clang is a good use case. Once we have some users who find it useful, we can scale up the effort.

      • Conan Central has good documentation on how to build new packages, will add a link to that, will document the settings specific to ASWF recipes. Maybe at some point we'll be able to upstream those. The process to add new packages to the Conan Central index seems pretty open, so we may be able to upstream, be able to differentiate between ASWF ABI and generic (Ubuntu?) ABI of most Conan packages.

      • Does it make sense to limit to VFX Platform 2022 and newer? Aloys: wanted to make sure we can support a matrix of versions to make sure it works with newer VFX Platform versions in the future, wanted to make sure Conan could support multiple versions in parallel. Maybe did a bit too much work with lower level packages. But makes sense to only support 2022 and newer. Hard to predict what we should support, use cases will happen "organically". Once clang packages are done, hopefully it will clarify / solidify the workflows.

    • Create documentation on how to push to Conan and how projects can consume these artifacts

      • Do you need environment variables if you are just consuming packages? Aloys: not sure solution in place is the best. The requirements file picks up the Python file from the environment, so need the ASWF_PYTHON_VERSION environment variable. Need to figure out how Conan stores dependencies at build time so it knows what version of Python was used to build. Main remaining issue, dependency chain is queried from environment variables, would like to find a better solution. Boost settings files is currently valid for a range of Boost versions, with some special case code for specific Boost versions (implemented in Python, very "dynamic"). What we have is "good enough" for now, Docker environment contains the environment variables when Conan is running within environment we have prepared with aswf-docker images, but will have missing env var if using outside that environment. So we should be able to improve things, but no clear solution yet.

      • Interesting to figure out how this would interact with spk.

    • Preferred solution for Windows / macOS?

      • Will need a solution for environment variables, maybe a simple source script? Aswf-docker Python package wraps Conan. In order to build packages, aswf-docker build package_name and aswf-docker creates the right Docker image, with all the environment variables, then runs "conan create". So adding Windows support should be able to just setting environment variables, using "pip install aswf-docker", and then do "aswf-docker build", and it should just wrap the conan executable with the right environment variables. That would work for CI needs, setting environment variables isn't an issue for a CI environment. Slightly trickier for developers who just want to grab the binaries generated by ASWF, which is where we want to get rid of dependency on environment variables.

  • Using linguist in ASWF projects

    • "detect blob languages, ignore binary or vendored files, suppress generated files in diffs, and generate language breakdown graphs"

    • Added to aswf-docker together with Conan support

    • Use in other ASWF projects? Worth promoting / documenting?

    • Configured in gitattributes file, was tired of seeing auto-generated files in code reviews, one way is to .gitignore these files, but then they aren't part of the PR, so can have auto-generated files included in a PR, but yet hidden from reviews. Quite useful in this case, good to know that it exists, worth advertising, could be part of "best practices", don't know of many generated files. USD has a lot of generated files, the beginning of the file is auto-generated, but may have custom code at the end.

  • Need updates to aswf-sample-project

    • GHA support, deprecate Azure Pipelines stuff?

    • Test suite tool recommendations

    • Conan support?

  • Larger GHA build instances (request from OpenVDB)

    • Some can be addressed by adjusting build matrix, but may still be useful for large single builds

      • Andrew: GitHub library functionality is now available, two things we can do, composable actions, which is "macro-ing" Actions together, "an Action of Actions", as well as "public workflows", available not just to your organization, but also publicly.

      • Reusable workflows have to be defined with "on: workflow_call:" at the top, has to be part of the definition.

      • Composition / Composite Actions can be integrated as well, between these two, get the ability to template out a lot of things. LF RelEng team hasn't started using this yet, but a concept that was used heavily in Jenkins, composition of steps with templates.

    • Would aswf-docker also benefit?

      • Qt builds are timing out (more than 6 hrs to build on a standard 2 core machine, times out), clang takes a few hours to build as well. Would benefit any release, could have a white list of release tags that could be directed to more powerful hardware.

      • Aloys: Qt GHA build to publish to Conan will fail, can do the build locally, but don't have credentials to push to Conan, is there a workaround? Andrew: workaround would be to set up a AWS CodeBuild builder, which is what we're doing for GPU builds. Everything CodeBuild is a custom setup, has to be customized for the job, so we would need to setup something specific for the job. Gives access to pretty much any EC2 instance.

    • AWS CodeBuild vs upcoming paid GHA builders?

      • Andrew: No updates for now, monthly meeting with GH happens next week, hoping to get an update, should hopefully see it by end of year.

      • We should be able to pay for bigger instances / GPU acceleration when that becomes available with less configuration overhead. Aloys: happy to wait for this functionality.

      • Using GH-provided functionality is typically simpler.

  • Guidance for build diversity, GHA best practices

    • Provide reusable library of build configurations, progress on this GH functionality?

    • Review CI pipelines for ASWF (and ASWF adjacent?) projects?

    • In scope of CI WG to provide reusable workflows

  • CMake Guidance

    • List of links to CMake resources (wg-ci repo?)

    • Should CI WG take lead / invite someone to present on the topic?

    • CI WG to define base CMake version? YAML file that defines "CMake version per year"? Larry: not in the spirit of VFX Platform, but what can we assume about CMake version? But it's not hard to download CMake version of lots of platforms, so maybe threshold is low, less opinionated about the specific version, but would be good to have A choice. Trevor: anyone still using CMake 2? Larry: almost every project using a minimum of 3.12, when is it safe to use any of the newer features? Some nice features in newer versions.

    • Survey what ASWF projects are specifying if anything, and what they would like to move to if they had the chance.

    • Aloys: VFX 2022 is using recent Cmake for this year, had discussion with Dan Bailey, trying to lock version of CMake for each VFX Platform year to recent version. Larry: sounds reasonable. 2019 is about as far back as we want to go.

    • aswf-docker CMake versions per year

      • 2019: cmake-3.12.4

      • 2020: cmake-3.18.4

      • 2021: cmake-3.19.3

      • 2022: cmake-3.20.5

  • Making life better for distro maintainers / repackagers

    • Someone from Fedora presenting at next TAC

    • They use https://release-monitoring.org/ to monitor changes to packages, any other such services? Should we make sure all our projects are correctly registered by these projects? OpenEXR example: https://release-monitoring.org/project/13289/

    • Engage maintainers from other projects / package managers

    • Larry: have been using Homebrew on macOS for a long time, seemed easy, and have mostly current dependencies. But some people have a strong opinion against Homebrew since it installs in /usr/local as opposed to MacPorts, but MacPorts is far behind in versions of packages we use. Was trying to get them to update OpenEXR to 3.1, but needed to modify a lot of other MacPorts packages. Some Mac developers are very opinionated about MacPorts vs Homebrew. Does Homebrew has more mindshare / more vibrant project?

  • Anyone else using sidefx-web-cli project for downloading SideFX releases?

    • SideFX API returns 200 even for errors, and code then errors out on non-JSON output

Follow Ups

  • Progress on LF code signing infrastructure

  • Any projects need help with master to main branch renaming?

    • OSL just did it this week

    • OCIO did it also