Support various array types in extensions

[gdalle]

Fix #766 by adding an extension for StaticArrays.jl and a custom dispatch for ktypeof(::StaticVector).

The extension is loaded conditionally with Requires.jl for Julia < 1.9.
This means that Requires comes as a new dependency.

EDIT: also fix #769

[gdalle]

I did the same thing for FillArrays, which allowed me to remove some lines in krylov_utils.jl. I also moved from Requires to PackageExtensionTools, which further cuts down the additional code.

If you disregard the new tests (new tests are never a waste of LOCs), I think you'll find this PR hits a very reasonable tradeoff in complexity :)

[gdalle]

Here are some benchmarks, obtained with the following commands:

shell> rm -rf ~/.julia/compiled; rm -rf ~/.julia/packages;

julia> ENV["JULIA_PKG_PRECOMPILE_AUTO"]=0;

julia> using Pkg

julia> Pkg.activate(temp=true)

julia> Pkg.add(url="https://github.com/gdalle/Krylov.jl", rev="staticarrays_extension")
# julia> Pkg.add(url="https://github.com/JuliaSmoothOptimizers/Krylov.jl", rev="main")

julia> Pkg.precompile("Krylov", timing=true)
# julia> @time Pkg.precompile("Krylov") # on < 1.9

julia> @time_imports using Krylov
# julia> @time using Krylov # on < 1.8

shell> rm -rf ~/.julia/compiled; rm -rf ~/.julia/packages;

julia> @show VERSION

Julia 1.9

This branch:

julia> Pkg.precompile("Krylov", timing=true)
Precompiling Krylov
    900.1 ms  ✓ PackageExtensionCompat
   7800.2 ms  ✓ Krylov
  2 dependencies successfully precompiled in 9 seconds

julia> @time_imports using Krylov
      0.9 ms  PackageExtensionCompat
    246.2 ms  Krylov

Main branch:

julia> Pkg.precompile("Krylov", timing=true)
Precompiling Krylov
   7111.4 ms  ✓ Krylov
  1 dependency successfully precompiled in 7 seconds

julia> @time_imports using Krylov
    224.4 ms  Krylov

Julia 1.8

This branch:

julia> @time Pkg.precompile("Krylov") # on < 1.9
Precompiling project...
  3 dependencies successfully precompiled in 9 seconds
  8.552460 seconds (98.20 k allocations: 5.775 MiB, 1.46% compilation time)

julia> @time_imports using Krylov
      0.5 ms  Requires
      0.2 ms  PackageExtensionCompat
     84.1 ms  Krylov 32.45% compilation time (8% recompilation)

Main branch:

julia> @time Pkg.precompile("Krylov") # on < 1.9
Precompiling project...
  1 dependency successfully precompiled in 7 seconds
  7.477803 seconds (67.81 k allocations: 3.921 MiB, 1.42% gc time, 2.57% compilation time)

julia> @time_imports using Krylov
     45.2 ms  Krylov

So essentially the performance hit seems negligible, except on Julia < 1.9 for the load time. And I still believe it is worth exploring this as a general paradigm, because that's how it will work in the future of Julia.

[gdalle]

@amontoison I think this is ready, if you want to approve the workflows

[gdalle]

Found this old PR while cleaning things up, is it still relevant @amontoison ?

[amontoison]

Hi @gdalle!
Do you have some array types that are not yet supported?
Otherwise, I propose to wait the next LTS version of Julia.

[oscardssmith]

I really think this approach is the right way to go. If we want to avoid the overhead for Julia <1.9, we could continue to use the .name.name approach and not load the extension with Requires at all for older julia versions. Especially since #768 didn't actually fix ComponentArrays (see #769 (comment)) I think this PR is the better way forward.

[amontoison]

@oscardssmith Is your use case with ComponentArrays?
I wanted to wait the next LTS release and add extensions for all relevant arrays types.

[oscardssmith]

Yes.
Why do you want to wait? I can understand not wanting to support them on versions <1.9, but there's no rule that says that all features need to be supported on all julia versions. Surely supporting ComponentArrays on Julia 1.9+ now is better than not supporting them on any julia version.

[amontoison]

Yes. Why do you want to wait? I can understand not wanting to support them on versions <1.9, but there's no rule that says that all features need to be supported on all julia versions. Surely supporting ComponentArrays on Julia 1.9+ now is better than not supporting them on any julia version.

I wanted to keep support for the LTS version but not using Requires, only package extensions.

[oscardssmith]

What's wrong with using Requires on LTS?

[amontoison]

@oscardssmith I opened #859 to fix the issue with ComponentArrays.

[oscardssmith]

Thanks! I do really think that pkg extension/requires is a better fix, but #859 is definitely better than status quo.

[gdalle]

You can also have package extensions that are just not loaded on <1.9, no need for Requires in that case

[amontoison]

Thanks! I do really think that pkg extension/requires is a better fix, but #859 is definitely better than status quo.

I agree with both of you Oscar and Guillaume.

You can also have package extensions that are just not loaded on <1.9, no need for Requires in that case

Oh, good to know!

I plan to do a release 1.0.0 of Krylov.jl this year, let's drop Julia 1.6 and requires at least Julia 1.9 and add package extensions at this moment. Is it fine for you?
The new LTS might not happen for a while.

[gdalle]

Now that 1.10 is the LTS, it might be time to revisit this

[amontoison]

Is it still relevant after #947?

[gdalle]

Thanks for taking steps towards this fix!
Unfortunately, judging by #769 (comment), the solution requires type piracy from the user?
If I am not the developer of either ComponentArrays or Krylov, then doing this is considered bad practice:

Krylov.ktypeof(v::ComponentArray) = typeof(v)

Of course the impacts are likely to be limited, but that is one of the reasons why I recommended package extensions instead. What do you think?

[amontoison]

Thanks for taking steps towards this fix! Unfortunately, judging by #769 (comment), the solution requires type piracy from the user? If I am not the developer of either ComponentArrays or Krylov, then doing this is considered bad practice:

Krylov.ktypeof(v::ComponentArray) = typeof(v)

Of course the impacts are likely to be limited, but that is one of the reasons why I recommended package extensions instead. What do you think?

@gdalle
We don't need any type piracy if I remove this line in Krylov.jl:
https://github.com/JuliaSmoothOptimizers/Krylov.jl/blob/main/src/krylov_utils.jl#L209

Krylov.ktypeof(v::ComponentArray) = typeof(v)

was just used to restore the default behavior.

I am just wondering what is the most relevant for the user. Keep some componentVector in the workspace or just Vector?

[gdalle]

My point from earlier still applies: checking sometype.name.name as in the line you mention is a bit of a hack and relies on internals. The clean way to do this is with a package extension, which also allows for proper versioning of the dependencies, etc. This is even more true now that 1.10 is the LTS.

I am just wondering what is the most relevant for the user. Keep some componentVector in the workspace or just Vector?

Does the user interact with the Krylov workspace? As long as the solution is a ComponentArray like the inputs, I don't think it is crucial what kind of data you use internally?

[gdalle]

On second thought it would probably be best to keep everything the same type as the input, eg for GPUs and StaticArrays and such