Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to prevent flake devShell dependencies from being garbage collected? #2249

Open
jameswhqi opened this issue Sep 16, 2024 · 4 comments
Open

Comments

@jameswhqi
Copy link

After reading this, this and this, I still can't figure out how to prevent flake devShell dependencies from being garbage collected by nix store gc. M(N)WE:

flake.nix

{
  description = "test";

  inputs = {
    flake-parts.url = "github:hercules-ci/flake-parts";
    haskellNix.url = "github:input-output-hk/haskell.nix";
    nixpkgs.follows = "haskellNix/nixpkgs-unstable";
  };

  outputs = inputs@{ flake-parts, nixpkgs, haskellNix, ... }:
    flake-parts.lib.mkFlake { inherit inputs; } {
      systems = [ "x86_64-linux" "aarch64-linux" "x86_64-darwin" "aarch64-darwin" ];
      perSystem = { self', system, ... }:
        let
          pkgs = import nixpkgs {
            inherit system;
            overlays = [ haskellNix.overlay ];
            inherit (haskellNix) config;
          };

          proj = pkgs.haskell-nix.project {
            src = ./.;
            compiler-nix-name = "ghc966";
          };
        in
        {
          devShells.default = proj.shellFor {
            withHoogle = false;
            buildInputs = [ proj.roots ];
          };
        };
    };
}

test.cabal

cabal-version: 1.12

name: test
version: 1.0.0
build-type: Simple
$ nix develop . --profile nixenv
$ nix store gc
129 store paths deleted, 1551.01 MiB freed

BTW if the proj.roots line is removed, the garbage is even larger:

188 store paths deleted, 3325.91 MiB freed

Usually this isn't a huge problem because only the flake inputs are GC'd (mostly nixpkgs), but with haskell.nix the collected garbage is much larger and seems not limited to flake inputs (I might be wrong). Can someone provide a minimal example where nothing is GC'd?

@cydparser
Copy link
Contributor

In addition to proj.roots, adding proj.plan-nix to buildInputs (or proj.stack-nix for Stack projects) will prevent most GC.

One can also prevent the flake sources from being GC'd:

          # Add a let binding below `pkgs`
          inherit (pkgs) lib;

          inputOutPaths =   inputs:
            lib.lists.unique (lib.lists.flatten (lib.lists.map (input:
                  if lib.attrsets.hasAttr "inputs" input then
                    [ input.outPath (inputOutPaths input.inputs) ]
                  else
                    input.outPath
                ) (lib.attrsets.attrValues inputs)
              ) );

          # Add an environment variable to the call to `shellFor`
          FLAKE_INPUTS = "${lib.strings.concatStringsSep " " (inputOutPaths inputs)}";

It would be nice if haskell.nix added an option like pinAllTheThings to shellFor.

@cdfa
Copy link

cdfa commented Dec 9, 2024

I have the following flake

{
  nixConfig = {
    extra-substituters = "https://cache.iog.io";
    extra-trusted-public-keys = "hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ=";
    allow-import-from-derivation = "true";
  };

  inputs = {
    flake-utils.url = "github:numtide/flake-utils";
    haskellNix.url = "github:input-output-hk/haskell.nix";
    nixpkgs.follows = "haskellNix/nixpkgs-unstable";
  };

  outputs = { nixpkgs, haskellNix, flake-utils, ... }@inputs:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs { inherit system overlays; inherit (haskellNix) config; };
        inherit (pkgs) lib;

        overlays = [
          haskellNix.overlay
          (final: prev: {
            # This overlay adds our project to pkgs
            mypackage =
              final.haskell-nix.stackProject' {
                src = ./.;
                compiler-nix-name = "ghc948";
                # This is used by `nix develop .` to open a shell for use with
                # `cabal`, `hlint` and `haskell-language-server`
                shell.tools = {
                  cabal = { };
                  hlint = { };
                  haskell-language-server = { };
                  hie-bios = { };
                  apply-refact = { };
                  stan = { };
                  weeder = { };
                };
                # Non-Haskell shell tools go here
                shell.buildInputs = with pkgs; [
                  nixpkgs-fmt
                  ffmpeg
                  lilypond-with-fonts
                ];
                # For some strange reason this allow stack to find ghc. See https://github.com/input-output-hk/haskell.nix/issues/1759.
                shell.additional = ps: with ps; [ Cabal ];

                stackYaml = nixpkgs.lib.mkForce "buildbot_stack.yaml";

                modules = [{
                  packages = {
                    proto-lens-protobuf-types.components.library.build-tools = [ pkgs.protobuf ];
                    tensorflow-proto.components.library.build-tools = [ pkgs.protobuf final.haskellPackages.proto-lens-protoc ];
                  };
                }];
              };
          })
        ];
        flake = pkgs.mypackage.flake { };

        inputOutPaths =   inputs:
          lib.lists.unique (lib.lists.flatten (lib.lists.map (input:
                if lib.attrsets.hasAttr "inputs" input then
                  [ input.outPath (inputOutPaths input.inputs) ]
                else
                  input.outPath
              ) (lib.attrsets.attrValues inputs)
            ) );
      in
      flake // {
        # Built by `nix build .`
        packages.default = flake.packages."mypackage:exe:mypackage";
        # Override stuff here.
        devShells.default = flake.devShells.default.overrideAttrs (prev: final: {
          # buildInputs = [ pkgs.mypackage.roots pkgs.mypackage.stack-nix ] ++ prev.buildInputs;

          FLAKE_INPUTS = "${lib.strings.concatStringsSep " " (inputOutPaths inputs)}";
        });
      }
    );
}

Adding either pkgs.mypackage.roots or pkgs.mypackage.stack-nix to buildInputs with the override at the bottom will cause an infinite recursion error.
Still, nix downloads stuff after garbage collection, specifically an stdenv-linux, some cabal-bootstrap-index archive and many .cabal files.
Is it possible there are some unpinned sources that don't show up in the flake.lock?

@cydparser
Copy link
Contributor

@cdfa, try swapping prev and final:

devShells.default = flake.devShells.default.overrideAttrs (final: prev: {

@cdfa
Copy link

cdfa commented Dec 11, 2024

I knew it had to be something silly! Thanks for pointing it out! :)
And apologies for the noise on this ticket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants