deno vendor should check compressed archive into git by default

[andykais]

so I have been mulling over the new deno vendor command. It seems like a very useful tool for safety, and robustness. However, I suspect I will find two things about it annoying:

1. editor fuzzy file searches will include the ./vendor folder by default
2. updating dependencies will explode git history

1 is solveable locally, but becomes annoying for contributors. The best you can do is find something common editors read from like an .editorconfig and add notes to the readme.
2 is another issue, which will make PRs harder to sift through and increase the size of repo significantly over time.

So, I had a thought, what if deno vendor did two things. The first is download all files into a ./vendor folder like it already does, but the second is to create an archive ./vendor.tar.gz (or ./vendor.zip) which is simply the vendor file compressed. The idea is, that the uncompressed vendor folder will be gitignored and vendor.tar.gz will be checked into git. Updating deps should change both the uncompressed vendor folder and the compressed vendor.tar.gz file. Git changes to those dependencies will still be reflected in version numbers where deps are used, so its not like we need the code diffs on deps.

This is something that can be built outside of deno now pretty simply, just with a script like so:

#! /bin/bash

tar xzvf ./vendor.tar.gz

the value in including this workflow in deno though, would be that deno could include some magic in deno run that detects if the uncompressed vendor folder and the compressed vendor.tar.gz file have become out of sync (like when a git pull is done), and then uncompress vendor.tar.gz before executing the program.

[andykais]

[edit] I was able to write my own tool to compress/decompress a vendor archive "magically" on deno run. An implementation is shared below:

// tgz.ts
import { readableStreamFromReader, readerFromStreamReader } from "https://deno.land/std/io/mod.ts";
import * as path from "https://deno.land/[email protected]/path/mod.ts";
import { walk, ensureDir, ensureFile } from "https://deno.land/[email protected]/fs/mod.ts";
import { Tar, Untar } from "https://deno.land/[email protected]/archive/tar.ts";


async function uncompress(input_filepath: string, output_filepath: string) {
  await ensureDir(output_filepath)
  const reader = await Deno.open(input_filepath, { read: true });
  const stream = reader.readable.pipeThrough(new DecompressionStream('gzip'))
  const untar = new Untar(readerFromStreamReader(stream.getReader()));
  for await (const entry of untar) {
    if (entry.type === 'directory') await ensureDir(path.join(output_filepath, entry.fileName))
    else {
      const filepath = path.join(output_filepath, entry.fileName)
      await ensureFile(filepath)
      const file = await Deno.create(filepath)
      await readableStreamFromReader(file).pipeTo(file.writable)
    }
  }
}

async function compress(input_filepath: string, output_filepath: string) {
  const tar = new Tar()
  for await (const entry of walk(input_filepath)) {
    if (entry.isFile) {
      await tar.append(entry.path, {filePath: entry.path})
    }
  }
  const output = await Deno.create(output_filepath);
  await readableStreamFromReader(tar.getReader())
    .pipeThrough(new CompressionStream('gzip'))
    .pipeTo(output.writable)
}

export { compress, uncompress }

import * as tgz from './tgz.ts'

const VENDOR_FOLDER = 'vendor'
const VENDOR_ARCHIVE = 'vendor.tgz'
const VENDOR_ENTRYPOINT = 'lib/mod.ts'
const VENDOR_ARCHIVE_CACHE = '.cache.vendor.tar.gz'


async function get_mtime(filepath: string) {
  const stat = await Deno.stat(filepath)
  if (stat.mtime === null) throw new Error('unsupported platform, sorry!')
  return stat.mtime.getTime()
}


async function ensure_vendor_archive() {
  try {
    return await get_mtime(VENDOR_ARCHIVE)
  } catch (e) {
    if (e instanceof Deno.errors.NotFound) {
      console.log(`[deno-local] no archive detected, running "deno vendor ${VENDOR_ENTRYPOINT}"`)
      const proc = Deno.run({ cmd: ['deno', 'vendor', '--force', VENDOR_ENTRYPOINT]})
      await proc.status()
      await tgz.compress(VENDOR_FOLDER, VENDOR_ARCHIVE)
      const mtime = await get_mtime(VENDOR_ARCHIVE)
      await Deno.writeTextFile(VENDOR_ARCHIVE_CACHE, mtime.toString())
      return mtime
    } else {
      throw e
    }
  }
}


const mtime_new = await ensure_vendor_archive()
const mtime_old = parseInt(await Deno.readTextFile(VENDOR_ARCHIVE_CACHE))
if (mtime_new > mtime_old) {
  console.log('[deno-local] vendor archive changed, uncomressing archive', VENDOR_ARCHIVE)
  await tgz.uncompress(VENDOR_ARCHIVE, VENDOR_FOLDER)
  await Deno.writeTextFile(VENDOR_ARCHIVE_CACHE, mtime_new.toString())
}
if (mtime_new < mtime_old) {
  console.log('[deno-local] vendor folder changed, updating archive', VENDOR_FOLDER)
  await tgz.compress(VENDOR_FOLDER, VENDOR_ARCHIVE)
}

const args = [...Deno.args]
if (args.includes('run')) {
  args.splice(args.indexOf('run'), 0, '--import-map', 'vendor/import_map.json')
}
await Deno.run({ cmd: ['deno', ...args]})

where it is used like so:

deno install -A deno-local.ts

deno-local run lib/mod.ts

obviously this would be more ergonomic if included within the deno cli, since the vendor folder and entrypoint could be cli args or even read from Deno.json