feat: scanning all targets related to the given target (#128)

* chore: comment out format check * chore: debug action * chore: debug action * chore: debug action * chore: debug action * chore: debug action * chore: ts to js * Revert "chore: introduce err for md and spelling to test CI" This reverts commit 6e3a69b. * chore: hotfix remove ci_cli_version * chore: hotfix run go build in install * chore: hotfix try include cli bin * chore: hotfix show ls * chore: hotfix ts to js * chore: hotfix add await * chore: debug setup * chore: debug discover * chore: debug discover * chore: debug install * chore: debug discover * chore: debug discover * chore: debug install * chore: debug install and discover * chore: debug install and discover * chore: debug install and discover * chore: debug install and discover * chore: debug install * chore: debug install * chore: debug install * chore: debug * chore: debug cmd * chore: debug cmd * chore: debug cmd * chore: debug ci cmd * chore: build new go bin * Revert "chore: hotfix remove ci_cli_version" This reverts commit c1fdc5b. * fix: new logic for target scanner and test * chore: fix readme * chore: fix wordking * chore: test new target scanner * chore: fix install lint and format * test: try run check-* * chore: printout go version * chore: install ci and printout directory list * fix: change branch name to commit hash * fix: typo * test: build cli * test: print out usr/local/bin * test: move build file * fix: add local flag to install * fix: path * test: debug local input * fix: print out cur directory * test: move file * test: try remove cd * fix: remove local bin * test: directory * test: print out directory * test: try add go os and arch * test: add move cmd * test: print out directory * test: printout directory * test: print out directory * test: print go.mod * test: print out error * test: go version * fix: fix path * fix: go install * test: ping google * test: curl instead of ping * test: change commit hash * test: comment fallback part * test: command * test: check GOBIN * test: ls gobin * test: log GOBIN * test: export GOBIN first * test: try -h * test: echo all steps * test: ignore reject * test: run cmdline as seq in one prom * test: promiss chain * test: combine cmd * fix: fix cmd * fix: bring back old install * fix: try go build instead of install * fix: return * fix: command * fix: command * test: print goroot * fix: command * fix: print out stderr and err * fix: divide cmd * fix: remove stderr, might not work when use * fix: log messages * fix: add log message * test: debug gettarget * test: print log * fix: handle target-* * testL debug run * fix: change earthly scan target logic * test: debug file permission * test: permission file * fix: add earthfile to path * fix: point to the right target * test: log * fix: read file to get targets * fixL regex * fix: change earthly scan target logic * fix: use line reader * test: log * fix: use fs instead of line reader * chore: fix linter and format * test: log args * chore: fix md and remove log * fix: spawn multiple process for multi target * test: log * fix: async * fix: run.yml to get inputs target * chore: cleanup * Update index.md docs: improve instructions * fix: target scan in yml * fix: change from exec lib * fix: try exec without then and catch * test: log directory * test: print out directory * test: print out directory * test: echo path * test: print out directory for /usr/bin * test: try adding arg * test: echo pwd * Revert "fix: target scan in yml" This reverts commit ad93385. * fix: targets in ci yml * feat: create new ci command called find * feat: add new struct called EarthTargets * fix: mod scanfortarget to return map of path to EarthTargets * test: debug earthfile and target * fix: use new find ci to get the targets * fix: ci command * fix: output of std cmd should be json * fix: create function to call ci find * fix: add async * fix: typo * fix: typo * fix: add path to earthfile * fix: regex * fix: remove img and make target a list in scan cmd * fix: action run * fix: ts to js * fix: check nullable * fix: add null check before return * fix: check null string * test: debug stdout * test: stdout null as string * chore: cleanup and add comment * fix: cleanup and fix async syntax * fix: find all target pattern * fix: parse json * test: wrong spelling still should run mdcheck * test: remove wrong spelling word * chore: hotfix comment out test * fix: add type checking * chore: remove comment code * test: try new logic * fix: command get key paths * fix: run condition * test: debug path * test: debug path * fix: remove command * fix: change output in discover * fix: remove json * fix: remove calling ci cmd to find filtered target * test: check and check-* * fix: remove find command * test: remove test multiple targets * fix: format and linting * test: log earthfile map * fix: discover stage * fix: run stage * fix: install stage * fix: install build ci command * fix: cmd quotation mark * fix: cmd quotation mark * fix: discover test * fix: page yaml for docs * fix: page yaml for docs format data * test: run only docs test * fix: remove " * fix: page github output * fix: bad substitution * fix: github output * fix: add bracket * test: print out pwd * fixL path for generate docs * fix: bring back all flow * fix: write test for target scanner * chore: add comments and edit name * chore: clean up * fix: target push command * fix: target should run in diff process * fix: install js file * fix: earthfile path for docs * fix: doc earthfile path * fix: doc earthfile path * fix: artifact tag * fix: push target * fix: args spawn * fix: pretier * fix: artifact test and push cmd * fix: format * test: try using relative path * fix: merge conflict * fix: change array of targets to string * test: debug output * test: add extract filter logic * test: try getting without jq * fix: extract filter logic * fix: typo * test: debug no output * fix: github output * test: debug output * test: bring back json parsing * fix: remove map and bring back targets in run * fix: github output * test: debug output targets * test: remove set github output * fix: convert targets to string seperated by space * fix: remove multi line in targets * fix: remove join * fix: try line break * fix: run to accept targets not map * fix: remove earthfile map * fix: update test for run * fix: targets as an array * fix: publish and release yaml file * chore: fix onboarding docs * chore: fix and add docs * fix: file scanner linting * fix: add log, fix typos, refactor * feat: implement catalyst ci simulate command (#139) * feat: add cat ci simulate command * fix: go syntax * chore: add markdown for simulate cmd * fix: wording * fix: change icon * feat: add generate command * fix: cleanup * chore: add and fix docs for simulate and generate * fix: default value for array targets * fix: return err * fix: refactor generate and simulate cmd * fix: simulate markdown * fix: default value for sim command * fix: earth target should be relative path * fix: simulate doc * fix: relative path --------- Co-authored-by: Steven Johnson <[email protected]> * chore: add doc for artifact and img * fix: revert to master * fix: ci local to false --------- Co-authored-by: Joshua Gilman <[email protected]> Co-authored-by: Steven Johnson <[email protected]>
input-output-hk · Jan 9, 2024 · 4855c00 · 4855c00
1 parent 687e9fa
commit 4855c00
Show file tree

Hide file tree

Showing 34 changed files with 2,295 additions and 457 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -95,7 +95,7 @@ jobs:
       aws_region: ${{ inputs.aws_region }}
       ci_cli_version: ${{ inputs.ci_cli_version }}
       earthly_version: ${{ inputs.earthly_version }}
-      target: check
+      target: check check-*
     secrets:
       dockerhub_token: ${{ secrets.dockerhub_token }}
       dockerhub_username: ${{ secrets.dockerhub_username }}
@@ -137,7 +137,7 @@ jobs:
       aws_region: ${{ inputs.aws_region }}
       ci_cli_version: ${{ inputs.ci_cli_version }}
       earthly_version: ${{ inputs.earthly_version }}
-      target: test
+      target: test test-*
       privileged: true
     secrets:
       dockerhub_token: ${{ secrets.dockerhub_token }}

diff --git a/.github/workflows/pages.yml b/.github/workflows/pages.yml
@@ -86,7 +86,7 @@ jobs:
         id: build
         with:
           earthfile: ${{ inputs.earthfile }}
-          target: ${{ inputs.target }}
+          targets: ${{ inputs.target }}
           runner_address: ${{ secrets.earthly_runner_address }}
           artifact: "true"
       - name: Publish docs

diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml
@@ -84,6 +84,7 @@ jobs:
     runs-on: ubuntu-latest
     outputs:
       json: ${{ steps.check.outputs.json }}
+      paths: ${{ steps.check.outputs.paths }}
     steps:
       - uses: actions/checkout@v3
       - name: Setup CI
@@ -100,24 +101,34 @@ jobs:
       - name: Check for empty output
         id: check
         run: |
-          output=$(echo '${{ steps.discover.outputs.json }}' | jq -rc)
+          json=$(echo '${{ steps.discover.outputs.json }}' | jq -rc)
+          paths=$(echo '${{ steps.discover.outputs.paths }}' | jq -rc)
           if [ "$output" == "null" ]; then
             echo "json=[]" >> $GITHUB_OUTPUT
+            echo "paths=[]" >> $GITHUB_OUTPUT
           else
-            echo "json=$output" >> $GITHUB_OUTPUT
+            echo "json=$json" >> $GITHUB_OUTPUT
+            echo "paths=$paths" >> $GITHUB_OUTPUT
           fi
 
   run:
     runs-on: ubuntu-latest
     needs: [discover]
-    if: needs.discover.outputs.json != '[]'
+    if: needs.discover.outputs.paths != '[]'
     strategy:
       fail-fast: false
       matrix:
         platform:
           - linux/amd64
-        earthfile: ${{ fromJson(needs.discover.outputs.json) }}
+        earthfile: ${{ fromJson(needs.discover.outputs.paths) }}
     steps:
+      - name: Get filtered targets
+        id: get_target
+        run: |
+          targets=$(echo '${{ needs.discover.outputs.json }}' | jq -r --arg key '${{ matrix.earthfile }}' '.[$key][]')
+          echo "Found targets: $targets"
+          targets_with_space=$(echo $targets | tr '\n' ' ')
+          echo "targets=$targets_with_space" >> $GITHUB_OUTPUT
       - uses: actions/checkout@v3
       - name: Setup CI
         uses: input-output-hk/catalyst-ci/actions/setup@master
@@ -135,7 +146,7 @@ jobs:
         id: build
         with:
           earthfile: ${{ matrix.earthfile }}
-          target: ${{ inputs.target }}
+          targets: ${{ steps.get_target.outputs.targets }}
           platform: ${{ matrix.platform }}
           runner_address: ${{ secrets.earthly_runner_address }}
       - name: Push image

diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -68,6 +68,7 @@ jobs:
     runs-on: ubuntu-latest
     outputs:
       json: ${{ steps.check.outputs.json }}
+      paths: ${{ steps.check.outputs.paths }}
     steps:
       - uses: actions/checkout@v3
       - name: Setup CI
@@ -84,24 +85,34 @@ jobs:
       - name: Check for empty output
         id: check
         run: |
-          output=$(echo '${{ steps.discover.outputs.json }}' | jq -rc)
+          json=$(echo '${{ steps.discover.outputs.json }}' | jq -rc)
+          paths=$(echo '${{ steps.discover.outputs.paths }}' | jq -rc)
           if [ "$output" == "null" ]; then
             echo "json=[]" >> $GITHUB_OUTPUT
+            echo "paths=[]" >> $GITHUB_OUTPUT
           else
-            echo "json=$output" >> $GITHUB_OUTPUT
+            echo "json=$json" >> $GITHUB_OUTPUT
+            echo "paths=$paths" >> $GITHUB_OUTPUT
           fi
 
   run:
     runs-on: ubuntu-latest
     needs: [discover]
-    if: needs.discover.outputs.json != '[]'
+    if: needs.discover.outputs.paths != '[]'
     strategy:
       fail-fast: false
       matrix:
         platform:
           - linux/amd64
-        earthfile: ${{ fromJson(needs.discover.outputs.json) }}
+        earthfile: ${{ fromJson(needs.discover.outputs.paths) }}
     steps:
+      - name: Get filtered targets
+        id: get_target
+        run: |
+          targets=$(echo '${{ needs.discover.outputs.json }}' | jq -r --arg key '${{ matrix.earthfile }}' '.[$key][]')
+          echo "Found targets: $targets"
+          targets_with_space=$(echo $targets | tr '\n' ' ')
+          echo "targets=$targets_with_space" >> $GITHUB_OUTPUT
       - uses: actions/checkout@v3
       - name: Setup CI
         uses: input-output-hk/catalyst-ci/actions/setup@master
@@ -118,7 +129,7 @@ jobs:
         id: build
         with:
           earthfile: ${{ matrix.earthfile }}
-          target: ${{ inputs.target }}
+          targets: ${{ steps.get_target.outputs.targets }}
           platform: ${{ matrix.platform }}
           runner_address: ${{ secrets.earthly_runner_address }}
           artifact: "true"

diff --git a/.github/workflows/run.yml b/.github/workflows/run.yml
@@ -1,67 +1,68 @@
 # WARNING: If you modify this workflow, please update the documentation
 
 on:
-    workflow_call:
-      inputs:
-        privileged:
-          description: |
-            Whether the workflow should run earthly in privileged mode (earthly -P flag).
-          required: false
-          type: boolean
-          default: false
-        target:
-          description: |
-            The target to run.
-          required: true
-          type: string
-        aws_role_arn:
-          description: |
-            The ARN of the AWS role that will be assumed by the workflow. Only
-            required when configuring a remote Earthly runner.
-          required: false
-          type: string
-        aws_region:
-          description: |
-            The AWS region that will be used by the workflow. Only required when
-            configuring a remote Earthly runner.
-          required: false
-          type: string
-        ci_cli_version:
-          description: |
-            The version of the CI CLI to use.
-          required: false
-          type: string
-          default: latest
-        earthly_version:
-          description: The version of Earthly to use.
-          required: false
-          type: string
-          default: latest
-      secrets:
-        dockerhub_username:
-          description: The token to use for logging into the DockerHub registry.
-          required: false
-        dockerhub_token:
-          description: The token to use for logging into the DockerHub registry.
-          required: false
-        earthly_runner_address:
-          description: |
-            The address of the Earthly runner that will be used to build the
-            Earthly files.
-          required: false
-        earthly_runner_secret:
-          description: |
-            The ID of the AWS secret holding Earthly remote runner credentials.
-            This secret must contain the runner address and the necessary TLS
-            certificates required to authenticate with it. If omitted, a remote
-            Earthly runner will not be configured.
-          required: false
+  workflow_call:
+    inputs:
+      privileged:
+        description: |
+          Whether the workflow should run earthly in privileged mode (earthly -P flag).
+        required: false
+        type: boolean
+        default: false
+      target:
+        description: |
+          The target to run.
+        required: true
+        type: string
+      aws_role_arn:
+        description: |
+          The ARN of the AWS role that will be assumed by the workflow. Only
+          required when configuring a remote Earthly runner.
+        required: false
+        type: string
+      aws_region:
+        description: |
+          The AWS region that will be used by the workflow. Only required when
+          configuring a remote Earthly runner.
+        required: false
+        type: string
+      ci_cli_version:
+        description: |
+          The version of the CI CLI to use.
+        required: false
+        type: string
+        default: latest
+      earthly_version:
+        description: The version of Earthly to use.
+        required: false
+        type: string
+        default: latest
+    secrets:
+      dockerhub_username:
+        description: The token to use for logging into the DockerHub registry.
+        required: false
+      dockerhub_token:
+        description: The token to use for logging into the DockerHub registry.
+        required: false
+      earthly_runner_address:
+        description: |
+          The address of the Earthly runner that will be used to build the
+          Earthly files.
+        required: false
+      earthly_runner_secret:
+        description: |
+          The ID of the AWS secret holding Earthly remote runner credentials.
+          This secret must contain the runner address and the necessary TLS
+          certificates required to authenticate with it. If omitted, a remote
+          Earthly runner will not be configured.
+        required: false
 
 jobs:
   discover:
     runs-on: ubuntu-latest
     outputs:
       json: ${{ steps.check.outputs.json }}
+      paths: ${{ steps.check.outputs.paths }}
     steps:
       - uses: actions/checkout@v3
       - name: Setup CI
@@ -78,22 +79,32 @@ jobs:
       - name: Check for empty output
         id: check
         run: |
-          output=$(echo '${{ steps.discover.outputs.json }}' | jq -rc)
+          json=$(echo '${{ steps.discover.outputs.json }}' | jq -rc)
+          paths=$(echo '${{ steps.discover.outputs.paths }}' | jq -rc)
           if [ "$output" == "null" ]; then
             echo "json=[]" >> $GITHUB_OUTPUT
+            echo "paths=[]" >> $GITHUB_OUTPUT
           else
-            echo "json=$output" >> $GITHUB_OUTPUT
+            echo "json=$json" >> $GITHUB_OUTPUT
+            echo "paths=$paths" >> $GITHUB_OUTPUT
           fi
 
   run:
     runs-on: ubuntu-latest
     needs: [discover]
-    if: needs.discover.outputs.json != '[]'
+    if: needs.discover.outputs.paths != '[]'
     strategy:
       fail-fast: false
       matrix:
-        earthfile: ${{ fromJson(needs.discover.outputs.json) }}
+        earthfile: ${{ fromJson(needs.discover.outputs.paths) }}
     steps:
+      - name: Get filtered targets
+        id: get_target
+        run: |
+          targets=$(echo '${{ needs.discover.outputs.json }}' | jq -r --arg key '${{ matrix.earthfile }}' '.[$key][]')
+          echo "Found targets: $targets"
+          targets_with_space=$(echo $targets | tr '\n' ' ')
+          echo "targets=$targets_with_space" >> $GITHUB_OUTPUT
       - uses: actions/checkout@v3
       - name: Setup CI
         uses: input-output-hk/catalyst-ci/actions/setup@master
@@ -111,5 +122,5 @@ jobs:
         with:
           privileged: ${{ inputs.privileged }}
           earthfile: ${{ matrix.earthfile }}
-          target: ${{ inputs.target }}
-          runner_address: ${{ secrets.earthly_runner_address }}
+          runner_address: ${{ secrets.earthly_runner_address }}
+          targets: ${{ steps.get_target.outputs.targets }}
diff --git a/Earthfile b/Earthfile
@@ -3,6 +3,7 @@ VERSION --global-cache 0.7
 
 # cspell: words livedocs sitedocs
 
+
 # check-markdown can be done remotely.
 check-markdown: 
     DO ./earthly/mdlint+CHECK
@@ -23,19 +24,6 @@ check-bash:
 
     DO ./earthly/bash+SHELLCHECK --src=.
 
-
-## -----------------------------------------------------------------------------
-##
-## Standard CI targets.
-##
-## These targets are discovered and executed automatically by CI.
-
-# check run all checks.
-check:
-    BUILD +check-spelling
-    BUILD +check-markdown
-    BUILD +check-bash
-
 # Internal: Reference to our repo root documentation used by docs builder.
 repo-docs:
     # Create artifacts of extra files we embed inside the documentation when its built.