Merge branch 'main' into future

Author: dkarnikis

.github/workflows/main.yml

@@ -24,20 +24,23 @@ jobs:
             ref: ${{ github.event.pull_request.head.sha }}
         - name: Running-Pash-Benchmarks
           run: |
-            bash scripts/distro-deps.sh
-            bash scripts/setup-pash.sh
+            sudo touch /.githubenv
+            # install the system deps and pash the environment
+            sudo -E bash scripts/distro-deps.sh -o
+            # install pash
+            sudo -E bash scripts/setup-pash.sh -o
             export PASH_TOP=$PWD
             export PATH=$PATH:$PASH_TOP
-            cd scripts
-            bash run_tests.sh  
-            cd workflow
-            bash parse.sh > results.log 2>&1  
-            # get the timer in english
-            d=$(LANG=en_us_88591; date)
+            # run all the tests (INTRO, INTERFACE, COMPILER)
+            cd scripts && bash run_tests.sh  
+            # fetch the execution results of the passed/failed
+            cd workflow && bash parse.sh > results.log 2>&1  
+            # get the timer
+            timer=$(LANG=en_us_88591; date)
             echo "VERSION<<EOF" >> $GITHUB_ENV
-            echo  "OS:${{matrix.os}}" >> $GITHUB_ENV
-            echo "$d" >> $GITHUB_ENV
-            # Parses the results and construct a github message post
+            echo "OS:${{matrix.os}}" >> $GITHUB_ENV
+            echo "$timer" >> $GITHUB_ENV
+            # Parse the results and construct a github message post
             # we append the data to the global env
             cat results.log >> $GITHUB_ENV
             echo 'EOF' >> $GITHUB_ENV
@@ -54,5 +57,5 @@ jobs:
               })     
         - name: Exit-code
           run: |
-            cd scripts/workflow
-            bash exit_code.sh
+            # check if everything executed without errors
+            cd scripts/workflow && bash exit_code.sh

README.md

@@ -1,36 +1,26 @@
 ## PaSh: Light-touch Data-Parallel Shell Processing
+
 > _A system for parallelizing POSIX shell scripts._
+> _Hosted by the [Linux Foundation](https://linuxfoundation.org/press-release/linux-foundation-to-host-the-pash-project-accelerating-shell-scripting-with-automated-parallelization-for-industrial-use-cases/)._
 
 Quick Jump: [Running PaSh](#running-pash) | [Installation](#installation) | [Testing](#testing) | [Repo Structure](#repo-structure) | [Community & More](#community--more) | [Citing](#citing)
 
 ## Running PaSh
 
-To parallelize, say, `./evaluation/intro/hello-world.sh` with parallelization width of `2`, from the top-level directory of the repository run:
+To parallelize, say, `./evaluation/intro/hello-world.sh` with parallelization degree of 2× run:
 
 ```sh
 ./pa.sh ./evaluation/intro/hello-world.sh
-``` 
+```
 
 Run `./pa.sh --help` to get more information about the available commands.
-See [docs/tutorial](docs/tutorial/) for a longer tutorial.
+Jump to [docs/tutorial](docs/tutorial/) for a longer tutorial.
 
 ## Installation
 
-On Ubuntu, Fedora, Debian, or Arch, run `curl up.pash.ndr.md | sh` to setup PaSh.
-Alternatively, you can clone the repo and run `./scripts/distro-deps.sh;
-./scripts/setup-pash.sh`
-
-You can also built the Docker container from scratch by running
-```sh
-git clone [email protected]:binpash/pash.git
-cd pash/scripts
-docker build -t "pash/18.04" .
-# Then launch container:
-docker run --name pash-play -it pash/18.04
-```
+On Ubuntu, Fedora, Debian, or Arch, run `curl up.pash.ndr.md | sh` to set up PaSh.
 
-More installation instructions in [the tutorial](./docs/tutorial.md#installation).
-Remember to `export` `PASH_TOP` in your startup scripts!
+For more details, manual installation, or other platforms see [installation instructions](./docs/install).
 
 ## Testing
 
@@ -45,20 +35,23 @@ To execute the current tests before committing and pushing code, simply run:
 This repo hosts the core `pash` development. The structure is as follows:
 
 * [annotations](./annotations/): DSL characterizing commands, parallelizability study, and associated annotations.
-* [compiler](./compiler): Shell-Dataflow translations and associated parallelization transformations.
+* [compiler](./compiler): Shell-dataflow translations and associated parallelization transformations.
 * [docs](./docs): Design documents, tutorials, installation instructions, etc.
-* [evaluation](./evaluation): Shell pipelines and example [scripts](./evaluation/scripts) used for the evaluation.
+* [evaluation](./evaluation): Shell pipelines and example [scripts](./evaluation/other/more-scripts) used for the evaluation.
 * [runtime](./runtime): Runtime component — e.g., `eager`, `split`, and assocaited combiners.
 * [scripts](./scripts): Scripts related to continuous integration, deployment, and testing.
 
 ## Community & More
 
-Mailing Lists: 
-* [pash-users](https://groups.google.com/g/pash-users): Join this mailing list for discussing all things `pash`
+Chat:
+* [Discord Server](ttps://discord.com/channels/947328962739187753/) ([Invite](http://join.binpa.sh/))
+
+Mailing Lists:
+* [pash-dev](https://groups.google.com/g/pash-dev): Join this mailing list for discussing all things `pash`
 * [pash-commits](https://groups.google.com/g/pash-commits): Join this mailing list for commit notifications
 
 Development/contributions:
-* Contribution guide: [docs/contrib](docs/contrib)
+* Contribution guide: [docs/contributing](docs/contributing/contrib.md)
 * Continuous Integration Server: [ci.binpa.sh](http://ci.binpa.sh)
 
 ## Citing

annotations/grep.json

@@ -39,7 +39,8 @@
             "predicate": "default",
             "class": "stateless",
             "inputs": ["args[:]"],
-            "outputs": ["stdout"]
+            "outputs": ["stdout"],
+            "comment": "This isn't _quite_ right---we need to identify a pattern somewhere and exclude it from `input`."
         }
     ],
     "comment": "For the stateless one we need to add -h|--no-filename as a requirement if it takes multiple files as input."

compiler/pash.py

@@ -261,4 +261,3 @@ def execute_script(compiled_script_filename, command, arguments, shell_name):
 if __name__ == "__main__":
     main()
 
- 

compiler/test_evaluation_scripts.sh

@@ -29,11 +29,10 @@ results_time="$test_results_dir/results.time"
 results_time_bash=${results_time}_bash
 results_time_pash=${results_time}_pash
 
-
 echo "Deleting eager intermediate files..."
-rm -f /tmp/eager*
-mkdir -p $intermediary_dir
 rm -rf "$test_results_dir"
+rm -rf "$intermediary_dir"
+mkdir -p $intermediary_dir
 mkdir -p "$test_results_dir"
 
 echo "Generating inputs..."
@@ -46,7 +45,6 @@ n_inputs=(
     8
 )
 
-
 if [ "$EXPERIMENTAL" -eq 1 ]; then
     configurations=(
         # "" # Commenting this out since the tests take a lot of time to finish
@@ -106,10 +104,20 @@ execute_pash_and_check_diff() {
         { time "$PASH_TOP/pa.sh" $@ ; } 1> "$pash_output" 2> >(tee -a "${pash_time}" >&2) &&
         diff -s "$seq_output" "$pash_output" | head | tee -a "${pash_time}" >&2
     else
-        { time "$PASH_TOP/pa.sh" $@ ; } 1> "$pash_output" 2> "${pash_time}"
-        timer=$(cat "$pash_time"); 
-        difference=$(diff -s "$seq_output" "$pash_output" | head)
-        echo "$difference$timer" > "${pash_time}"
+
+        { time "$PASH_TOP/pa.sh" $@ ; } 1> "$pash_output" 2>> "${pash_time}" &&
+        b=$(cat "$pash_time"); 
+        test_diff_ec=0
+        $(diff "$seq_output" "$pash_output" || test_diff_ec=$?)
+        echo "$c$b" > "${pash_time}"
+        # differ
+        script=$(basename $script_to_execute)
+        if [ $test_diff_ec -ne 0 ]; then
+            echo "$script are not identical" >> $test_results_dir/result_status
+        else
+            echo "$script are identical" >> $test_results_dir/result_status
+        fi
+
     fi
 }
 
@@ -135,7 +143,7 @@ execute_tests() {
         export seq_output="${intermediary_dir}/${microbenchmark}_seq_output"
         seq_time="$test_results_dir/${microbenchmark}_seq.time"
 
-        script_to_execute="${prefix}.sh"
+        export script_to_execute="${prefix}.sh"
         env_file="${prefix}_env_test.sh"
         funs_file="${prefix}_funs.sh"
         input_file="${prefix}_test.in"
@@ -214,13 +222,11 @@ echo "group,Bash,Pash2,Pash8" > ${results_time}
 paste -d'@' $test_results_dir/results.time_*  | sed 's\,\.\g' | sed 's\:\,\g' | sed 's\@\,\g' >> ${results_time}
 
 echo "Below follow the identical outputs:"
-grep --files-with-match "are identical" "$test_results_dir"/*_distr*.time |
-    sed "s,^$PASH_TOP/,,"
+grep "are identical" "$test_results_dir"/result_status | awk '{print $1}'
 
-echo "Below follow the non-identical outputs:"
-grep -L "are identical" "$test_results_dir"/*_distr*.time |
-    sed "s,^$PASH_TOP/,,"
+echo "Below follow the non-identical outputs:"     
+grep "are not identical" "$test_results_dir"/result_status | awk '{print $1}'
 
-TOTAL_TESTS=$(ls -la "$test_results_dir"/*_distr*.time | wc -l)
-PASSED_TESTS=$(grep --files-with-match "are identical" "$test_results_dir"/*_distr*.time | wc -l)
+TOTAL_TESTS=$(cat "$test_results_dir"/result_status | wc -l)
+PASSED_TESTS=$(grep -c "are identical" "$test_results_dir"/result_status)
 echo "Summary: ${PASSED_TESTS}/${TOTAL_TESTS} tests passed."

docs/contributing/contrib.md

@@ -1,3 +1,16 @@
+## Commits & Commit Messages
+
+Here are a few important guidelines w.r.t to `git` and GitHub:
+
+1. Push your work in a separate branch and create a PR when ready. PRs are always *squashed* so you can push as many corrections as you want to a PR.
+
+2. Strive for [clear and short commit messages](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html). At a minimum, start with an 80-character summary, with verbs in present tense.
+
+3. Add the `--signoff` or `-s` flag to `git commit`. This flag is intended to certify that you have created the patch in question, or that you certify that to the best of your knowledge, it was created under an appropriate open-source license, or that it has been provided to you by someone else under those terms.
+
+> To admins: You can always add `Signed-off-by: Name <email>` when squashing commits.
+
+
 ## PaSh on Docker: A Pocket Guide
 
 This is a pocket guide for running PaSh in a docker container.
@@ -109,7 +122,7 @@ Restart-Computer
 Run the `wsl` command (or find the installed Linux distribution in Windows Start menu and run it).
 After a few minutes of installation, enter a username and password for the internal WSL account to be created.
 
-Continue the PaSh installation process from [here](https://github.com/binpash/pash/blob/main/docs/tutorial.md#installation) inside the WSL installation.
+Continue the PaSh installation process from [here](https://github.com/binpash/pash/blob/main/docs/tutorial/tutorial.md#installation) inside the WSL installation.
 
 
 ## Docker TODO
@@ -153,11 +166,6 @@ git merge master           # fetch changes from main/master
 
 (You can use `rebase` instead of `merge` if your branch is local and hasn't been pushed to GitHub, but `merge` if your branch is already pushed.)
 
-## Commit Messages
-
-It's important to write [clear commit messages](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html).
-At a minimum, a short single-line summary at the top with verbs in present tense:-)
-
 ## Process for Using EC2 Instance 
 
 A reason to use Amazon Elastic Compute Cloud (EC2) is having insufficient computing power in your local machine. The steps to do are as follows. The generated key is of the form user@hostname.

docs/install/README.md

@@ -0,0 +1,83 @@
+# Installation
+
+On Ubuntu, Fedora, Debian, or Arch, run `curl up.pash.ndr.md | sh` to get PaSh up and running.
+
+If on other environments or prefer manual setup, there are essentially three steps required to set PaSh up:
+1. Clone repo: `git clone [email protected]:binpash/pash.git`
+2. Run `distro-deps.sh` (with `sudo`) and `setup-pash.sh`
+3. Export `PASH_TOP` and, optionally, add it to your `PATH`
+
+Most scripts support a `--help` flag that documents their options. These three steps are described in detail below, depending on the environment.
+
+
+Quick Jump: [Clone & Setup](#) | [Manual Setup](#) | [Docker Setup](#) | [Windows using WSL](#repo-structure) 
+
+
+### Clone & Setup
+
+The following steps clone the repo, set up dependencies (e.g., compilers), and then build PaSh:
+
+```sh
+git clone [email protected]:binpash/pash.git
+sudo pash/scripts/distro-deps.sh
+./pash/scripts/setup-pash.sh
+```
+
+These scripts have been tested on Ubuntu, Fedora, Debian, and Arch.
+
+### Manual Setup
+
+Use the system's package manager to install the following packages (some of which might already be installed in your system):
+
+```
+automake bc bsdmainutils curl gcc git libffi-dev libtool locales locales-all m4 netcat-openbsd pkg-config python python3 python3-pip python3-setuptools python3-testresources sudo wamerican-insane wget
+```
+
+Then clone the PaSh repository and run `setup-pash.sh` as follows:
+
+```sh
+git clone [email protected]:binpash/pash.git
+./pash/scripts/setup-pash.sh
+```
+
+As noted at the end of `setup-pash.sh`, make sure you set `PASH_TOP` pointing to the absolute path of the directory `pa.sh` resides (you can optionally place that in your `PATH`).
+
+### Docker Setup
+
+If Docker [is available](https://docs.docker.com/get-docker/), PaSh can be used as part of a Docker container.
+Depending on the setup, PaSh running on Docker may or may not be able to exploit all available hardware resources.
+There are two main options for setting up PaSh on Docker:
+
+_Pull from DockerHub (Major Releases):_
+To `pull` the docker image [from Docker Hub](https://hub.docker.com/r/binpash/pash), run:
+```sh
+docker pull binpash/pash
+```
+We refresh this image (as well as other images) on every major release.
+
+[//]: # "TODO(@nvasilakis, @dkarnikis): Need to automate this per release."
+
+_Build Image (Latest Commit):_
+To build the latest Docker container, run `docker build` in [scripts/docker](https://github.com/binpash/pash/tree/main/scripts/docker):
+```sh
+git clone [email protected]:binpash/pash.git
+cd pash/scripts/docker/
+docker build -f ./ubuntu/Dockerfile -t "pash:latest" .
+```
+In the last line, replace `ubuntu` with `fedora` or `debian` to get the corresponding base.
+
+
+To run the container:
+```sh
+docker run --name pash-play -it pash:latest
+```
+
+PaSh can be found in the container's `/opt/pash` directory, so run `cd pash; git pull` to fetch the latest updates.
+More information in the [pash-on-docker guide](../contributing/contrib.md#pash-on-docker-a-pocket-guide).
+
+### Windows using WSL
+
+To run PaSh on windows without Docker, install [WSL](https://docs.microsoft.com/en-us/windows/wsl/install-win10).
+A short tutorial is included in the [contributing](../contributing/contrib.md) guide.
+
+[//]: # "TODO(@nvasilakis, @dkarnikis): Need to add instructions for OS X."

docs/troubleshooting.md

@@ -0,0 +1,14 @@
+# PaSh Troubleshooting
+
+This is a list of issues when setting up or running PaSh (and how to solve them).
+
+### Module `pexpect` not found OR libdash library for parser not working  
+
+_Solution_: run dependency scipt with **sudo** and build pash libraries 
+
+```console
+cd $PASH_TOP  
+sudo bash scripts/distro-deps.sh  
+bash scripts/setup-pash.sh 
+```
+