feature: uica / other objective functions (length / stack usage/ uica tp-prediction)

Author: dderjoel

INSTALL.md

@@ -1,4 +1,4 @@
-# Overview 
+# Overview
 
 We recommend using the Docker based setup to play around with CryptOpt.
 If you want to actually use CryptOpt for production, the results can typically be improved by running it bare metal.
@@ -11,9 +11,9 @@ You can `sudo make install-zsh`, if you want to get `zsh` completion.
 
 ```
 curl -L https://raw.githubusercontent.com/0xADE1A1DE/CryptOpt/main/Dockerfile > Dockerfile
-docker build . -t cryptopt 
+docker build . -t cryptopt
 docker run --name CryptOpt -ti cryptopt zsh
-# shell changes to 123456# 
+# shell changes to 123456#
 ./CryptOpt --help
 ```
 
@@ -22,24 +22,24 @@ docker run --name CryptOpt -ti cryptopt zsh
 1. [Install Docker](https://docs.docker.com/get-docker) or in the [Install Docker.md](./INSTALL_docker.md).
 1. Clone this repository, then change into the directory containing the `Dockerfile`.
 1. Build Container  
-Build the container with `docker build . -t cryptopt`. (`.` is the *build context*. It's the path containing the `Dockerfile`)
-This can take a while. (Maybe around 20 minutes, depending on Internet bandwidth and machine) (Note: Depending on your Docker version, it is expected that the some output is red. This is warnings of the build process piped to stderr).
-The build was successful if it ends `naming to docker.io/library/cryptopt` (or `Sucessfully tagged cryptopt:latest`)
-The build command will create a container image tagged `cryptopt`, where all the dependencies are installed and the projects are built, ready to go.
+   Build the container with `docker build . -t cryptopt`. (`.` is the _build context_. It's the path containing the `Dockerfile`)
+   This can take a while. (Maybe around 20 minutes, depending on Internet bandwidth and machine) (Note: Depending on your Docker version, it is expected that the some output is red. This is warnings of the build process piped to stderr).
+   The build was successful if it ends `naming to docker.io/library/cryptopt` (or `Sucessfully tagged cryptopt:latest`)
+   The build command will create a container image tagged `cryptopt`, where all the dependencies are installed and the projects are built, ready to go.
 
 1. Run the container image with `docker run --name CryptOpt -ti cryptopt zsh` -> you are now in the built project, your terminal should change to something like `abcdef1234#`
 
-
 ## Bare Metal
-CryptOpt itself will only write files in the operating systems' temp directory (`/tmp/` on Linux) and in its own subdirectories.
-It will require internet access to download the (Node.js) runtime and dependencies
+
+CryptOpt itself will only write files in the operating system's temp directory (`/tmp/` on Linux) and in its own subdirectories.
+It will require Internet access to download the (Node.js) runtime and dependencies
 
 1. Install dependencies (will install globally) (c.f. Dockerfile `apt install` command(s))
 1. Install [AssemblyLine](https://0xADE1A1DE.github.io/Assemblyline) (will install globally)
 1. Clone the repo with `--recurse-submodules` to also clone submodules for a bunch of useful scripts.
 1. Enable performance counters `echo "1" | sudo tee /proc/sys/kernel/perf_event_paranoid` ([MeasureSuite](https://0xADE1A1DE.github.io/MeasureSuite) will otherwise fall back to use `RDTSCP` to count cycles)
 1. Build CryptOpt with `make all`. (or `DEBUG=1 make all` if you want debug info and `--verbose` Will slow down execution by around 50%)
-CryptOpt already contains pre-built binaries for fiat-crypto.
-If you want to build them fresh, too, follow the build instructions in [the Dockerfile](./Dockerfile) or [on Fiat-Cryptography's GitHub](https://github.com/mit-plv/fiat-crypto).
-Then copy the standalone-ocaml binaries from `./src/ExtractionOCaml/{dettman_multiplication,solinas_reduction,unsaturated_solinas,word_by_word_montgomery}` to `./src/bridge/fiat-bridge/data`
-
+   CryptOpt already contains pre-built binaries for fiat-crypto.
+   If you want to build them fresh, too, follow the build instructions in [the Dockerfile](./Dockerfile) or [on Fiat-Cryptography's GitHub](https://github.com/mit-plv/fiat-crypto).
+   Then copy the standalone-ocaml binaries from `./src/ExtractionOCaml/{dettman_multiplication,solinas_reduction,unsaturated_solinas,word_by_word_montgomery}` to `./src/bridge/fiat-bridge/data`
+1. If you want to use uiCA prediction instead of running the code natively, install [uiCA](https://github.com/andreas-abel/uiCA) and create a link to the `uiCA.py` next to `./CryptOpt`, e.g `ln -s ln -s ~/github/uiCA/uiCA.py ~/github/CryptOpt/uiCA`. Then run CryptOpt with `--objectiveFunction=uiCA --uicaarch=SKL` to use the prediction of `uiCA` with the Skylake architecture. (this is currently not supported in the Docker conainer, because I didn't get around doing it.) This feature also needs `asmline` in `PATH`, which comes with installing AssemblyLine globally. This also disables Monkey-Testing.

completion/_cryptopt

@@ -37,6 +37,8 @@ _arguments -S -s \
     '(-s --seed)'{-s,--seed=}'[Seed to start the randomness with\[number\]]:seed' \
     '--framePointer=[Specify how the register for the frame pointer (rbp) is used]:framepointer:->framepointeroptions' \
     '--memoryConstraints=[Specify any memory contraints. No contraints, all, or out1--arg1 may be aliased]:memorycontraints:->memorycontraintoptions' \
+    '--uicaarch=[use uiCA estimation instead of running on hardware.]:uiCAConstraints:->uiCAOptions' \
+    '--objectiveFunction=[use objective Function instead of running on hardware.]:objectiveFunctionConstraints:->objectiveFunctionOptions' \
     '(-h,--help)'{-h,--help}'[Help]:get help prompt' \
     '(--version)'{-v,--version}'[Version]:Version information' \
     && ret=0
@@ -81,6 +83,28 @@ case "$state" in
             'all[will read all memroy from arguments (argN\[n\]) before writing any values to outN\[n\]]' \
             'out1-arg1[will not read arg1\[n\] after out1\[n\] has been written]' \
         ;;
+    uiCAOptions)
+        _values 'uiCAConstraints' \
+            'SNB[Sandy Bridge]' \
+            'IVB[Ivy Bridge]' \
+            'HSW[Haswell]' \
+            'BDW[Broadwell]' \
+            'SKL[Skylake]' \
+            'SKX[Skylake-X]' \
+            'KBL[Kaby Lake]' \
+            'CFL[Coffee Lake]' \
+            'CLX[Coffee Lake-X]' \
+            'ICL[Ice Lake]' \
+            'TGL[Tiger Lake]' \
+            'RKL[Rocket Lake]' \
+            ;;
+    objectiveFunctionOptions)
+        _values 'objectiveFunctionConstraints' \
+            'cycles[run on hardware]' \
+            'stack[used stack slots (# `mov \[rsp + off\], riz` instructions)]' \
+            'length[length of the function (# instructions)]' \
+            'uiCA[throughput estimation from uiCA. Speicfy arch with --uicaarch]' \
+            ;;
 esac
 
 

src/CryptOpt.ts

@@ -76,7 +76,7 @@ else if (parsedArgsFromCli.readState) {
   }
 }
 
-const { single, bets, betRatio, curve, method, verbose } = parsedArgs;
+const { single, bets, betRatio, curve, method, verbose, objectiveFunction, uicaarch } = parsedArgs;
 if (parsedArgs.resultDir == "") {
   parsedArgs.resultDir = resolve(process.cwd(), "results");
 }
@@ -97,7 +97,7 @@ if (!verbose) {
 const symbolname = new Optimizer(parsedArgs).getSymbolname(true);
 registerExitHooks({ ...parsedArgs, symbolname });
 
-type RunResult = { statefile: string; ratio: number; convergence: string[] };
+type RunResult = { statefile: string; ratio: number; convergenceMetric: string[]; convergence: string[] };
 
 async function allBets(evals: number, bets: number): Promise<RunResult[]> {
   const runRes = [] as RunResult[];
@@ -147,8 +147,8 @@ async function run(args: OptimizerArgs): Promise<RunResult> {
 
   const [statefile] = generateResultFilename({ ...args, symbolname: optimizer.getSymbolname() });
   Model.persist(statefile, parsedArgs);
-  const { ratio, convergence } = Model.getState();
-  return { statefile, ratio, convergence };
+  const { ratio, convergence, convergenceMetric } = Model.getState();
+  return { statefile, ratio, convergence, convergenceMetric };
 }
 
 let runResults: RunResult[];
@@ -193,22 +193,28 @@ if ("time" in parsed) {
 const lastConvergence = runResults[runResults.length - 1].convergence;
 const longestDataRow = lastConvergence.length;
 
-const spaceSeparated = runResults.reduce((arr, { convergence }) => {
-  // in order to create a matrix for gnuplot, we need to pad with " ?"
-  const paddingAmount = longestDataRow - convergence.length;
-  const paddingArray = new Array(paddingAmount).fill("  ?   ");
-  arr.push(convergence.concat(paddingArray).join(" "));
-  return arr;
-}, [] as string[]);
+const spaceSeparated = runResults.reduce(
+  (arr, { convergence, convergenceMetric }) => {
+    // in order to create a matrix for gnuplot, we need to pad with " ?"
+    const paddingAmount = longestDataRow - convergence.length;
+    const paddingArray = new Array(paddingAmount).fill("  ?   ");
+    arr.conv.push(convergence.concat(paddingArray).join(" "));
+    arr.metric.push(convergenceMetric.concat(paddingArray).join(" "));
+    return arr;
+  },
+  { conv: [], metric: [] } as { conv: string[]; metric: string[] },
+);
 
-const [datFileFull, gpFileFull, pdfFileFull] = generateResultFilename({ ...parsedArgs, symbolname }, [
-  ".dat",
-  ".gp",
-  ".pdf",
-]);
+const [datFileFull, datMetricFileFull, gpFileFull, pdfFileFull] = generateResultFilename(
+  { ...parsedArgs, symbolname },
+  [".dat", "_metric.dat", ".gp", ".pdf"],
+);
 
-writeString(datFileFull, spaceSeparated.join("\n"));
-process.stdout.write(`Wrote ${cy}${datFileFull}${re} ${spaceSeparated.length}x${longestDataRow}`);
+writeString(datFileFull, spaceSeparated.conv.join("\n"));
+writeString(datMetricFileFull, spaceSeparated.metric.join("\n"));
+process.stdout.write(
+  `Wrote ${cy}${datFileFull},${datMetricFileFull}${re} ${spaceSeparated.conv.length}x${longestDataRow}`,
+);
 
 Logger.log(JSON.stringify(times));
 const title = [
@@ -218,30 +224,46 @@ const title = [
   new Date().toISOString(),
   hostname(),
   Object.entries(times).map((k, v) => `Time for ${k}: ${(v / 60).toFixed(2)}min`),
-].join(", ");
-
-writeString(
-  gpFileFull,
-  [
-    `#!/usr/bin/env gnuplot\n`,
-    `set title "${title}"`,
-    "# missing values are the ones from earlier-finished seed-searching evaluations",
-    `set datafile missing "?"\n`,
-    "# setting output sizes and filename",
-    "set terminal pdf size 80cm,20cm",
-    `set output '${pdfFileFull}'\n`,
-    "# set x",
-    'set xlabel "Mutation"',
-    "set logscale x 10\n",
-    "# set y",
-    // "set yrange [0:2]\n",
-    `set ylabel "ratio: '${env.CC}-compiled cycle lib'/'cycle good' "\n`,
-    "# remove legend",
-    "unset key\n",
-    "# and plot the matrix with line colors, and a line at y=1 with color 0 (gre)",
-    `plot "${datFileFull}" matrix using ($1*${PRINT_EVERY}):3:2 linecolor variable with lines, 1 lc 0`,
-  ].join("\n"),
-);
+  objectiveFunction,
+  objectiveFunction === "uiCA" ? uicaarch : null,
+]
+  .filter((f) => f)
+  .join(", ");
+
+const common = [
+  `#!/usr/bin/env gnuplot\n`,
+  `set title "${title}"`,
+  "# missing values are the ones from earlier-finished seed-searching evaluations",
+  `set datafile missing "?"\n`,
+  "# setting output sizes and filename",
+  "set terminal pdf size 80cm,20cm",
+  `set output '${pdfFileFull}'\n`,
+  "# set x",
+  'set xlabel "Mutation"',
+  "set logscale x 10\n",
+  "# remove legend",
+  "unset key\n",
+  "# set y",
+  // "set yrange [0:2]\n",
+  `set ylabel "ratio: '${env.CC}-compiled cycle lib'/'cycle good' "\n`,
+];
+
+const plots = [
+  `"${datFileFull}" matrix using ($1*${PRINT_EVERY}):3:2 with lines linecolor variable`,
+  "1 lc 0",
+];
+if (objectiveFunction !== "cycles") {
+  common.push("set y2tics", `set y2label "${objectiveFunction}"`);
+  plots.push(
+    `"${datMetricFileFull}" matrix using ($1*${PRINT_EVERY}):3:2 with lines linecolor variable dt 3 axes x1y2`,
+  );
+}
+
+const plot = [
+  "# and plot the matrix with line colors, and a line at y=1 with color 0 (gre)",
+  `plot ${plots.join(", ")}`,
+];
+writeString(gpFileFull, common.concat(plot).join("\n"));
 
 process.stdout.write(" Gen Pdf...");
 const d = (chunk: Buffer | string) => {

src/errors/errors.ts

@@ -18,23 +18,23 @@ type Err = { exitCode: number; msg: string };
 export const ERRORS: { [k: string]: Err } = {
   measureGeneric: {
     exitCode: 20,
-    msg: "measuresuite.measure should return a result but didn't. RES/generic_error_{A,B}.asm as been written for debug.",
+    msg: "measureUtil.measure should return a result but didn't. RES/generic_error_{A,B}.asm as been written for debug.",
   },
   measureIncorrect: {
     exitCode: 21,
-    msg: `measuresuite.measure should return a result but didn't, because the result is not the same as per measureCheck. RES/tested_incorrect{_A.asm,_B.asm,.json} haven been written for debug.`,
+    msg: `measureUtil.measure should return a result but didn't, because the result is not the same as per measureCheck. RES/tested_incorrect{_A.asm,_B.asm,.json} haven been written for debug.`,
   },
   measureInvalid: {
     exitCode: 22,
-    msg: `measuresuite.measure should return a result but didn't, because the ASM string could not be assembled. RES/tested_incorrect{_A.asm,_B.asm,.json} haven been written for debug.`,
+    msg: `measureUtil.measure should return a result but didn't, because the ASM string could not be assembled. RES/tested_incorrect{_A.asm,_B.asm,.json} haven been written for debug.`,
   },
   measureInsufficientData: {
     exitCode: 23,
-    msg: "measuresuite.measure did not yield even one data point.",
+    msg: "measureUtil.measure did not yield even one data point.",
   },
   measureCannotAnalyze: {
     exitCode: 24,
-    msg: "measuresuite.measure did yield data points, but could not be analyzed.",
+    msg: "measureUtil.measure did yield data points, but could not be analyzed.",
   },
   bcbMakeFail: {
     exitCode: 30,

src/helper/analyse.ts

@@ -19,13 +19,8 @@ import type { AsmFunctionSummary } from "measuresuite";
 import * as Stats from "simple-statistics";
 
 import { errorOut, ERRORS } from "@/errors";
-import type {
-  AnalyseMeasureResultOptions,
-  AnalyseResult,
-  MeasureResult,
-  numTripel,
-  QuickStats,
-} from "@/types";
+import type { AnalyseMeasureResultOptions, AnalyseResult, numTripel, QuickStats } from "@/types";
+import type { CryptOptMeasureResult } from "@/optimizer/measure.class";
 
 /**
  * @param result - the result to analyse
@@ -34,7 +29,7 @@ import type {
  * @param options.checkCorrectness // throws if not correct; if this options-switch is correct
  */
 export function analyseMeasureResult(
-  result: MeasureResult | null,
+  result: CryptOptMeasureResult | null,
   options: AnalyseMeasureResultOptions,
 ): AnalyseResult {
   // assign default values
@@ -59,18 +54,28 @@ export function analyseMeasureResult(
     errorOut(ERRORS.measureInsufficientData);
   }
 
-  const [cc, ca, cb] = result.cycles.map(analyseRow);
-  const rawMedian: numTripel = [ca.pre.median, cb.pre.median, cc.pre.median];
+  let ca: { pre: QuickStats; post: QuickStats };
+  let cb: { pre: QuickStats; post: QuickStats };
+  let cc: { pre: QuickStats; post: QuickStats } | null = null;
+  if (result.cycles.length == 3) {
+    // e.g. assuming its cc ca cb, the default when running the code on the hardware
+    [cc, ca, cb] = result.cycles.map(analyseRow);
+  } else {
+    // e.g. if using estimates rather than execution data
+    [ca, cb] = result.cycles.map(analyseRow);
+    cc = null;
+  }
+  const rawMedian: numTripel = [ca.pre.median, cb.pre.median, cc?.pre.median ?? -1];
 
   if (rawMedian.some(isNaN)) {
     console.error("TSNH. Some mean is NaN." + JSON.stringify(result));
     process.exit(4);
   }
 
-  const rawStddev: numTripel = [ca.pre.stddev, cb.pre.stddev, cc.pre.stddev];
-  const noOutlierMedian: numTripel = [ca.post.median, cb.post.median, cc.post.median];
+  const rawStddev: numTripel = [ca.pre.stddev, cb.pre.stddev, cc?.pre.stddev ?? -1];
+  const noOutlierMedian: numTripel = [ca.post.median, cb.post.median, cc?.post.median ?? -1];
 
-  const noOutlierStddev: numTripel = [ca.post.stddev, cb.post.stddev, cc.post.stddev];
+  const noOutlierStddev: numTripel = [ca.post.stddev, cb.post.stddev, cc?.post.stddev ?? -1];
 
   const scale = (cyc: number): number => cyc / options.batchSize;
 
@@ -105,9 +110,22 @@ function cleanRow(arr: Array<number | undefined>, thres = 0): number[] {
 //helpers -getmedian
 export function analyseRow(arr: Array<number | undefined>): { pre: QuickStats; post: QuickStats } {
   const cleaned = cleanRow(arr);
-  // which data to consider;
-  const [pre, post] = createStatistics(cleaned);
 
+  // if we only have one datapoint, no need for statistics.
+  if (cleaned.length == 1) {
+    const p = {
+      median: cleaned[0],
+      stddev: -1,
+      n: 1,
+    };
+    return {
+      pre: p,
+      post: p,
+    };
+  }
+
+  // which data to consider; i.e remove outliers
+  const [pre, post] = createStatistics(cleaned);
   return {
     pre: {
       median: Stats.median(pre),

src/helper/argParse.ts

@@ -30,7 +30,13 @@ import {
 } from "@/bridge/fiat-bridge/constants";
 import { errorOut, ERRORS } from "@/errors";
 
-import { FRAME_POINTER_OPTIONS, MEMORY_CONSTRAINTS_OPTIONS, ParsedArgsT } from "../types";
+import {
+  FRAME_POINTER_OPTIONS,
+  UICA_OPTIONS,
+  MEMORY_CONSTRAINTS_OPTIONS,
+  ParsedArgsT,
+  OBJECTIVE_FUNCTION_OPTIONS,
+} from "../types";
 
 const y = await yargs(process.argv.slice(2));
 
@@ -216,6 +222,21 @@ export const parsedArgs = y
       "Defines if memory reads are contraint. 'none' will not enforce anything. All reads are permitted at any time. 'all' enforces that no read from any `argN[n]` happens after any write to `outN[n]`. 'out1-arg1' enforces that no read from arg1[n] is permitted after `out1[n]` has been written (essentially permits mul(r,r,x) and sq(a,a); but not if elemets overlap but not align. (e.g. mul(r+1,r,x)))",
     choices: MEMORY_CONSTRAINTS_OPTIONS,
   })
+  .option("uicaarch", {
+    default: "SKL",
+    string: true,
+    describe:
+      "Requires --objectiveFunction=uiCA. Then, CryptOpt uses estimation of uiCA instead of running the code natively. This option specifies the simulated architecture. Ensure uiCA is available in the CryptOpt directory. See INSTALL.md for details.",
+    choices: UICA_OPTIONS,
+  })
+  .option("objectiveFunction", {
+    default: "cycles",
+    string: true,
+    describe:
+      "The opjective function which describes which mutation should be kept or discarded. Options: 'cycles' run on hardware, use cycles; stack: size of the used stack slots; length: purely optimise for fewer instructions; uiCA: use uiCA's estimation. Everything but cycles effectively disables monkey-testing becuase the code is not executed.",
+
+    choices: OBJECTIVE_FUNCTION_OPTIONS,
+  })
   .help("help")
   .alias("h", "help")
   .wrap(Math.min(160, y.terminalWidth()))

src/helper/globals.ts

@@ -19,6 +19,7 @@ import type { CryptoptGlobals } from "@/types";
 const globals: CryptoptGlobals = {
   currentRatio: Infinity,
   convergence: [] as string[], // numbers, but .toFixed(4)
+  convergenceMetric: [] as string[], // see interface
   time: {
     // in seconds
     validate: 0,