tutorials/modelcard-tribuo-v4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Card Tutorial\n",
    "\n",
    "Even with the increasing use of ML systems, proper documentation of such systems has yet to gain traction. This has often been linked to the absence of standardized documentation procedures, which has recently led to the proposal of new documentation frameworks for ML models. One example includes the Model Card project, proposed by M. Mitchell et al. in [Model Cards for Model Reporting](https://arxiv.org/pdf/1810.03993.pdf) (2019). This framework aims to allow more transparent model reporting by having the model creator document numerous suggested details (e.g., training algorithms, intended users, etc.). \n",
    "\n",
    "To support this framework and contribute to the efforts towards transparent model reporting, we added a Model Card system to Tribuo. To decrease the workload on the developer's end and partially automate the process, the Model Card system uses Tribuo's built-in provenance to fill in the details relating to the model, its training, and its testing while also allowing the developer to explicitly specify the model's usage details either programmatically or by running a CLI.\n",
    "\n",
    "## Setup\n",
    "\n",
    "We are going to load in the LibSVM anomaly detection and Model Card jars and also import a few packages. Note that the Model Card jar is written in Java 17, and so this tutorial requires Java 17 or later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "vscode": {
     "languageId": "java"
    }
   },
   "outputs": [],
   "source": [
    "%jars ./tribuo-anomaly-libsvm-4.3.0-jar-with-dependencies.jar\n",
    "%jars ./tribuo-modelcard-4.3.0-jar-with-dependencies.jar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "vscode": {
     "languageId": "java"
    }
   },
   "outputs": [],
   "source": [
    "import com.oracle.labs.mlrg.olcut.provenance.ProvenanceUtil;\n",
    "import org.tribuo.MutableDataset;\n",
    "import org.tribuo.interop.modelcard.ModelCard;\n",
    "import org.tribuo.interop.modelcard.UsageDetailsBuilder;\n",
    "import org.tribuo.anomaly.evaluation.AnomalyEvaluator;\n",
    "import org.tribuo.anomaly.example.GaussianAnomalyDataSource;\n",
    "import org.tribuo.anomaly.libsvm.LibSVMAnomalyTrainer;\n",
    "import org.tribuo.anomaly.libsvm.SVMAnomalyType;\n",
    "import org.tribuo.common.libsvm.KernelType;\n",
    "import org.tribuo.common.libsvm.SVMParameters;\n",
    "\n",
    "import java.io.File;\n",
    "import java.io.IOException;\n",
    "import java.util.Map;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Model Card for a Tribuo Model\n",
    "\n",
    "The Model Card system only supports Tribuo models and is currently incompatible with external models. The information stored within a `ModelCard` is separated into four components: `ModelDetails`, `TrainingDetails`, `TestingDetails`, and `UsageDetails`. Construction of `ModelDetails` and `TrainingDetails` only relies on the built-in provenance of a `Model` instance while `TestingDetails` relies on the built-in provenance of an `Evaluation` instance and can be further augmented by a `Map` of testing metrics pre-selected by the user. In contrast, `UsageDetails` relies entirely on fields set by the user and can either be constructed programmatically using the `UsageDetailsBuilder` or interactively using the `ModelCardCLI`. \n",
    "\n",
    "We will first re-create the anomaly detection model from the Anomaly Detection tutorial and then construct numerous `ModelCard` objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "vscode": {
     "languageId": "java"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*\n",
      "optimization finished, #iter = 653\n",
      "obj = 289.5926348816893, rho = 3.144570476807895\n",
      "nSV = 296, nBSV = 114\n",
      "\n",
      "LibSVMModel(\n",
      "\tclass-name = org.tribuo.common.libsvm.LibSVMModel\n",
      "\tdataset = MutableDataset(\n",
      "\t\t\tclass-name = org.tribuo.MutableDataset\n",
      "\t\t\tdatasource = GaussianAnomalyDataSource(\n",
      "\t\t\t\t\tclass-name = org.tribuo.anomaly.example.GaussianAnomalyDataSource\n",
      "\t\t\t\t\texpectedMeans = List[\n",
      "\t\t\t\t\t\t1.0\n",
      "\t\t\t\t\t\t2.0\n",
      "\t\t\t\t\t\t1.0\n",
      "\t\t\t\t\t\t2.0\n",
      "\t\t\t\t\t\t5.0\n",
      "\t\t\t\t\t]\n",
      "\t\t\t\t\tanomalousMeans = List[\n",
      "\t\t\t\t\t\t-2.0\n",
      "\t\t\t\t\t\t2.0\n",
      "\t\t\t\t\t\t-2.0\n",
      "\t\t\t\t\t\t2.0\n",
      "\t\t\t\t\t\t-10.0\n",
      "\t\t\t\t\t]\n",
      "\t\t\t\t\tseed = 1\n",
      "\t\t\t\t\tnumSamples = 2000\n",
      "\t\t\t\t\tfractionAnomalous = 0.0\n",
      "\t\t\t\t\tanomalousVariances = List[\n",
      "\t\t\t\t\t\t1.0\n",
      "\t\t\t\t\t\t0.5\n",
      "\t\t\t\t\t\t0.25\n",
      "\t\t\t\t\t\t1.0\n",
      "\t\t\t\t\t\t0.1\n",
      "\t\t\t\t\t]\n",
      "\t\t\t\t\texpectedVariances = List[\n",
      "\t\t\t\t\t\t1.0\n",
      "\t\t\t\t\t\t0.5\n",
      "\t\t\t\t\t\t0.25\n",
      "\t\t\t\t\t\t1.0\n",
      "\t\t\t\t\t\t0.1\n",
      "\t\t\t\t\t]\n",
      "\t\t\t\t\thost-short-name = DataSource\n",
      "\t\t\t\t)\n",
      "\t\t\ttransformations = List[]\n",
      "\t\t\tis-sequence = false\n",
      "\t\t\tis-dense = true\n",
      "\t\t\tnum-examples = 2000\n",
      "\t\t\tnum-features = 5\n",
      "\t\t\tnum-outputs = 2\n",
      "\t\t\ttribuo-version = 4.3.0\n",
      "\t\t)\n",
      "\ttrainer = LibSVMAnomalyTrainer(\n",
      "\t\t\tclass-name = org.tribuo.anomaly.libsvm.LibSVMAnomalyTrainer\n",
      "\t\t\tcost = 1.0\n",
      "\t\t\tcoef0 = 0.0\n",
      "\t\t\tseed = 12345\n",
      "\t\t\tcache_size = 500.0\n",
      "\t\t\tprobability = false\n",
      "\t\t\tnu = 0.1\n",
      "\t\t\tdegree = 3\n",
      "\t\t\teps = 0.001\n",
      "\t\t\tkernelType = RBF\n",
      "\t\t\tp = 0.1\n",
      "\t\t\tshrinking = true\n",
      "\t\t\tsvmType = SVMAnomalyType(\n",
      "\t\t\t\t\tclass-name = org.tribuo.anomaly.libsvm.SVMAnomalyType\n",
      "\t\t\t\t\ttype = ONE_CLASS\n",
      "\t\t\t\t\thost-short-name = SVMType\n",
      "\t\t\t\t)\n",
      "\t\t\tgamma = 1.0\n",
      "\t\t\ttribuo-version = 4.3.0\n",
      "\t\t\ttrain-invocation-count = 0\n",
      "\t\t\tis-sequence = false\n",
      "\t\t\thost-short-name = Trainer\n",
      "\t\t)\n",
      "\ttrained-at = 2022-10-07T12:03:06.539476091-04:00\n",
      "\tinstance-values = Map{}\n",
      "\ttribuo-version = 4.3.0\n",
      "\tjava-version = 17.0.4.1\n",
      "\tos-name = Linux\n",
      "\tos-arch = amd64\n",
      ")\n",
      "\n",
      "EvaluationProvenance(\n",
      "\tclass-name = org.tribuo.provenance.EvaluationProvenance\n",
      "\tmodel-provenance = LibSVMModel(\n",
      "\t\t\tclass-name = org.tribuo.common.libsvm.LibSVMModel\n",
      "\t\t\tdataset = MutableDataset(\n",
      "\t\t\t\t\tclass-name = org.tribuo.MutableDataset\n",
      "\t\t\t\t\tdatasource = GaussianAnomalyDataSource(\n",
      "\t\t\t\t\t\t\tclass-name = org.tribuo.anomaly.example.GaussianAnomalyDataSource\n",
      "\t\t\t\t\t\t\texpectedMeans = List[\n",
      "\t\t\t\t\t\t\t\t1.0\n",
      "\t\t\t\t\t\t\t\t2.0\n",
      "\t\t\t\t\t\t\t\t1.0\n",
      "\t\t\t\t\t\t\t\t2.0\n",
      "\t\t\t\t\t\t\t\t5.0\n",
      "\t\t\t\t\t\t\t]\n",
      "\t\t\t\t\t\t\tanomalousMeans = List[\n",
      "\t\t\t\t\t\t\t\t-2.0\n",
      "\t\t\t\t\t\t\t\t2.0\n",
      "\t\t\t\t\t\t\t\t-2.0\n",
      "\t\t\t\t\t\t\t\t2.0\n",
      "\t\t\t\t\t\t\t\t-10.0\n",
      "\t\t\t\t\t\t\t]\n",
      "\t\t\t\t\t\t\tseed = 1\n",
      "\t\t\t\t\t\t\tnumSamples = 2000\n",
      "\t\t\t\t\t\t\tfractionAnomalous = 0.0\n",
      "\t\t\t\t\t\t\tanomalousVariances = List[\n",
      "\t\t\t\t\t\t\t\t1.0\n",
      "\t\t\t\t\t\t\t\t0.5\n",
      "\t\t\t\t\t\t\t\t0.25\n",
      "\t\t\t\t\t\t\t\t1.0\n",
      "\t\t\t\t\t\t\t\t0.1\n",
      "\t\t\t\t\t\t\t]\n",
      "\t\t\t\t\t\t\texpectedVariances = List[\n",
      "\t\t\t\t\t\t\t\t1.0\n",
      "\t\t\t\t\t\t\t\t0.5\n",
      "\t\t\t\t\t\t\t\t0.25\n",
      "\t\t\t\t\t\t\t\t1.0\n",
      "\t\t\t\t\t\t\t\t0.1\n",
      "\t\t\t\t\t\t\t]\n",
      "\t\t\t\t\t\t\thost-short-name = DataSource\n",
      "\t\t\t\t\t\t)\n",
      "\t\t\t\t\ttransformations = List[]\n",
      "\t\t\t\t\tis-sequence = false\n",
      "\t\t\t\t\tis-dense = true\n",
      "\t\t\t\t\tnum-examples = 2000\n",
      "\t\t\t\t\tnum-features = 5\n",
      "\t\t\t\t\tnum-outputs = 2\n",
      "\t\t\t\t\ttribuo-version = 4.3.0\n",
      "\t\t\t\t)\n",
      "\t\t\ttrainer = LibSVMAnomalyTrainer(\n",
      "\t\t\t\t\tclass-name = org.tribuo.anomaly.libsvm.LibSVMAnomalyTrainer\n",
      "\t\t\t\t\tcost = 1.0\n",
      "\t\t\t\t\tcoef0 = 0.0\n",
      "\t\t\t\t\tseed = 12345\n",
      "\t\t\t\t\tcache_size = 500.0\n",
      "\t\t\t\t\tprobability = false\n",
      "\t\t\t\t\tnu = 0.1\n",
      "\t\t\t\t\tdegree = 3\n",
      "\t\t\t\t\teps = 0.001\n",
      "\t\t\t\t\tkernelType = RBF\n",
      "\t\t\t\t\tp = 0.1\n",
      "\t\t\t\t\tshrinking = true\n",
      "\t\t\t\t\tsvmType = SVMAnomalyType(\n",
      "\t\t\t\t\t\t\tclass-name = org.tribuo.anomaly.libsvm.SVMAnomalyType\n",
      "\t\t\t\t\t\t\ttype = ONE_CLASS\n",
      "\t\t\t\t\t\t\thost-short-name = SVMType\n",
      "\t\t\t\t\t\t)\n",
      "\t\t\t\t\tgamma = 1.0\n",
      "\t\t\t\t\ttribuo-version = 4.3.0\n",
      "\t\t\t\t\ttrain-invocation-count = 0\n",
      "\t\t\t\t\tis-sequence = false\n",
      "\t\t\t\t\thost-short-name = Trainer\n",
      "\t\t\t\t)\n",
      "\t\t\ttrained-at = 2022-10-07T12:03:06.539476091-04:00\n",
      "\t\t\tinstance-values = Map{}\n",
      "\t\t\ttribuo-version = 4.3.0\n",
      "\t\t\tjava-version = 17.0.4.1\n",
      "\t\t\tos-name = Linux\n",
      "\t\t\tos-arch = amd64\n",
      "\t\t)\n",
      "\tdataset-provenance = MutableDataset(\n",
      "\t\t\tclass-name = org.tribuo.MutableDataset\n",
      "\t\t\tdatasource = GaussianAnomalyDataSource(\n",
      "\t\t\t\t\tclass-name = org.tribuo.anomaly.example.GaussianAnomalyDataSource\n",
      "\t\t\t\t\texpectedMeans = List[\n",
      "\t\t\t\t\t\t1.0\n",
      "\t\t\t\t\t\t2.0\n",
      "\t\t\t\t\t\t1.0\n",
      "\t\t\t\t\t\t2.0\n",
      "\t\t\t\t\t\t5.0\n",
      "\t\t\t\t\t]\n",
      "\t\t\t\t\tanomalousMeans = List[\n",
      "\t\t\t\t\t\t-2.0\n",
      "\t\t\t\t\t\t2.0\n",
      "\t\t\t\t\t\t-2.0\n",
      "\t\t\t\t\t\t2.0\n",
      "\t\t\t\t\t\t-10.0\n",
      "\t\t\t\t\t]\n",
      "\t\t\t\t\tseed = 2\n",
      "\t\t\t\t\tnumSamples = 2000\n",
      "\t\t\t\t\tfractionAnomalous = 0.2\n",
      "\t\t\t\t\tanomalousVariances = List[\n",
      "\t\t\t\t\t\t1.0\n",
      "\t\t\t\t\t\t0.5\n",
      "\t\t\t\t\t\t0.25\n",
      "\t\t\t\t\t\t1.0\n",
      "\t\t\t\t\t\t0.1\n",
      "\t\t\t\t\t]\n",
      "\t\t\t\t\texpectedVariances = List[\n",
      "\t\t\t\t\t\t1.0\n",
      "\t\t\t\t\t\t0.5\n",
      "\t\t\t\t\t\t0.25\n",
      "\t\t\t\t\t\t1.0\n",
      "\t\t\t\t\t\t0.1\n",
      "\t\t\t\t\t]\n",
      "\t\t\t\t\thost-short-name = DataSource\n",
      "\t\t\t\t)\n",
      "\t\t\ttransformations = List[]\n",
      "\t\t\tis-sequence = false\n",
      "\t\t\tis-dense = true\n",
      "\t\t\tnum-examples = 2000\n",
      "\t\t\tnum-features = 5\n",
      "\t\t\tnum-outputs = 2\n",
      "\t\t\ttribuo-version = 4.3.0\n",
      "\t\t)\n",
      "\ttribuo-version = 4.3.0\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "var trainData = new MutableDataset<>(new GaussianAnomalyDataSource(2000,0.0f, 1L));\n",
    "var evalData = new MutableDataset<>(new GaussianAnomalyDataSource(2000,0.2f,2L));\n",
    "var params = new SVMParameters<>(new SVMAnomalyType(SVMAnomalyType.SVMMode.ONE_CLASS), KernelType.RBF);\n",
    "params.setGamma(1.0);\n",
    "params.setNu(0.1);\n",
    "\n",
    "var trainer = new LibSVMAnomalyTrainer(params);\n",
    "var model = trainer.train(trainData);\n",
    "var evaluator = new AnomalyEvaluator();\n",
    "var evaluation = evaluator.evaluate(model,evalData);\n",
    "\n",
    "System.out.println();\n",
    "System.out.println(ProvenanceUtil.formattedProvenanceString(model.getProvenance()));\n",
    "System.out.println();\n",
    "System.out.println(ProvenanceUtil.formattedProvenanceString(evaluation.getProvenance()));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the very least, constructing a `ModelCard`  requires a `Model` and its `Evaluation`. Providing only these two parameters results in a `ModelCard` without any testing metrics and a null `UsageDetails`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "vscode": {
     "languageId": "java"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"ModelDetails\" : {\n",
      "    \"schema-version\" : \"1.0\",\n",
      "    \"model-type\" : \"LibSVMAnomalyModel\",\n",
      "    \"model-package\" : \"org.tribuo.anomaly.libsvm.LibSVMAnomalyModel\",\n",
      "    \"tribuo-version\" : \"4.3.0\",\n",
      "    \"java-version\" : \"17.0.4.1\",\n",
      "    \"configured-parameters\" : {\n",
      "      \"cost\" : \"1.0\",\n",
      "      \"coef0\" : \"0.0\",\n",
      "      \"seed\" : \"12345\",\n",
      "      \"cache_size\" : \"500.0\",\n",
      "      \"probability\" : \"false\",\n",
      "      \"nu\" : \"0.1\",\n",
      "      \"train-invocation-count\" : \"0\",\n",
      "      \"is-sequence\" : \"false\",\n",
      "      \"degree\" : \"3\",\n",
      "      \"eps\" : \"0.001\",\n",
      "      \"host-short-name\" : \"Trainer\",\n",
      "      \"class-name\" : \"org.tribuo.anomaly.libsvm.LibSVMAnomalyTrainer\",\n",
      "      \"kernelType\" : \"RBF\",\n",
      "      \"p\" : \"0.1\",\n",
      "      \"shrinking\" : \"true\",\n",
      "      \"svmType\" : {\n",
      "        \"type\" : \"ONE_CLASS\",\n",
      "        \"host-short-name\" : \"SVMType\",\n",
      "        \"class-name\" : \"org.tribuo.anomaly.libsvm.SVMAnomalyType\"\n",
      "      },\n",
      "      \"tribuo-version\" : \"4.3.0\",\n",
      "      \"gamma\" : \"1.0\"\n",
      "    }\n",
      "  },\n",
      "  \"TrainingDetails\" : {\n",
      "    \"schema-version\" : \"1.0\",\n",
      "    \"training-time\" : \"2022-10-07T12:03:06.539476091-04:00\",\n",
      "    \"training-set-size\" : 2000,\n",
      "    \"num-features\" : 5,\n",
      "    \"features-list\" : [ \"A\", \"B\", \"C\", \"D\", \"E\" ],\n",
      "    \"num-outputs\" : 2,\n",
      "    \"outputs-distribution\" : {\n",
      "      \"ANOMALOUS\" : 0,\n",
      "      \"EXPECTED\" : 2000\n",
      "    }\n",
      "  },\n",
      "  \"TestingDetails\" : {\n",
      "    \"schema-version\" : \"1.0\",\n",
      "    \"testing-set-size\" : 2000,\n",
      "    \"metrics\" : { }\n",
      "  },\n",
      "  \"UsageDetails\" : null\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "ModelCard card1 = new ModelCard(model, evaluation);\n",
    "System.out.println(card1.toString());"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To include testing metrics in the `TestingDetails` section of a `ModelCard`, a map of the metrics (with the key being the metric description and the value being the metric value) can be provided as an additional parameter to the constructor. Note that the `ModelCard` constructor copies all the items from the parameter map rather than storing a reference to that map. This means that any changes applied to the parameter map after a `ModelCard` is constructed will not be present in the metrics map of the `ModelCard`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "vscode": {
     "languageId": "java"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"ModelDetails\" : {\n",
      "    \"schema-version\" : \"1.0\",\n",
      "    \"model-type\" : \"LibSVMAnomalyModel\",\n",
      "    \"model-package\" : \"org.tribuo.anomaly.libsvm.LibSVMAnomalyModel\",\n",
      "    \"tribuo-version\" : \"4.3.0\",\n",
      "    \"java-version\" : \"17.0.4.1\",\n",
      "    \"configured-parameters\" : {\n",
      "      \"cost\" : \"1.0\",\n",
      "      \"coef0\" : \"0.0\",\n",
      "      \"seed\" : \"12345\",\n",
      "      \"cache_size\" : \"500.0\",\n",
      "      \"probability\" : \"false\",\n",
      "      \"nu\" : \"0.1\",\n",
      "      \"train-invocation-count\" : \"0\",\n",
      "      \"is-sequence\" : \"false\",\n",
      "      \"degree\" : \"3\",\n",
      "      \"eps\" : \"0.001\",\n",
      "      \"host-short-name\" : \"Trainer\",\n",
      "      \"class-name\" : \"org.tribuo.anomaly.libsvm.LibSVMAnomalyTrainer\",\n",
      "      \"kernelType\" : \"RBF\",\n",
      "      \"p\" : \"0.1\",\n",
      "      \"shrinking\" : \"true\",\n",
      "      \"svmType\" : {\n",
      "        \"type\" : \"ONE_CLASS\",\n",
      "        \"host-short-name\" : \"SVMType\",\n",
      "        \"class-name\" : \"org.tribuo.anomaly.libsvm.SVMAnomalyType\"\n",
      "      },\n",
      "      \"tribuo-version\" : \"4.3.0\",\n",
      "      \"gamma\" : \"1.0\"\n",
      "    }\n",
      "  },\n",
      "  \"TrainingDetails\" : {\n",
      "    \"schema-version\" : \"1.0\",\n",
      "    \"training-time\" : \"2022-10-07T12:03:06.539476091-04:00\",\n",
      "    \"training-set-size\" : 2000,\n",
      "    \"num-features\" : 5,\n",
      "    \"features-list\" : [ \"A\", \"B\", \"C\", \"D\", \"E\" ],\n",
      "    \"num-outputs\" : 2,\n",
      "    \"outputs-distribution\" : {\n",
      "      \"ANOMALOUS\" : 0,\n",
      "      \"EXPECTED\" : 2000\n",
      "    }\n",
      "  },\n",
      "  \"TestingDetails\" : {\n",
      "    \"schema-version\" : \"1.0\",\n",
      "    \"testing-set-size\" : 2000,\n",
      "    \"metrics\" : {\n",
      "      \"overall-recall\" : 1.0,\n",
      "      \"overall-precision\" : 0.6357927786499215\n",
      "    }\n",
      "  },\n",
      "  \"UsageDetails\" : null\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "Map<String, Double> testingMetrics = new HashMap<>();\n",
    "testingMetrics.put(\"overall-precision\", evaluation.getPrecision());\n",
    "testingMetrics.put(\"overall-recall\", evaluation.getRecall());\n",
    "\n",
    "ModelCard card2 = new ModelCard(model, evaluation, testingMetrics);\n",
    "System.out.println(card2);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition, we can construct a `UsageDetails` object using the `UsageDetailsBuilder` and specify details on the appropriate usage of our trained model. The `UsageDetails` object can then be provided as an additional parameter to the `ModelCard` constructor. Note that `UsageDetailsBuilder` will have the empty string or list as the default value for any of its fields, which will also get carried over to the `UsageDetails` object after building. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "vscode": {
     "languageId": "java"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"ModelDetails\" : {\n",
      "    \"schema-version\" : \"1.0\",\n",
      "    \"model-type\" : \"LibSVMAnomalyModel\",\n",
      "    \"model-package\" : \"org.tribuo.anomaly.libsvm.LibSVMAnomalyModel\",\n",
      "    \"tribuo-version\" : \"4.3.0\",\n",
      "    \"java-version\" : \"17.0.4.1\",\n",
      "    \"configured-parameters\" : {\n",
      "      \"cost\" : \"1.0\",\n",
      "      \"coef0\" : \"0.0\",\n",
      "      \"seed\" : \"12345\",\n",
      "      \"cache_size\" : \"500.0\",\n",
      "      \"probability\" : \"false\",\n",
      "      \"nu\" : \"0.1\",\n",
      "      \"train-invocation-count\" : \"0\",\n",
      "      \"is-sequence\" : \"false\",\n",
      "      \"degree\" : \"3\",\n",
      "      \"eps\" : \"0.001\",\n",
      "      \"host-short-name\" : \"Trainer\",\n",
      "      \"class-name\" : \"org.tribuo.anomaly.libsvm.LibSVMAnomalyTrainer\",\n",
      "      \"kernelType\" : \"RBF\",\n",
      "      \"p\" : \"0.1\",\n",
      "      \"shrinking\" : \"true\",\n",
      "      \"svmType\" : {\n",
      "        \"type\" : \"ONE_CLASS\",\n",
      "        \"host-short-name\" : \"SVMType\",\n",
      "        \"class-name\" : \"org.tribuo.anomaly.libsvm.SVMAnomalyType\"\n",
      "      },\n",
      "      \"tribuo-version\" : \"4.3.0\",\n",
      "      \"gamma\" : \"1.0\"\n",
      "    }\n",
      "  },\n",
      "  \"TrainingDetails\" : {\n",
      "    \"schema-version\" : \"1.0\",\n",
      "    \"training-time\" : \"2022-10-07T12:03:06.539476091-04:00\",\n",
      "    \"training-set-size\" : 2000,\n",
      "    \"num-features\" : 5,\n",
      "    \"features-list\" : [ \"A\", \"B\", \"C\", \"D\", \"E\" ],\n",
      "    \"num-outputs\" : 2,\n",
      "    \"outputs-distribution\" : {\n",
      "      \"ANOMALOUS\" : 0,\n",
      "      \"EXPECTED\" : 2000\n",
      "    }\n",
      "  },\n",
      "  \"TestingDetails\" : {\n",
      "    \"schema-version\" : \"1.0\",\n",
      "    \"testing-set-size\" : 2000,\n",
      "    \"metrics\" : {\n",
      "      \"overall-recall\" : 1.0,\n",
      "      \"overall-precision\" : 0.6357927786499215\n",
      "    }\n",
      "  },\n",
      "  \"UsageDetails\" : {\n",
      "    \"schema-version\" : \"1.0\",\n",
      "    \"intended-use\" : \"Anomaly detection\",\n",
      "    \"intended-users\" : \"ML learners\",\n",
      "    \"out-of-scope-uses\" : [ ],\n",
      "    \"pre-processing-steps\" : [ ],\n",
      "    \"considerations-list\" : [ ],\n",
      "    \"relevant-factors-list\" : [ ],\n",
      "    \"resources-list\" : [ ],\n",
      "    \"primary-contact\" : \"Alice\",\n",
      "    \"model-citation\" : \"\",\n",
      "    \"model-license\" : \"\"\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "var builder = new UsageDetailsBuilder();\n",
    "builder = builder.intendedUse(\"Anomaly detection\")\n",
    "                 .intendedUsers(\"ML learners\")\n",
    "                 .primaryContact(\"Alice\");\n",
    "\n",
    "ModelCard card3 = new ModelCard(model, evaluation, testingMetrics, builder.build());\n",
    "System.out.println(card3);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, the `UsageDetails` can also be appended to a `ModelCard` after the construction of the `ModelCard`. This will require using the `ModelCardCLI`, which will launch a new shell and allow the user to interactively set the fields of `UsageDetails`. In the end, to append the `UsageDetails` to a desired `ModelCard`, the `ModelCardCLI` will require a file containing a serialized version of the desired `ModelCard`. We can easily create the serialized version of a `ModelCard` and save it to a file. The file containing the serialized `ModelCard` can also be deserialized to instantiate a `ModelCard` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "vscode": {
     "languageId": "java"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"ModelDetails\" : {\n",
      "    \"schema-version\" : \"1.0\",\n",
      "    \"model-type\" : \"LibSVMAnomalyModel\",\n",
      "    \"model-package\" : \"org.tribuo.anomaly.libsvm.LibSVMAnomalyModel\",\n",
      "    \"tribuo-version\" : \"4.3.0\",\n",
      "    \"java-version\" : \"17.0.4.1\",\n",
      "    \"configured-parameters\" : {\n",
      "      \"cost\" : \"1.0\",\n",
      "      \"coef0\" : \"0.0\",\n",
      "      \"seed\" : \"12345\",\n",
      "      \"cache_size\" : \"500.0\",\n",
      "      \"probability\" : \"false\",\n",
      "      \"nu\" : \"0.1\",\n",
      "      \"train-invocation-count\" : \"0\",\n",
      "      \"is-sequence\" : \"false\",\n",
      "      \"degree\" : \"3\",\n",
      "      \"eps\" : \"0.001\",\n",
      "      \"host-short-name\" : \"Trainer\",\n",
      "      \"class-name\" : \"org.tribuo.anomaly.libsvm.LibSVMAnomalyTrainer\",\n",
      "      \"kernelType\" : \"RBF\",\n",
      "      \"p\" : \"0.1\",\n",
      "      \"shrinking\" : \"true\",\n",
      "      \"svmType\" : {\n",
      "        \"type\" : \"ONE_CLASS\",\n",
      "        \"host-short-name\" : \"SVMType\",\n",
      "        \"class-name\" : \"org.tribuo.anomaly.libsvm.SVMAnomalyType\"\n",
      "      },\n",
      "      \"tribuo-version\" : \"4.3.0\",\n",
      "      \"gamma\" : \"1.0\"\n",
      "    }\n",
      "  },\n",
      "  \"TrainingDetails\" : {\n",
      "    \"schema-version\" : \"1.0\",\n",
      "    \"training-time\" : \"2022-10-07T12:03:06.539476091-04:00\",\n",
      "    \"training-set-size\" : 2000,\n",
      "    \"num-features\" : 5,\n",
      "    \"features-list\" : [ \"A\", \"B\", \"C\", \"D\", \"E\" ],\n",
      "    \"num-outputs\" : 2,\n",
      "    \"outputs-distribution\" : {\n",
      "      \"ANOMALOUS\" : 0,\n",
      "      \"EXPECTED\" : 2000\n",
      "    }\n",
      "  },\n",
      "  \"TestingDetails\" : {\n",
      "    \"schema-version\" : \"1.0\",\n",
      "    \"testing-set-size\" : 2000,\n",
      "    \"metrics\" : {\n",
      "      \"overall-precision\" : 0.6357927786499215,\n",
      "      \"overall-recall\" : 1.0\n",
      "    }\n",
      "  },\n",
      "  \"UsageDetails\" : {\n",
      "    \"schema-version\" : \"1.0\",\n",
      "    \"intended-use\" : \"Anomaly detection\",\n",
      "    \"intended-users\" : \"ML learners\",\n",
      "    \"out-of-scope-uses\" : [ ],\n",
      "    \"pre-processing-steps\" : [ ],\n",
      "    \"considerations-list\" : [ ],\n",
      "    \"relevant-factors-list\" : [ ],\n",
      "    \"resources-list\" : [ ],\n",
      "    \"primary-contact\" : \"Alice\",\n",
      "    \"model-citation\" : \"\",\n",
      "    \"model-license\" : \"\"\n",
      "  }\n",
      "}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "true"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "File output = File.createTempFile(\"output\", \"json\");\n",
    "card3.saveToFile(output.toPath());\n",
    "\n",
    "ModelCard card4 = ModelCard.deserializeFromJson(output.toPath());\n",
    "System.out.println(card4);\n",
    "\n",
    "output.delete();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that `ModelCardCLI` will only append a `UsageDetails` to a `ModelCard` that has a null `UsageDetails` and will throw an error in all other cases. This means that a serialized `ModelCard` file that had its `UsageDetails` set programmatically with the `UsageDetailsBuilder` cannot be provided to the `ModelCardCLI`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "We showed how to create a partially automated documentation of a Tribuo ML model using the Model Card system. The system relies on Tribuo's built-in provenance to fill in the details relating to the model, its training, and its testing. It then allows the developer to specify details on the model's usage either programmatically or interactively with a CLI. Over time, we plan to allow developers to specify additional details about their models, with a special focus on supporting more quantitative and statistical details."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Java",
   "language": "java",
   "name": "java"
  },
  "language_info": {
   "codemirror_mode": "java",
   "file_extension": ".jshell",
   "mimetype": "text/x-java-source",
   "name": "Java",
   "pygments_lexer": "java",
   "version": "17.0.4.1+1-LTS-2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}