03_Tasks/sem_wellformedness.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*David Schlangen, 2019-03-24*\n",
    "\n",
    "# Task: Predicting Semantic Well-Formedness\n",
    "\n",
    "What is it that makes the assertion odd that \"colourless green ideas sleep furiously\"? It is not a *syntactic* problem, goes the famous claim \\cite{chomsky:synstruc}, but rather a semantic one.\n",
    "\n",
    "In a recent book \\cite{asher_2011}, Nicholas Asher analyses the fact that certain expressions appear odd (e.g., \"Tigers are financial institutions\") as being due to the semantic ill-formedness of the *predications* they express, proposing that this is something that lexical semantics ought to explain. (In Asher's approach, this is done via fine-grained semantic types and type hierarchies, through which certain applications can be blocked.) (See also \\cite{Vecchi2017}.)\n",
    "\n",
    "Let's explore whether our corpora might provide material to test or even develop approaches to this phenomenon.\n",
    "\n",
    "**Technical Note**\n",
    "\n",
    "If you want to execute this notebook, besides the usual preconditions, you also need to have a [Stanford CoreNLP](https://stanfordnlp.github.io/CoreNLP/) server running on localhost on port 9000. (Download the package at the link, if you don't have it.) Like so:\n",
    "\n",
    "```\n",
    "cd StanfordCORE_NLP_DIR\n",
    "java -mx4g -cp \"*\" edu.stanford.nlp.pipeline.StanfordCoreNLPServer \\\n",
    "-preload tokenize,ssplit,pos,lemma,ner,parse,depparse \\\n",
    "-status_port 9000 -port 9000 -timeout 15000 & \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "code_folding": [
     0
    ],
    "tags": [
     "notebook"
    ]
   },
   "outputs": [],
   "source": [
    "# imports\n",
    "\n",
    "from collections import defaultdict\n",
    "import configparser\n",
    "import os\n",
    "import random\n",
    "from textwrap import fill\n",
    "import sys\n",
    "from copy import deepcopy\n",
    "\n",
    "from nltk.parse import CoreNLPParser\n",
    "import nltk\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "from IPython.display import Latex, display\n",
    "\n",
    "pd.set_option('max_colwidth', 250)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "code_folding": [
     0
    ],
    "tags": [
     "notebook"
    ]
   },
   "outputs": [],
   "source": [
    "# Load up config file (needs path; adapt env var if necessary); local imports\n",
    "\n",
    "# load config file, set up paths, make project-specific imports\n",
    "config_path = os.environ.get('VISCONF')\n",
    "if not config_path:\n",
    "    # try default location, if not in environment\n",
    "    default_path_to_config = '../../clp-vision/Config/default.cfg'\n",
    "    if os.path.isfile(default_path_to_config):\n",
    "        config_path = default_path_to_config\n",
    "\n",
    "assert config_path is not None, 'You need to specify the path to the config file via environment variable VISCONF.'        \n",
    "\n",
    "config = configparser.ConfigParser()\n",
    "with open(config_path, 'r', encoding='utf-8') as f:\n",
    "    config.readfp(f)\n",
    "\n",
    "corpora_base = config.get('DEFAULT', 'corpora_base')\n",
    "preproc_path = config.get('DSGV-PATHS', 'preproc_path')\n",
    "dsgv_home = config.get('DSGV-PATHS', 'dsgv_home')\n",
    "\n",
    "\n",
    "sys.path.append(dsgv_home + '/Utils')\n",
    "from utils import icorpus_code, plot_labelled_bb, get_image_filename, query_by_id\n",
    "from utils import plot_img_cropped, plot_img_ax, invert_dict, get_a_by_b\n",
    "sys.path.append(dsgv_home + '/WACs/WAC_Utils')\n",
    "from wac_utils import create_word2den, is_relational\n",
    "sys.path.append(dsgv_home + '/Preproc')\n",
    "from sim_preproc import load_imsim, n_most_sim\n",
    "\n",
    "sys.path.append('../Common')\n",
    "from data_utils import load_dfs, plot_rel_by_relid, get_obj_bb, compute_distance_objs\n",
    "from data_utils import get_obj_key, compute_relpos_relargs_row, get_all_predicate\n",
    "from data_utils import compute_distance_relargs_row, get_rel_type, get_rel_instances\n",
    "from data_utils import compute_obj_sizes_row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "code_folding": [
     0
    ],
    "tags": [
     "notebook"
    ]
   },
   "outputs": [],
   "source": [
    "# Load up preprocessed DataFrames. Slow!\n",
    "# These DataFrames are the result of pre-processing the original corpus data,\n",
    "# as per dsg-vision/Preprocessing/preproc.py\n",
    "\n",
    "df_names = ['vgattdf', 'vgobjdf', 'vgregdf', #'vgimgdf', 'vgobjdf', 'vgreldf',\n",
    "           ]\n",
    "df = load_dfs(preproc_path, df_names)\n",
    "\n",
    "# a derived DF, containing only those region descriptions which I was able to resolve\n",
    "df['vgpregdf'] = df['vgregdf'][df['vgregdf']['pphrase'].notnull() & \n",
    "                               (df['vgregdf']['pphrase'] != '')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Data \n",
    "It is relatively straightforward to create odd expressions by manipulating attested expressions found in a corpus. (For example, by taking a source sentence like \"Pierre Vinken will join the board as a nonexecutive director.\" and swapping in an NP taken from a distractor sentence, to yield for example \"Pierre Vinken will join the board as the striped zebra.\") A statistical language model, provided that it has enough capacity, could be expected to easily *detect* such manipulations (as the string as a whole presumably will be assigned a lower likelihood than the original version). It can however not contribute to an *explanation* of this oddness. (Which for the tiger example above would be something like \"tigers are concrete objects, financial institutions are abstract objects; nothing can be both a concrete and an abstract object\", and for the Vinken example would be something like \"being a striped zebras is not a function on an executive board\".) What is required here is a *conceptual* analysis that addresses the predication failure.\n",
    "\n",
    "While manipulated expressions can be constructed from any text corpus, using the corpora described here could potentially have some advantages. First, the expressions in these corpora are relatively simple (compared to newspaper text, for example), and so offer a simpler access to the phenomenon. Second, having the semantc annotation available (the segmented images) gives us more control over the distractor sentences and hence over the types of manipulation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adjectival Modification\n",
    "\n",
    "We'll start with adjectival modification / predication. As discussed above, visual genome provides separate annotation for attributes and names. We can use this to assemble pairs that have a certain chance to be semantically ill-formed. \n",
    "\n",
    "Here is a random sample showing first a pairing of attribute and name (adjective and noun) as it was annotated for an object, and then the same name with a randomly sampled attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                           green • trees || black • trees                           \n",
      "                          lounging • cat || blue • cat                              \n",
      "                            black • trim || long • trim                             \n",
      "                          orange • tower || abstract • tower                        \n",
      "                       signaling • light || for hotdog • light                      \n",
      "                          french • fries || off • fries                             \n",
      "                          blue • sticker || closed • sticker                        \n",
      "                            small • tree || the man's • tree                        \n",
      "                         red • red cloth || yellow • red cloth                      \n",
      "                   another • snowboarder || green • snowboarder                     \n",
      "                            wood • table || small • table                           \n",
      "                         man made • pond || glass • pond                            \n",
      "                           green • pants || sharp white • pants                     \n",
      "                         red • curlyhair || cooked • curlyhair                      \n",
      "                            large • bell || white • bell                            \n",
      "                             black • cap || gray • cap                              \n",
      "                           fluffy • coat || blue • coat                             \n",
      "                          small  • sheep || silver  • sheep                         \n",
      "                            brown • shoe || target • shoe                           \n",
      "                       Window • building || black  • building                       \n"
     ]
    }
   ],
   "source": [
    "# sampling actual and random ADJ N pairings\n",
    "def sample_NP_pair(attdf, objdf, n_filter=None):\n",
    "    ic, ii, oi, atts = attdf.sample()['i_corpus image_id obj_id attributes'.split()].values[0]\n",
    "    name = objdf[objdf['obj_id'] == oi]['name'].values[0]\n",
    "    # TODO: test that this name type has never been annotated with this attribute\n",
    "    \n",
    "    neg_atts = []\n",
    "    if n_filter is not None:\n",
    "        while len(neg_atts) == 0:\n",
    "            neg_atts = attdf.sample()['attributes'].values[0]\n",
    "            neg_att = np.random.choice(neg_atts)\n",
    "            if neg_att in n_filter and name in n_filter[neg_att]:\n",
    "                neg_atts = []\n",
    "    else:\n",
    "        neg_atts = attdf.sample()['attributes'].values[0]\n",
    "        neg_att = np.random.choice(neg_atts)\n",
    "\n",
    "    pos = np.random.choice(atts) + u' • ' + name\n",
    "    neg = neg_att + u' • ' + name\n",
    "    return pos, neg\n",
    "\n",
    "n_pairs = 20\n",
    "for pos, neg in [sample_NP_pair(df['vgattdf'], df['vgobjdf']) for _ in range(n_pairs)]:\n",
    "    print(u'{:>40} || {:<40}'.format(pos, neg))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(The character \"·\" is used here to separate attribute and name; this makes visible that sometimes the names aren't properly segmented and contain what should be split off as attributes. If this data were to be used for this task, some post processing would need to be performed.)\n",
    "\n",
    "As this shows, simply randomly sampling attributes from the corpus often does not lead to obviously incoherent pairings. There are attributes that seem quite generally applicable (e.g., colours). We can try to make it more likely that the constructed pairing is incoherent by at least checking that it is not attested in the corpus as a whole."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [],
   "source": [
    "# prepare mapping of attributes to attested modified Ns, for sample. SLOW!\n",
    "# Would other way round be faster? Map from N to all ADJ that occur with it. Same?\n",
    "n_atts_to_sample = 20000\n",
    "attr2den = defaultdict(list)\n",
    "_ = [attr2den[a].append(oi) for oi, atts in df['vgattdf']\\\n",
    "     .sample(n_atts_to_sample)[['obj_id', 'attributes']].values for a in atts]\n",
    "\n",
    "attr2applicable_types = dict([(k, set([get_obj_key(df['vgobjdf'], oi, key='name') for oi in ois]))\n",
    "                                for  k, ois in attr2den.items()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                            dark • water || plaid • water                           \n",
      "                            black • bird || large • bird                            \n",
      "                        chicken  • salad || multiple • salad                        \n",
      "                            man's • nose || rustic • nose                           \n",
      "                          rainbow • flag || silver • flag                           \n",
      "                              tan  • hat || utility • hat                           \n",
      "                           brown • shirt || off white • shirt                       \n",
      "                          large • window || gray • window                           \n",
      "                          yellow • train || clear • train                           \n",
      "                          silver • spoon || green • spoon                           \n",
      "                            white • line || whole • line                            \n",
      "                            wire • fence || mud • fence                             \n",
      "                          red • red sign || blue • red sign                         \n",
      "                       concrete • stairs || here • stairs                           \n",
      "                         chipped • paint || lighted • paint                         \n",
      "                           open • window ||  blue • window                          \n",
      "                            black • boot || stop, pause and play • boot             \n",
      "                            black • ears || green • ears                            \n",
      "                             tan • snout || silver • snout                          \n",
      "                   wearing blue • person || ski • person                            \n"
     ]
    }
   ],
   "source": [
    "# sampling ADJ N pairings; ensuring that neg ADJ N is not attested\n",
    "n_pairs = 20\n",
    "for pos, neg in [sample_NP_pair(df['vgattdf'], df['vgobjdf'],\n",
    "                                n_filter=attr2applicable_types) for _ in range(n_pairs)]:\n",
    "    print(u'{:>40} || {:<40}'.format(pos, neg))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Intuitively, these samples are somewhat better (in that they are odder). (To what degree this holds of course is an empirical question that could answered by collecting acceptability ratings via crowdsourcing, for example.) We can at least use this data to ask what the place should be for modelling this oddness. Is it just a frequency effect? (We can assume that these are rarer pairings, as they at least don't occur in this corpus.) Is it world knowledge that makes some pairings appear off (e.g., \"khaki sky\")? Is there a conceptual mismatch? And if so, is that different from being unlikely based on world knowledge?\n",
    "\n",
    "* **Dataset:** ADJ + N\n",
    "* **Negative Instances:** ADJ sampled from different context\n",
    "* **Source:** visual genome, derived\n",
    "* **Uses:** predict which predications are semantically odd\n",
    "\n",
    "These questions also arise when larger phrases are manipulated, as we show in the following."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## V, VP, PP application to NPs\n",
    "\n",
    "The following examples first show a source expression taken from the corpus, and then a manipulated version where one NP of the source expression is replaced by an NP from a randomly sampled distractor expression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          a garden with different plants || building with different plants          \n",
      " small white air conditioner in the wall || small white air the computer            \n",
      "                    window of a building || window of the airplane                  \n",
      "            the word \"gassing\" on a sign || the building on a sign                  \n",
      "Bag on top of table with book on top of it. || Bag on top of table with book on top of two rivets .\n",
      "                   Numbers on the clock. || white object on the clock .             \n",
      "                     Dirt on the ground. || Dirt on a nose .                        \n",
      " The tennis racket in the player's hand. || man in the player 's hand .             \n",
      "                         mane of a zebra || door of a zebra                         \n",
      "                      bricks on the wall || bricks on the arm                       \n",
      "                      waves in the water || waves in a motorcycle                   \n",
      "             the windshield of a trolley || the windshield of a white helmet        \n",
      "                         head of the boy || a kite of the boy                       \n",
      "           man and woman posing together || box posing together                     \n",
      "                      bottom of left bun || bottom of blender                       \n",
      "                    A tree in the woods. || A tree in sunglasses .                  \n",
      "                  Man wearing gray pants || Man wearing trees along edge            \n",
      "                blue jeans worn by woman || blue jeans worn by tail                 \n",
      "          A candle sitting on the shelf  || A candle sitting on the water           \n",
      "                      A tile in a floor. || some grass in a floor .                 \n"
     ]
    }
   ],
   "source": [
    "# Take original phrase, randomly sampled other phrase, swap in NP from latter\n",
    "parser = CoreNLPParser(url=\"http://localhost:9000\")\n",
    "def parse_phrase(parser, phrase):\n",
    "    return list(parser.raw_parse(phrase))[0]\n",
    "\n",
    "def is_cat(tree, cat='NP'):\n",
    "    return tree.label() == cat\n",
    "\n",
    "def traverse_and_copy(tree, replacement, target_np):\n",
    "    return _traverse_and_copy(tree, replacement, target_np, 0)\n",
    "\n",
    "def _traverse_and_copy(tree, replacement, target_np, nps_seen):\n",
    "    out_tree = []\n",
    "    for subtree in tree:\n",
    "        if type(subtree) == nltk.tree.Tree:\n",
    "            if is_cat(subtree):\n",
    "                nps_seen += 1\n",
    "                if nps_seen == target_np:\n",
    "                    out_tree.append(replacement)\n",
    "                else:\n",
    "                    out_tree.append(nltk.tree.Tree('NP', _traverse_and_copy(subtree, replacement,\n",
    "                                                              target_np, nps_seen)))\n",
    "            else:\n",
    "                out_tree.append(nltk.tree.Tree(subtree.label(), _traverse_and_copy(subtree, replacement,\n",
    "                                                      target_np, nps_seen)))\n",
    "        else:\n",
    "            out_tree.append(subtree)\n",
    "    return out_tree\n",
    "\n",
    "def swap_in_np(original, distractor):\n",
    "    phrase_pd = parse_phrase(parser, original)\n",
    "    n_nps = len(list(phrase_pd.subtrees(filter=is_cat)))\n",
    "    start_np = 1\n",
    "    if phrase_pd[0,0].label() == 'NP':\n",
    "        start_np = 2\n",
    "    target_np = random.randint(start_np, n_nps)\n",
    "\n",
    "    distr_phrase_pd = parse_phrase(parser, distractor)\n",
    "    distr_nps = list(distr_phrase_pd.subtrees(filter=is_cat))\n",
    "    replacement = random.choice(distr_nps[1:])\n",
    "\n",
    "    tmp = traverse_and_copy(phrase_pd, replacement, target_np)[0]\n",
    "    manipulated_string = ' '.join(tmp.leaves())\n",
    "    return manipulated_string\n",
    "\n",
    "for _ in range(20):\n",
    "    phrase = df['vgpregdf'].sample()['phrase'].values[0]\n",
    "    rand_distr_phrase = df['vgpregdf'].sample()['phrase'].values[0]\n",
    "    try:\n",
    "        manipulated_phrase = swap_in_np(phrase, rand_distr_phrase)\n",
    "    except:\n",
    "        continue\n",
    "    print(u'{:>40} || {:<40}'.format(phrase, manipulated_phrase))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The semantic oddness, to the degree there is any for the samples, resides in the relation that is confronted with a non-attested pair of arguments. (Unlike in the previous section, we do not actually check for that; given the much wider range of NPs, it is very likely in any case that randomly sampling will lead to a non-attested pair. More control would be possible, if desired.)\n",
    "\n",
    "To give an example, one run (remember that these examples are indeed generated randomly and will be different for each run of the notebook) resulted in the expression \"white clouds in a bear\". What is it that makes this at least curious, more so than the source expression \"white clouds in blue sky\"? We know that clouds are a metereological phenomenon, and as such there is a very limited range of entities that they can be \"in\". To assign any meaning to the manipulated phrase, we must coerce \"white clouds\" into something like \"pictures of white clouds\"; even then, \"in\" isn't quite the right choice of preposition. \n",
    "\n",
    "We will leave it at that for now and only note that this simple manipulation seems to create an interesting challenge that only begins with predicting *that* something is odd (which a language model should be able to do) and that in its full stage involves quite sophisticated conceptual knowledge."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Dataset:** complex phrases with NPs swapped out\n",
    "* **Negative Instances:** NPs taken from other expressions, potentially selected via image similarity\n",
    "* **Source:** visual genome, derived\n",
    "* **Uses:** predict which predications are semantically odd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References\n",
    "\n",
    "[<a id=\"cit-chomsky:synstruc\" href=\"#call-chomsky:synstruc\">1</a>] Noam Chomsky, ``_Syntactic Structures_'',  1957.\n",
    "\n",
    "[<a id=\"cit-asher_2011\" href=\"#call-asher_2011\">2</a>] Nicholas Asher, ``_Lexical Meaning in Context: A Web of Words_'',  2011.\n",
    "\n",
    "[<a id=\"cit-Vecchi2017\" href=\"#call-Vecchi2017\">3</a>] Vecchi Eva M., Marelli Marco, Zamparelli Roberto <em>et al.</em>, ``_Spicy Adjectives and Nominal Donkeys: Capturing Semantic Deviance Using Compositionality in Distributional Spaces_'', Cognitive Science, vol. , number , pp. ,  2017.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "author": "Natural Language Semantics with Pictures: Some Language & Vision Datasets and Potential Uses for Computational Semantics",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "../Common/joint.bib",
   "cite_by": "number",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "alt-ctrl-e",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "274px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}