2_Complex_Queries_Basic_Visualization_py3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Complex Queries and Basic Visualization\n",
    "\n",
    "This notebook will cover how to make more complex queries, using both basic HTTP requests and using sharepa - the SHARE parsing and analysis library.\n",
    "\n",
    "We'll also go over aggregations, or queries that will return summary statistics about the whole dataset. We'll use those aggregations to make some simple data visualizations using pandas and matplotlib."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Here we'll define a helper function and specify the SHARE API url that we'll use for querying.\n",
    "\n",
    "We'll also define another helper function to nicely print out our results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import requests\n",
    "\n",
    "SHARE_API = 'https://staging-share.osf.io/api/search/abstractcreativework/_search'\n",
    "\n",
    "def query_share(url, query):\n",
    "    # A helper function that will use the requests library,\n",
    "    # pass along the correct headers,\n",
    "    # and make the query we want\n",
    "    headers = {'Content-Type': 'application/json'}\n",
    "    data = json.dumps(query)\n",
    "    return requests.post(url, headers=headers, data=data, verify=False).json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def print_numbered_results(results):\n",
    "    print(\n",
    "        'There are {} total results and {} results on this page'.format(\n",
    "            results['hits']['total'],\n",
    "            len(results['hits']['hits'])\n",
    "        )\n",
    "    )\n",
    "    print('---------------')    \n",
    "    for result in enumerate(results['hits']['hits']):\n",
    "        print('{}. {}'.format(result[0] + 1, result[1]['_source']['title']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Complex Queries\n",
    "\n",
    "### Pagination\n",
    "\n",
    "One request to the SHARE API will return just 10 results by default. To get more results, you can use the \"size\" parameter in your request, or paginate through the results you get back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "basic_query = {\n",
    "    \"query\": {\n",
    "        \"query_string\": {\n",
    "            \"query\": \"frogs\"\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "query_results = query_share(SHARE_API, basic_query)\n",
    "\n",
    "print_numbered_results(query_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get more results either by changing the number of results returned, or by paginating through the results.\n",
    "\n",
    "First, we'll return 20 results by specifying the size in our query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "basic_query = {\n",
    "    \"query\": {\n",
    "        \"query_string\": {\n",
    "            \"query\": \"frogs\"\n",
    "        }\n",
    "    },\n",
    "    \"size\": 20\n",
    "}\n",
    "\n",
    "query_results = query_share(SHARE_API, basic_query)\n",
    "\n",
    "print_numbered_results(query_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also paginate through results by specifying the place to start in all of the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "basic_query = {\n",
    "    \"query\": {\n",
    "        \"query_string\": {\n",
    "            \"query\": \"frogs\"\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "query_results = query_share(SHARE_API, basic_query)\n",
    "print_numbered_results(query_results)\n",
    "\n",
    "print('------------------------------------------')\n",
    "print('*** Making another query for the next page ***')\n",
    "print('*** These next titles will be different! ***')\n",
    "print('------------------------------------------')\n",
    "\n",
    "basic_query['from'] = 10  # Add the 'from' parameter to the query to pick up at the next page of results\n",
    "\n",
    "query_results = query_share(SHARE_API, basic_query)\n",
    "print_numbered_results(query_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pagination with sharepa\n",
    "\n",
    "You can also use sharepa to paginate through all of the results in your query, and to access slices of your query at any time. The ShareSearch object returns a generator that you can use to access all results, using slices.\n",
    "\n",
    "First, we'll redefine our helper function for nicer printing with data returned from sharepa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def print_numbered_sharepa_results(search_obj):\n",
    "    results = search_obj.execute()\n",
    "    print(\n",
    "        'There are {} total results and {} results on this page'.format(\n",
    "            search_obj.count(),\n",
    "            len(results.hits)\n",
    "        )\n",
    "    )\n",
    "    print('---------------')    \n",
    "    for result in enumerate(results.hits):\n",
    "        print('{}. {}'.format(result[0] + 1, result[1]['title']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sharepa import ShareSearch\n",
    "from sharepa.helpers import pretty_print\n",
    "\n",
    "frogs_search = ShareSearch()\n",
    "\n",
    "frogs_search = frogs_search.query(\n",
    "    'query_string',\n",
    "    query='frogs'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print_numbered_sharepa_results(frogs_search)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print_numbered_sharepa_results(frogs_search[10:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aggregations\n",
    "\n",
    "While searching for individual results is useful, sharepa also lets you make aggregation queries that give you results across the entirety of the SHARE dataset at once. This is useful if you're curious about the completeness of data sets.\n",
    "\n",
    "For example, we can find the number of documents per source that are missing tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "missing_tags_aggregation = {\n",
    "    \"query\": {\n",
    "        \"bool\": {\n",
    "            \"must_not\": {\n",
    "                \"exists\": {\n",
    "                    \"field\": \"tags\"\n",
    "                  }\n",
    "            }\n",
    "        }\n",
    "    },\n",
    "    \"aggregations\": {\n",
    "        \"sources\": {\n",
    "            \"terms\": {\n",
    "                \"field\": \"sources\", # A field where the SHARE source is stored                \n",
    "                \"min_doc_count\": 0, \n",
    "                \"size\": 0  # Will return all sources, regardless if there are results\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "results_without_tags = query_share(SHARE_API, missing_tags_aggregation)\n",
    "\n",
    "missing_tags_counts = results_without_tags['aggregations']['sources']['buckets']\n",
    "\n",
    "for source in missing_tags_counts:\n",
    "    print('{} has {} documents without tags'.format(source['key'], source['doc_count'], ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This information isn't terribly useful if we don't also know how many documents are in each source.\n",
    "\n",
    "Let's get that information as well, along stats for what percentage of documents from each source are missing titles. \n",
    "\n",
    "We'll do this with an elasticsearch \"sigificant terms\" aggregation. We're only interested in results that have 1 document or more, meaning all documents from the other sources have titles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "no_tags_query = {\n",
    "    \"query\": {\n",
    "        \"bool\": {\n",
    "            \"must_not\": {\n",
    "                \"exists\": {\n",
    "                    \"field\": \"tags\"\n",
    "                  }\n",
    "            }\n",
    "        }\n",
    "    },\n",
    "    \"aggs\": {\n",
    "        \"sources\":{\n",
    "            \"significant_terms\":{\n",
    "                \"field\": \"sources\", # A field where the SHARE source is stored                \n",
    "                \"min_doc_count\": 0, \n",
    "                \"size\": 0,  # Will return all sources, regardless if there are results\n",
    "                \"percentage\": {} # This will make the \"score\" parameter a percentage\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "docs_with_no_tags_results = query_share(SHARE_API, no_tags_query)\n",
    "docs_with_no_tags = docs_with_no_tags_results['aggregations']['sources']['buckets']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for source in docs_with_no_tags:\n",
    "    print(\n",
    "        '{}% (or {}/{}) of documents from {} have no tags'.format(\n",
    "            format(source['score']*100, '.2f'),\n",
    "            source['doc_count'],\n",
    "            source['bg_count'],\n",
    "            source['key']\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Aggregations with sharepa\n",
    "\n",
    "You can also use sharepa to do aggregations.\n",
    "\n",
    "Let's make a sharepa search object that will give us the number of documents per sourcethat don't have language specified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "no_language_search = ShareSearch()\n",
    "\n",
    "no_language_search = no_language_search.query(\n",
    "    'bool',\n",
    "    must_not={\"exists\": {\"field\": \"language\"}}\n",
    ")\n",
    "\n",
    "no_language_search.aggs.bucket(\n",
    "    'sources',  # Every aggregation needs a name\n",
    "    'significant_terms',  # There are many kinds of aggregations\n",
    "    field='sources',  # We store the source of a document in its type, so this will aggregate by source\n",
    "    min_doc_count=1,\n",
    "    percentage={},\n",
    "    size=0\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see which query is actually going to be sent to elasticsearch by printing out the query. This is very similar to the queries we were defining by hand up above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pretty_print(no_language_search.to_dict())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "aggregated_results = no_language_search.execute()\n",
    "\n",
    "for source in aggregated_results.aggregations['sources']['buckets']:\n",
    "    print(\n",
    "        '{}% of documents from {} do not have language'.format(\n",
    "            format(source['score']*100, '.2f'),\n",
    "            source['key'] \n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Top tags \n",
    "\n",
    "Let's do an elasticsearch query to find out what the most used tags are used in the dataset across all sources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "top_tag_search = ShareSearch()\n",
    "\n",
    "top_tag_search.aggs.bucket(\n",
    "    'tagsTermFilter',  # Every aggregation needs a name\n",
    "    'terms',  # There are many kinds of aggregations\n",
    "    field='tags',  # We store the source of a document in its type, so this will aggregate by source\n",
    "    min_doc_count=1,\n",
    "    exclude= \"of|and|or\",\n",
    "    size=10\n",
    ")\n",
    "\n",
    "# pretty_print(top_tag_search.to_dict())\n",
    "\n",
    "top_tag_results_executed = top_tag_search.execute()\n",
    "top_tag_results = top_tag_results_executed.aggregations.tagsTermFilter.to_dict()['buckets']\n",
    "\n",
    "pretty_print(top_tag_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Plotting\n",
    "\n",
    "Sharepa has some basic functions to get you started making plots using matplotlib and pandas.\n",
    "\n",
    "### Creating a dataframe from sharepa data\n",
    "\n",
    "Raw sharepa data is in the same format as elasticsearch results, represented as a nested structure. To convert the data into a format that pandas can recognize, we have to convert it into a dataframe.\n",
    "\n",
    "Let's take our top tags aggregation, make it into a pandas data frame, and plot a bar graph. Then, we'll plot the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "top_tags_dataframe = pd.DataFrame(top_tag_results)\n",
    "top_tags_dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from matplotlib import pyplot\n",
    "%matplotlib inline\n",
    "\n",
    "top_tags_dataframe.plot(kind='bar', x='key', y='doc_count')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Complex Queries and Dataframes\n",
    "\n",
    "Let's plot the number of document that each source has. We'll limit it to the top 30 sources to make sure that the graph is readable. Here we'll use the sharepa helper function bucket_to_dataframe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sharepa import bucket_to_dataframe\n",
    "\n",
    "all_results = ShareSearch()\n",
    "\n",
    "all_results = all_results.query(\n",
    "    'query_string', # Type of query, will accept a lucene query string\n",
    "    query='*', # This lucene query string will find all documents that don't have tags\n",
    "    analyze_wildcard=True  # This will make elasticsearch pay attention to the asterisk (which matches anything)\n",
    ")\n",
    "\n",
    "all_results.aggs.bucket(\n",
    "    'sources',  # Every aggregation needs a name\n",
    "    'terms',  # There are many kinds of aggregations, terms is a pretty useful one though\n",
    "    field='sources',  # We store the source of a document in its type, so this will aggregate by source\n",
    "    size=0,  # These are just to make sure we get numbers for all the sources, to make it easier to combine graphs\n",
    "    min_doc_count=0\n",
    ")\n",
    "\n",
    "all_results = all_results.execute()\n",
    "\n",
    "all_results_frame = bucket_to_dataframe('# documents by source', all_results.aggregations.sources.buckets)\n",
    "all_results_frame_sorted = all_results_frame.sort(ascending=False,  columns='# documents by source')\n",
    "all_results_frame_sorted[:30].plot(kind='bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can choose different types of plots to generate. Here, we'll make a pie chart of the data sources with the top 10 most results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "all_results_frame_sorted[:10].plot(kind='pie', y=\"# documents by source\", legend=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}