Exersice5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ef8b7a4b-8c6a-453d-821b-788d9f15901e",
   "metadata": {},
   "source": [
    "<h1>Exercise 5</h1>\n",
    "TKO_7093 Statistical Data Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "e0a7d2f6-fdad-4897-8103-bb89a3417fab",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import scipy\n",
    "from scipy import stats\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce454d77-5f95-4a46-9307-b53f0f8d2c60",
   "metadata": {},
   "source": [
    "<h2>1.</h2>\n",
    "Load the data available in the file 51-data.csv.<br><br>\n",
    "\n",
    "1. Find invalid values in the data and replace them either with a correct value (if possible) or with NaN.<br>\n",
    "2. Replace all missing values of the purchases variable with zero.<br>\n",
    "3. Use median imputation to fill in all missing values of the retention_time variable.<br>\n",
    "(BONUS) Group the observations by sex and location before calculating the substitute median(s)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "8dc2132b-7b9d-43a2-a8c3-752f5922072b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sex</th>\n",
       "      <th>location</th>\n",
       "      <th>purchases</th>\n",
       "      <th>retention_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>female</td>\n",
       "      <td>Helsinki</td>\n",
       "      <td>7.0</td>\n",
       "      <td>57.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>female</td>\n",
       "      <td>Helsinki</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>female</td>\n",
       "      <td>Turku</td>\n",
       "      <td>8.0</td>\n",
       "      <td>50.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>male</td>\n",
       "      <td>Helsinki</td>\n",
       "      <td>5.0</td>\n",
       "      <td>31.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>female</td>\n",
       "      <td>Helsinki</td>\n",
       "      <td>2.0</td>\n",
       "      <td>120.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>male</td>\n",
       "      <td>Turku</td>\n",
       "      <td>3.0</td>\n",
       "      <td>40.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>male</td>\n",
       "      <td>Helsinki</td>\n",
       "      <td>3.0</td>\n",
       "      <td>36.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>female</td>\n",
       "      <td>Helsinki</td>\n",
       "      <td>6.0</td>\n",
       "      <td>59.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>male</td>\n",
       "      <td>Helsinki</td>\n",
       "      <td>5.0</td>\n",
       "      <td>31.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199</th>\n",
       "      <td>female</td>\n",
       "      <td>Helsinki</td>\n",
       "      <td>6.0</td>\n",
       "      <td>181.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>200 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        sex  location  purchases  retention_time\n",
       "0    female  Helsinki        7.0            57.1\n",
       "1    female  Helsinki        5.0             NaN\n",
       "2    female     Turku        8.0            50.6\n",
       "3      male  Helsinki        5.0            31.7\n",
       "4    female  Helsinki        2.0           120.0\n",
       "..      ...       ...        ...             ...\n",
       "195    male     Turku        3.0            40.5\n",
       "196    male  Helsinki        3.0            36.9\n",
       "197  female  Helsinki        6.0            59.2\n",
       "198    male  Helsinki        5.0            31.8\n",
       "199  female  Helsinki        6.0           181.9\n",
       "\n",
       "[200 rows x 4 columns]"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loc = r'C:\\Users\\peppi\\Desktop\\lipasto\\statisticalDataAnalysis\\51-data.csv'\n",
    "df = pd.read_csv(loc, sep=\",\", na_values=\"NA\", header=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "id": "143fb374-2ea9-48be-ba27-cff045d02289",
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "'<' not supported between instances of 'str' and 'int'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "Input \u001b[1;32mIn [161]\u001b[0m, in \u001b[0;36m<cell line: 5>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      3\u001b[0m df\u001b[38;5;241m.\u001b[39mloc[df\u001b[38;5;241m.\u001b[39mlocation \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124m33100\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mlocation\u001b[39m\u001b[38;5;124m'\u001b[39m] \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mTampere\u001b[39m\u001b[38;5;124m'\u001b[39m\n\u001b[0;32m      4\u001b[0m df\u001b[38;5;241m.\u001b[39mloc[df\u001b[38;5;241m.\u001b[39mlocation \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124m20100\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mlocation\u001b[39m\u001b[38;5;124m'\u001b[39m] \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mTurku\u001b[39m\u001b[38;5;124m'\u001b[39m\n\u001b[1;32m----> 5\u001b[0m df\u001b[38;5;241m.\u001b[39mloc[\u001b[43mdf\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mretention_time\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m<\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;241;43m0\u001b[39;49m, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mretention_time\u001b[39m\u001b[38;5;124m'\u001b[39m] \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39mnan\n\u001b[0;32m      6\u001b[0m df\u001b[38;5;241m.\u001b[39mdescribe()\n",
      "File \u001b[1;32m~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\pandas\\core\\ops\\common.py:70\u001b[0m, in \u001b[0;36m_unpack_zerodim_and_defer.<locals>.new_method\u001b[1;34m(self, other)\u001b[0m\n\u001b[0;32m     66\u001b[0m             \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mNotImplemented\u001b[39m\n\u001b[0;32m     68\u001b[0m other \u001b[38;5;241m=\u001b[39m item_from_zerodim(other)\n\u001b[1;32m---> 70\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mmethod\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mother\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32m~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\pandas\\core\\arraylike.py:48\u001b[0m, in \u001b[0;36mOpsMixin.__lt__\u001b[1;34m(self, other)\u001b[0m\n\u001b[0;32m     46\u001b[0m \u001b[38;5;129m@unpack_zerodim_and_defer\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m__lt__\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m     47\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m__lt__\u001b[39m(\u001b[38;5;28mself\u001b[39m, other):\n\u001b[1;32m---> 48\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_cmp_method\u001b[49m\u001b[43m(\u001b[49m\u001b[43mother\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43moperator\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mlt\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32m~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\pandas\\core\\series.py:5623\u001b[0m, in \u001b[0;36mSeries._cmp_method\u001b[1;34m(self, other, op)\u001b[0m\n\u001b[0;32m   5620\u001b[0m rvalues \u001b[38;5;241m=\u001b[39m extract_array(other, extract_numpy\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m, extract_range\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n\u001b[0;32m   5622\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m np\u001b[38;5;241m.\u001b[39merrstate(\u001b[38;5;28mall\u001b[39m\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mignore\u001b[39m\u001b[38;5;124m\"\u001b[39m):\n\u001b[1;32m-> 5623\u001b[0m     res_values \u001b[38;5;241m=\u001b[39m \u001b[43mops\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcomparison_op\u001b[49m\u001b[43m(\u001b[49m\u001b[43mlvalues\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mrvalues\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mop\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m   5625\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_construct_result(res_values, name\u001b[38;5;241m=\u001b[39mres_name)\n",
      "File \u001b[1;32m~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\pandas\\core\\ops\\array_ops.py:283\u001b[0m, in \u001b[0;36mcomparison_op\u001b[1;34m(left, right, op)\u001b[0m\n\u001b[0;32m    280\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m invalid_comparison(lvalues, rvalues, op)\n\u001b[0;32m    282\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m is_object_dtype(lvalues\u001b[38;5;241m.\u001b[39mdtype) \u001b[38;5;129;01mor\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(rvalues, \u001b[38;5;28mstr\u001b[39m):\n\u001b[1;32m--> 283\u001b[0m     res_values \u001b[38;5;241m=\u001b[39m \u001b[43mcomp_method_OBJECT_ARRAY\u001b[49m\u001b[43m(\u001b[49m\u001b[43mop\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mlvalues\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mrvalues\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m    285\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m    286\u001b[0m     res_values \u001b[38;5;241m=\u001b[39m _na_arithmetic_op(lvalues, rvalues, op, is_cmp\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n",
      "File \u001b[1;32m~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\pandas\\core\\ops\\array_ops.py:73\u001b[0m, in \u001b[0;36mcomp_method_OBJECT_ARRAY\u001b[1;34m(op, x, y)\u001b[0m\n\u001b[0;32m     71\u001b[0m     result \u001b[38;5;241m=\u001b[39m libops\u001b[38;5;241m.\u001b[39mvec_compare(x\u001b[38;5;241m.\u001b[39mravel(), y\u001b[38;5;241m.\u001b[39mravel(), op)\n\u001b[0;32m     72\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m---> 73\u001b[0m     result \u001b[38;5;241m=\u001b[39m \u001b[43mlibops\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mscalar_compare\u001b[49m\u001b[43m(\u001b[49m\u001b[43mx\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mravel\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43my\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mop\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m     74\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m result\u001b[38;5;241m.\u001b[39mreshape(x\u001b[38;5;241m.\u001b[39mshape)\n",
      "File \u001b[1;32m~\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\pandas\\_libs\\ops.pyx:107\u001b[0m, in \u001b[0;36mpandas._libs.ops.scalar_compare\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;31mTypeError\u001b[0m: '<' not supported between instances of 'str' and 'int'"
     ]
    }
   ],
   "source": [
    "# 1.1\n",
    "df['purchases'] = df['purchases'].fillna(0)\n",
    "df.loc[df.location == '33100', 'location'] = 'Tampere'\n",
    "df.loc[df.location == '20100', 'location'] = 'Turku'\n",
    "# df.loc[df.retention_time < 0, 'retention_time'] = np.nan\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "42db583d-b3c6-4a4a-9f04-81dc35d50ad6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sex</th>\n",
       "      <th>location</th>\n",
       "      <th>purchases</th>\n",
       "      <th>retention_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>female</td>\n",
       "      <td>Helsinki</td>\n",
       "      <td>7.0</td>\n",
       "      <td>57.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>female</td>\n",
       "      <td>Helsinki</td>\n",
       "      <td>5.0</td>\n",
       "      <td>32.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>female</td>\n",
       "      <td>Turku</td>\n",
       "      <td>8.0</td>\n",
       "      <td>50.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>male</td>\n",
       "      <td>Helsinki</td>\n",
       "      <td>5.0</td>\n",
       "      <td>31.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>female</td>\n",
       "      <td>Helsinki</td>\n",
       "      <td>2.0</td>\n",
       "      <td>120.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>male</td>\n",
       "      <td>Turku</td>\n",
       "      <td>3.0</td>\n",
       "      <td>40.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>196</th>\n",
       "      <td>male</td>\n",
       "      <td>Helsinki</td>\n",
       "      <td>3.0</td>\n",
       "      <td>36.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197</th>\n",
       "      <td>female</td>\n",
       "      <td>Helsinki</td>\n",
       "      <td>6.0</td>\n",
       "      <td>59.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>male</td>\n",
       "      <td>Helsinki</td>\n",
       "      <td>5.0</td>\n",
       "      <td>31.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>199</th>\n",
       "      <td>female</td>\n",
       "      <td>Helsinki</td>\n",
       "      <td>6.0</td>\n",
       "      <td>181.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>200 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        sex  location  purchases  retention_time\n",
       "0    female  Helsinki        7.0            57.1\n",
       "1    female  Helsinki        5.0            32.9\n",
       "2    female     Turku        8.0            50.6\n",
       "3      male  Helsinki        5.0            31.7\n",
       "4    female  Helsinki        2.0           120.0\n",
       "..      ...       ...        ...             ...\n",
       "195    male     Turku        3.0            40.5\n",
       "196    male  Helsinki        3.0            36.9\n",
       "197  female  Helsinki        6.0            59.2\n",
       "198    male  Helsinki        5.0            31.8\n",
       "199  female  Helsinki        6.0           181.9\n",
       "\n",
       "[200 rows x 4 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 1.3\n",
    "df['retention_time'] = df['retention_time'].fillna((df['retention_time'].median()))\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0daf0b74-be01-4049-ad67-e129b9f5e473",
   "metadata": {},
   "source": [
    "<h2>2.</h2>\n",
    "Load the file GDS5037.soft (see the exercise 4.3) and filter the data such that it only contains the probe ids listed in the file 52-probes.ids and the subject ids listed in the files 52-control.ids and 52-asthma.ids.<br><br>\n",
    "\n",
    "1. Create a box plot of gene expression levels for each subject. Do you notice visible differences between the subjects or between the groups?\n",
    "\n",
    "2. Use a T-test to find differentially expressed genes (i.e. probes for which the means are different between the control and asthma groups). Adjust the p-values with the Benjamini-Hochberg method.\n",
    "\n",
    "3. Create histograms of the unadjusted and adjusted p-values. Why do these two histograms differ?\n",
    "\n",
    "4. How many differentially expressed genes (i.e. statistically significant differences) are there at the false discovery rate of 0.05?\n",
    "\n",
    "5. Sort the adjusted p-values in ascending order. Which genes do the first ten probes correspond to?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "id": "61a22542-48eb-4fc5-9100-f84bc0c9ffad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2.1\n",
    "soft = pd.read_csv('GDS5037.soft', sep=\"\\s+\" ,header=160, skipfooter=1, engine='python')\n",
    "df_soft = pd.DataFrame(soft)\n",
    "df_soft = df_soft.set_index('ID_REF')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "id": "a999c30a-948f-405e-8a88-8494c04b0caa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>GSM1068458</th>\n",
       "      <th>GSM1068459</th>\n",
       "      <th>GSM1068460</th>\n",
       "      <th>GSM1068461</th>\n",
       "      <th>GSM1068462</th>\n",
       "      <th>GSM1068463</th>\n",
       "      <th>GSM1068464</th>\n",
       "      <th>GSM1068465</th>\n",
       "      <th>GSM1068466</th>\n",
       "      <th>GSM1068467</th>\n",
       "      <th>...</th>\n",
       "      <th>GSM1068556</th>\n",
       "      <th>GSM1068557</th>\n",
       "      <th>GSM1068558</th>\n",
       "      <th>GSM1068559</th>\n",
       "      <th>GSM1068560</th>\n",
       "      <th>GSM1068561</th>\n",
       "      <th>GSM1068562</th>\n",
       "      <th>GSM1068563</th>\n",
       "      <th>GSM1068564</th>\n",
       "      <th>GSM1068565</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ID_REF</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>A_23_P100355</th>\n",
       "      <td>9.73450</td>\n",
       "      <td>9.82071</td>\n",
       "      <td>10.38230</td>\n",
       "      <td>10.43700</td>\n",
       "      <td>10.37030</td>\n",
       "      <td>10.59030</td>\n",
       "      <td>10.48990</td>\n",
       "      <td>10.76750</td>\n",
       "      <td>10.57110</td>\n",
       "      <td>9.91477</td>\n",
       "      <td>...</td>\n",
       "      <td>10.94870</td>\n",
       "      <td>10.28070</td>\n",
       "      <td>10.21060</td>\n",
       "      <td>10.48320</td>\n",
       "      <td>10.67700</td>\n",
       "      <td>10.76620</td>\n",
       "      <td>10.75330</td>\n",
       "      <td>10.59410</td>\n",
       "      <td>10.63990</td>\n",
       "      <td>10.53500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A_23_P101829</th>\n",
       "      <td>7.34121</td>\n",
       "      <td>8.00249</td>\n",
       "      <td>7.17964</td>\n",
       "      <td>7.53055</td>\n",
       "      <td>7.10362</td>\n",
       "      <td>7.07410</td>\n",
       "      <td>6.69177</td>\n",
       "      <td>7.37524</td>\n",
       "      <td>7.49978</td>\n",
       "      <td>7.34268</td>\n",
       "      <td>...</td>\n",
       "      <td>7.18555</td>\n",
       "      <td>7.18596</td>\n",
       "      <td>7.50311</td>\n",
       "      <td>7.61950</td>\n",
       "      <td>7.35386</td>\n",
       "      <td>7.63585</td>\n",
       "      <td>7.29750</td>\n",
       "      <td>7.18897</td>\n",
       "      <td>7.11162</td>\n",
       "      <td>7.49630</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A_23_P102235</th>\n",
       "      <td>12.12020</td>\n",
       "      <td>12.31900</td>\n",
       "      <td>12.42870</td>\n",
       "      <td>12.23300</td>\n",
       "      <td>12.14920</td>\n",
       "      <td>12.22640</td>\n",
       "      <td>12.16290</td>\n",
       "      <td>12.19920</td>\n",
       "      <td>12.26940</td>\n",
       "      <td>12.09810</td>\n",
       "      <td>...</td>\n",
       "      <td>12.28210</td>\n",
       "      <td>12.28820</td>\n",
       "      <td>12.29720</td>\n",
       "      <td>12.35510</td>\n",
       "      <td>12.30030</td>\n",
       "      <td>12.24170</td>\n",
       "      <td>12.40860</td>\n",
       "      <td>12.05510</td>\n",
       "      <td>12.23110</td>\n",
       "      <td>12.02790</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A_23_P102607</th>\n",
       "      <td>7.07051</td>\n",
       "      <td>6.91728</td>\n",
       "      <td>7.16459</td>\n",
       "      <td>6.19421</td>\n",
       "      <td>6.86695</td>\n",
       "      <td>6.84001</td>\n",
       "      <td>7.34184</td>\n",
       "      <td>7.09113</td>\n",
       "      <td>6.87948</td>\n",
       "      <td>6.67208</td>\n",
       "      <td>...</td>\n",
       "      <td>7.20520</td>\n",
       "      <td>7.04128</td>\n",
       "      <td>6.85812</td>\n",
       "      <td>6.87079</td>\n",
       "      <td>7.12792</td>\n",
       "      <td>6.55122</td>\n",
       "      <td>6.75280</td>\n",
       "      <td>7.29706</td>\n",
       "      <td>7.01142</td>\n",
       "      <td>6.79003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A_23_P102743</th>\n",
       "      <td>7.31329</td>\n",
       "      <td>6.38099</td>\n",
       "      <td>7.14409</td>\n",
       "      <td>6.72997</td>\n",
       "      <td>7.03673</td>\n",
       "      <td>6.75132</td>\n",
       "      <td>6.62888</td>\n",
       "      <td>6.63785</td>\n",
       "      <td>6.87825</td>\n",
       "      <td>6.71155</td>\n",
       "      <td>...</td>\n",
       "      <td>6.98354</td>\n",
       "      <td>7.11257</td>\n",
       "      <td>6.91354</td>\n",
       "      <td>7.21952</td>\n",
       "      <td>6.96467</td>\n",
       "      <td>6.99149</td>\n",
       "      <td>6.88292</td>\n",
       "      <td>6.80598</td>\n",
       "      <td>7.04567</td>\n",
       "      <td>6.81248</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A_32_P80255</th>\n",
       "      <td>9.72013</td>\n",
       "      <td>9.20204</td>\n",
       "      <td>8.81502</td>\n",
       "      <td>9.55290</td>\n",
       "      <td>9.88867</td>\n",
       "      <td>9.22377</td>\n",
       "      <td>9.70530</td>\n",
       "      <td>9.14508</td>\n",
       "      <td>9.19637</td>\n",
       "      <td>9.92583</td>\n",
       "      <td>...</td>\n",
       "      <td>9.53174</td>\n",
       "      <td>9.59771</td>\n",
       "      <td>9.75629</td>\n",
       "      <td>9.44850</td>\n",
       "      <td>9.54217</td>\n",
       "      <td>9.48188</td>\n",
       "      <td>9.00412</td>\n",
       "      <td>9.22302</td>\n",
       "      <td>9.81349</td>\n",
       "      <td>9.51958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A_32_P83784</th>\n",
       "      <td>9.33227</td>\n",
       "      <td>8.47086</td>\n",
       "      <td>9.01771</td>\n",
       "      <td>9.30044</td>\n",
       "      <td>8.65403</td>\n",
       "      <td>8.70030</td>\n",
       "      <td>8.26453</td>\n",
       "      <td>8.52626</td>\n",
       "      <td>8.62973</td>\n",
       "      <td>8.91668</td>\n",
       "      <td>...</td>\n",
       "      <td>8.59478</td>\n",
       "      <td>8.24108</td>\n",
       "      <td>8.50516</td>\n",
       "      <td>8.47277</td>\n",
       "      <td>8.62158</td>\n",
       "      <td>8.80231</td>\n",
       "      <td>8.72801</td>\n",
       "      <td>8.87190</td>\n",
       "      <td>8.61916</td>\n",
       "      <td>8.55743</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A_32_P902988</th>\n",
       "      <td>6.90708</td>\n",
       "      <td>8.63739</td>\n",
       "      <td>6.96754</td>\n",
       "      <td>7.68574</td>\n",
       "      <td>8.46000</td>\n",
       "      <td>8.44160</td>\n",
       "      <td>9.05813</td>\n",
       "      <td>9.64739</td>\n",
       "      <td>7.50242</td>\n",
       "      <td>9.80580</td>\n",
       "      <td>...</td>\n",
       "      <td>8.96406</td>\n",
       "      <td>9.65092</td>\n",
       "      <td>12.11820</td>\n",
       "      <td>8.48703</td>\n",
       "      <td>9.87380</td>\n",
       "      <td>11.01930</td>\n",
       "      <td>10.15100</td>\n",
       "      <td>8.71636</td>\n",
       "      <td>11.18370</td>\n",
       "      <td>9.48001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A_32_P9368</th>\n",
       "      <td>8.16925</td>\n",
       "      <td>9.71353</td>\n",
       "      <td>8.21237</td>\n",
       "      <td>8.02641</td>\n",
       "      <td>8.57337</td>\n",
       "      <td>8.79693</td>\n",
       "      <td>8.34014</td>\n",
       "      <td>8.47326</td>\n",
       "      <td>8.50564</td>\n",
       "      <td>7.63345</td>\n",
       "      <td>...</td>\n",
       "      <td>9.18419</td>\n",
       "      <td>8.49307</td>\n",
       "      <td>8.35604</td>\n",
       "      <td>8.17409</td>\n",
       "      <td>8.65650</td>\n",
       "      <td>9.05695</td>\n",
       "      <td>8.54015</td>\n",
       "      <td>8.81673</td>\n",
       "      <td>8.84424</td>\n",
       "      <td>8.31765</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>A_32_P96807</th>\n",
       "      <td>10.32810</td>\n",
       "      <td>9.57402</td>\n",
       "      <td>10.16880</td>\n",
       "      <td>9.97312</td>\n",
       "      <td>10.22750</td>\n",
       "      <td>9.10535</td>\n",
       "      <td>10.10640</td>\n",
       "      <td>9.90884</td>\n",
       "      <td>10.21520</td>\n",
       "      <td>9.99832</td>\n",
       "      <td>...</td>\n",
       "      <td>9.76844</td>\n",
       "      <td>9.80273</td>\n",
       "      <td>9.44613</td>\n",
       "      <td>9.55091</td>\n",
       "      <td>9.72005</td>\n",
       "      <td>9.83834</td>\n",
       "      <td>9.61187</td>\n",
       "      <td>9.79608</td>\n",
       "      <td>9.76633</td>\n",
       "      <td>9.68503</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>793 rows × 58 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              GSM1068458  GSM1068459  GSM1068460  GSM1068461  GSM1068462  \\\n",
       "ID_REF                                                                     \n",
       "A_23_P100355     9.73450     9.82071    10.38230    10.43700    10.37030   \n",
       "A_23_P101829     7.34121     8.00249     7.17964     7.53055     7.10362   \n",
       "A_23_P102235    12.12020    12.31900    12.42870    12.23300    12.14920   \n",
       "A_23_P102607     7.07051     6.91728     7.16459     6.19421     6.86695   \n",
       "A_23_P102743     7.31329     6.38099     7.14409     6.72997     7.03673   \n",
       "...                  ...         ...         ...         ...         ...   \n",
       "A_32_P80255      9.72013     9.20204     8.81502     9.55290     9.88867   \n",
       "A_32_P83784      9.33227     8.47086     9.01771     9.30044     8.65403   \n",
       "A_32_P902988     6.90708     8.63739     6.96754     7.68574     8.46000   \n",
       "A_32_P9368       8.16925     9.71353     8.21237     8.02641     8.57337   \n",
       "A_32_P96807     10.32810     9.57402    10.16880     9.97312    10.22750   \n",
       "\n",
       "              GSM1068463  GSM1068464  GSM1068465  GSM1068466  GSM1068467  ...  \\\n",
       "ID_REF                                                                    ...   \n",
       "A_23_P100355    10.59030    10.48990    10.76750    10.57110     9.91477  ...   \n",
       "A_23_P101829     7.07410     6.69177     7.37524     7.49978     7.34268  ...   \n",
       "A_23_P102235    12.22640    12.16290    12.19920    12.26940    12.09810  ...   \n",
       "A_23_P102607     6.84001     7.34184     7.09113     6.87948     6.67208  ...   \n",
       "A_23_P102743     6.75132     6.62888     6.63785     6.87825     6.71155  ...   \n",
       "...                  ...         ...         ...         ...         ...  ...   \n",
       "A_32_P80255      9.22377     9.70530     9.14508     9.19637     9.92583  ...   \n",
       "A_32_P83784      8.70030     8.26453     8.52626     8.62973     8.91668  ...   \n",
       "A_32_P902988     8.44160     9.05813     9.64739     7.50242     9.80580  ...   \n",
       "A_32_P9368       8.79693     8.34014     8.47326     8.50564     7.63345  ...   \n",
       "A_32_P96807      9.10535    10.10640     9.90884    10.21520     9.99832  ...   \n",
       "\n",
       "              GSM1068556  GSM1068557  GSM1068558  GSM1068559  GSM1068560  \\\n",
       "ID_REF                                                                     \n",
       "A_23_P100355    10.94870    10.28070    10.21060    10.48320    10.67700   \n",
       "A_23_P101829     7.18555     7.18596     7.50311     7.61950     7.35386   \n",
       "A_23_P102235    12.28210    12.28820    12.29720    12.35510    12.30030   \n",
       "A_23_P102607     7.20520     7.04128     6.85812     6.87079     7.12792   \n",
       "A_23_P102743     6.98354     7.11257     6.91354     7.21952     6.96467   \n",
       "...                  ...         ...         ...         ...         ...   \n",
       "A_32_P80255      9.53174     9.59771     9.75629     9.44850     9.54217   \n",
       "A_32_P83784      8.59478     8.24108     8.50516     8.47277     8.62158   \n",
       "A_32_P902988     8.96406     9.65092    12.11820     8.48703     9.87380   \n",
       "A_32_P9368       9.18419     8.49307     8.35604     8.17409     8.65650   \n",
       "A_32_P96807      9.76844     9.80273     9.44613     9.55091     9.72005   \n",
       "\n",
       "              GSM1068561  GSM1068562  GSM1068563  GSM1068564  GSM1068565  \n",
       "ID_REF                                                                    \n",
       "A_23_P100355    10.76620    10.75330    10.59410    10.63990    10.53500  \n",
       "A_23_P101829     7.63585     7.29750     7.18897     7.11162     7.49630  \n",
       "A_23_P102235    12.24170    12.40860    12.05510    12.23110    12.02790  \n",
       "A_23_P102607     6.55122     6.75280     7.29706     7.01142     6.79003  \n",
       "A_23_P102743     6.99149     6.88292     6.80598     7.04567     6.81248  \n",
       "...                  ...         ...         ...         ...         ...  \n",
       "A_32_P80255      9.48188     9.00412     9.22302     9.81349     9.51958  \n",
       "A_32_P83784      8.80231     8.72801     8.87190     8.61916     8.55743  \n",
       "A_32_P902988    11.01930    10.15100     8.71636    11.18370     9.48001  \n",
       "A_32_P9368       9.05695     8.54015     8.81673     8.84424     8.31765  \n",
       "A_32_P96807      9.83834     9.61187     9.79608     9.76633     9.68503  \n",
       "\n",
       "[793 rows x 58 columns]"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# choose based on files\n",
    "probes = pd.read_table('52-probes.ids', header=None)[0]\n",
    "controls = pd.read_table('52-control.ids', header=None)[0]\n",
    "asthma = pd.read_table('52-asthma.ids', header=None)[0]\n",
    "subjects = pd.concat([controls, asthma])\n",
    "\n",
    "df_full = df_soft.loc[probes,subjects]\n",
    "df_full"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "id": "57ced8d8-a2f3-4d72-83f3-8d2efeedeef3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SYT4\n",
      "CCL15\n",
      "BMP7\n",
      "DHX37\n",
      "BMP2K\n",
      "PLPP6\n",
      "SNTG2\n",
      "RORA\n",
      "NBL1\n",
      "PARM1\n"
     ]
    }
   ],
   "source": [
    "# Use a T-test to find differentially expressed genes (i.e. probes for which the means are different between the control and asthma groups). \n",
    "# Adjust the p-values with the Benjamini-Hochberg method.\n",
    "df_control = df_full.loc[:,controls]\n",
    "df_asthma = df_full.loc[:,asthma]\n",
    "\n",
    "dft = pd.DataFrame(columns=['result_of_t-test'], index=[df_control.index])\n",
    "for ind in df_control.index:\n",
    "    dft.at[ind, 'result_of_t-test'] = stats.ttest_ind( df_control.loc[ind, :], df_asthma.loc[ind, :] ).pvalue\n",
    "    \n",
    "newdf = dft.sort_values(by='result_of_t-test')\n",
    "ids = newdf.head(10).index.values\n",
    "for id in ids:\n",
    "    print(df_soft.loc[id].IDENTIFIER)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff727aa5-e295-4135-a24c-83f50b110b57",
   "metadata": {},
   "source": [
    "<h2>3.</h2>\n",
    "Load the data available in the file 53-data.csv, which contains daily numbers of cyclists spotted on selected streets.<br><br>\n",
    "\n",
    "For each weekday (Monday - Sunday), calculate the mean daily number of cyclists for each street. Plot the means as a bar plot.<br><br>\n",
    "\n",
    "For each month (January - December), calculate the mean daily number of cyclists for each street. Plot the means as a bar plot.<br><br>\n",
    "\n",
    "Calculate the Spearman's correlation coefficient and create a scatter plot for each pair of streets.<br><br>\n",
    "\n",
    "What information do these plots reveal?<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f23543c-6104-40bc-993a-4643ee3410c5",
   "metadata": {},
   "source": [
    "<h2>4.</h2>\n",
    "An advertisement company followed online customers to discover how effectively advertisements lure them to spend money. The data files 54-image.csv and 54-video.csv contain information on how much customers spent in total after clicking on advertisements.<br><br>\n",
    "\n",
    "Is there statistical evidence to claim that the total amount spent by customers is different if they click on image advertisements than on video advertisements?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1ba5961-95ac-400d-ac88-6609078e36dd",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}