data_detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Task2 数据读取与数据分析**\n",
    "\n",
    "本章主要内容为数据读取和数据分析，具体使用`Pandas`库完成数据读取操作，并对赛题数据进行分析构成。\n",
    "\n",
    "### **学习目标**\n",
    "\n",
    "- 学习使用`Pandas`读取赛题数据\n",
    "- 分析赛题数据的分布规律\n",
    "\n",
    "### **数据读取**\n",
    "\n",
    "赛题数据虽然是文本数据，每个新闻是不定长的，但任然使用csv格式进行存储。因此可以直接用`Pandas`完成数据读取的操作。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "train_df = pd.read_csv('./data/train_set.csv', sep='\\t', nrows=100,encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里的`read_csv`由三部分构成：\n",
    "\n",
    "- 读取的文件路径，这里需要根据改成你本地的路径，可以使用相对路径或绝对路径；\n",
    "\n",
    "- 分隔符`sep`，为每列分割的字符，设置为`\\t`即可；\n",
    "- 读取行数`nrows`，为此次读取文件的函数，是数值类型（由于数据集比较大，建议先设置为100）；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "   label                                               text\n0      2  2967 6758 339 2021 1854 3731 4109 3792 4149 15...\n1     11  4464 486 6352 5619 2465 4802 1452 3137 5778 54...\n2      3  7346 4068 5074 3747 5681 6093 1777 2226 7354 6...\n3      2  7159 948 4866 2109 5520 2490 211 3956 5520 549...\n4      3  3646 3055 3055 2490 4659 6065 3370 5814 2465 5...",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>label</th>\n      <th>text</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>2</td>\n      <td>2967 6758 339 2021 1854 3731 4109 3792 4149 15...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>11</td>\n      <td>4464 486 6352 5619 2465 4802 1452 3137 5778 54...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>3</td>\n      <td>7346 4068 5074 3747 5681 6093 1777 2226 7354 6...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2</td>\n      <td>7159 948 4866 2109 5520 2490 211 3956 5520 549...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>3</td>\n      <td>3646 3055 3055 2490 4659 6065 3370 5814 2465 5...</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上图是读取好的数据，是表格的形式。第一列为新闻的类别，第二列为新闻的字符。\n",
    "\n",
    "### **数据分析**\n",
    "\n",
    "在读取完成数据集后，我们还可以对数据集进行数据分析的操作。虽然对于非结构数据并不需要做很多的数据分析，但通过数据分析还是可以找出一些规律的。\n",
    "\n",
    "\n",
    "\n",
    "此步骤我们读取了所有的训练集数据，在此我们通过数据分析希望得出以下结论：\n",
    "\n",
    "- 赛题数据中，新闻文本的长度是多少？\n",
    "- 赛题数据的类别分布是怎么样的，哪些类别比较多？\n",
    "- 赛题数据中，字符分布是怎么样的？\n",
    "\n",
    "\n",
    "\n",
    "#### **句子长度分析**\n",
    "\n",
    "在赛题数据中每行句子的字符使用空格进行隔开，所以可以直接统计单词的个数来得到每个句子的长度。统计并如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n",
      "count     100.000000\n",
      "mean      872.320000\n",
      "std       923.138191\n",
      "min        64.000000\n",
      "25%       359.500000\n",
      "50%       598.000000\n",
      "75%      1058.000000\n",
      "max      7125.000000\n",
      "Name: text_len, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "train_df['text_len'] = train_df['text'].apply(lambda x: len(x.split(' ')))\n",
    "print(train_df['text_len'].describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对新闻句子的统计可以得出，本次赛题给定的文本比较长，每个句子平均由907个字符构成，最短的句子长度为2，最长的句子长度为57921。\n",
    "\n",
    "下图将句子长度绘制了直方图，可见大部分句子的长度都几种在2000以内。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    },
    {
     "data": {
      "text/plain": "Text(0.5, 1.0, 'Histogram of char count')"
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": "<Figure size 432x288 with 1 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAEWCAYAAABPON1ZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAXQUlEQVR4nO3deZSldX3n8ffHbnYIKNQYImDhhkEnKulBUMdxH9x1DjmC0YjL9JkYFxwdT6Mm0ZOTDGbUxEkyarskGRdEEYyBuBCXEIyi3WyyyLC1AgoUKAIuyPKdP55f4aWo7roFdat+0O/XOffcZ3++z+3bn3ru79lSVUiS+nWflS5AkrRlBrUkdc6glqTOGdSS1DmDWpI6Z1BLUucM6q1QknOTPGml61hJSV6Y5LIkNyZ5zCLme3uSj02yNmkug/peJsmmJE+bM+yIJKfO9lfVI6rqawssZzpJJVk9oVJX2ruA11TVzlV1xkoXs5L849M/g1orooM/AA8Ezl3JAjr4DHQPYVBvhUb3upMcmGRDkuuTXJXkPW2yU9r7da154OAk90nytiTfS3J1kv+bZNeR5f5eG3dtkj+cs563JzkuyceSXA8c0db9jSTXJflhkr9Osu3I8irJq5NcmOSGJH+S5MFJ/q3V+6nR6eds47y1JtkuyY3AKuCsJBdvZv5HJDk5yY/a5/KWkdHbtuXd0JqR1ozMty7JxW3ceUleODLuiCRfT/IXSa4F3j7PelclecvIMjYm2buNe1ySbyf5SXt/3Hz/piOf98da9+yvo5cl+X6Sa5K8tY07BHgL8KL273zWfJ+HVlhV+boXvYBNwNPmDDsCOHW+aYBvAC9t3TsDB7XuaaCA1SPzvQK4CHhQm/Z44KNt3P7AjcATgG0ZmhZuHlnP21v/Cxh2EHYAfhs4CFjd1nc+cOTI+gr4B+DXgEcANwFfbuvfFTgPeNlmPofN1jqy7IdsZt5dgB8CbwS2b/2PHdmOXwDPYgj7/wl8c2Te3wF+o23ji4CfAnuO/DvcAry2bfMO86z7fwDfAfYDAjwK2B24H/Bj4KVt3sNb/+7z/bu3Oj8259/yg+1zf1T7LH9z7rS++ny5R33v9Nm2l3pdkuuA/7OFaW8GHpJkj6q6saq+uYVpfxd4T1VdUlU3AkcBh7Wf8IcC/1hVp1bVL4E/YgiHUd+oqs9W1W1V9fOq2lhV36yqW6pqE/AB4D/NmefPq+r6qjoXOAf4Ulv/T4DPA5s7ELilWhfyHODKqnp3Vf2iqm6oqtNGxp9aVf9UVbcCH2UIPgCq6tNV9YO2jccCFwIHjsz7g6r6q7bNP59n3a8C3lZVF9TgrKq6Fng2cGFVfbTNewzwXeC5Y2zPrHe0z/0s4KzRutU3g/re6QVVtdvsC3j1FqZ9JfAw4Lvt5/RztjDtbwDfG+n/HsPe3f3buMtmR1TVz4Br58x/2WhPkoclOTHJla055M+APebMc9VI98/n6d/5LtS6kL2BeZtEmitHun8GbD/7B6A1/5w58kfykdxxm+7wGSxi3XO3h9b/gAWWt6W6N/fZqTMG9Vauqi6sqsOBfwe8EzguyU7ceW8Y4AcMB+Fm7cPwU/4qhqaCvWZHJNmB4Sf7HVY3p/99DHuFD62qX2NoK81d35qxa13IZQxNJouS5IEMzQuvYWiS2I3hV8DoNi10u8rLgAfPM3zu9sCwTVe07p8CO46M+/Xxqh6rJq0wg3orl+QlSaaq6jbgujb4NmCmvY8G1jHAG5Lsm2Rnhj3gY6vqFuA44LntgNe2DO2eC4XuLsD1wI1JHg78/hJt1kK1LuREYM8kR7aDj7skeewY883+gZsBSPJyhj3qxfgQ8CdJHprBbyXZHfgn4GFJXpxkdZIXMRwXOLHNdyZD08427eDmoYtY51XAdBLzoFP+w+gQ4Nx2JsR7gcNaO+bPgD8Fvt5+xh8EfIShTfYU4FKGg2qvBWhtyK8FPsmwd30jcDXDQavNeRPwYuAGhj3RY5dwuzZb60Kq6gbg6Qztv1cytDM/eYz5zgPezXCA9irg3wNfX2Td7wE+BXyJ4Y/YhxkOOl7L0Hb+RoYmpTcDz6mqa9p8f8iwJ/5j4B3AJxaxzk+392uTnL7IerUMUuWvHi29thd7HUOzxqUrXI50j+YetZZMkucm2bG1cb+L4TSzTStblXTPZ1BrKT2f4aDXD4CHMjSj+JNNupts+pCkzrlHLUmdm8hNYfbYY4+anp6exKIl6V5p48aN11TV1HzjJhLU09PTbNiwYRKLlqR7pSRzrzy9nU0fktQ5g1qSOmdQS1LnDGpJ6pxBLUmdM6glqXNjBXWSN7Rnw52T5Jgk20+6MEnSYMGgTvIA4HXAmqp6JMNz4g6bdGGSpMG4TR+rgR3a44Z2ZLjpjiRpGSwY1FV1BcMtK7/PcEP4n1TVl+ZOl2Rtkg1JNszMzCx9pYs0ve4kptedtNJlSNLdNk7Tx30Zbl+5L8MDNndK8pK501XV+qpaU1VrpqbmvVxdknQXjNP08TTg0qqaqaqbgeOBx022LEnSrHGC+vvAQe3JHQGeCpw/2bIkSbPGaaM+jeEJ06czPFrpPsD6CdclSWrGus1pVf0x8McTrkWSNA+vTJSkzhnUktQ5g1qSOmdQS1LnDGpJ6pxBLUmdM6glqXMGtSR1zqCWpM4Z1JLUOYNakjpnUEtS5wxqSeqcQS1JnTOoJalzBrUkdW6ch9vul+TMkdf1SY5chtokSYzxhJequgB4NECSVcAVwAmTLUuSNGuxTR9PBS6uqu9NohhJ0p0tNqgPA46ZRCGSpPmNHdRJtgWeB3x6M+PXJtmQZMPMzMxS1SdJW73F7FE/Ezi9qq6ab2RVra+qNVW1ZmpqammqkyQtKqgPx2YPSVp2YwV1kp2ApwPHT7YcSdJcC56eB1BVPwV2n3AtkqR5eGWiJHXOoJakzhnUktQ5g1qSOmdQS1LnDGpJ6pxBLUmdM6glqXMGtSR1zqCWpM4Z1JLUOYNakjpnUEtS5wxqSeqcQS1JnTOoJalzBrUkdW7cR3HtluS4JN9Ncn6SgyddmCRpMNajuID3Al+oqkOTbAvsOMGaJEkjFgzqJLsCTwSOAKiqXwK/nGxZkqRZ4zR97AvMAH+b5IwkH2pPJb+DJGuTbEiyYWZmZskLHTW97qSJLl+SejJOUK8GDgDeV1WPAX4KrJs7UVWtr6o1VbVmampqicuUpK3XOEF9OXB5VZ3W+o9jCG5J0jJYMKir6krgsiT7tUFPBc6baFWSpNuNe9bHa4GPtzM+LgFePrmSJEmjxgrqqjoTWDPZUiRJ8/HKREnqnEEtSZ0zqCWpcwa1JHXOoJakzhnUktQ5g1qSOmdQS1LnDGpJ6pxBLUmdM6glqXMGtSR1zqCWpM4Z1JLUOYNakjpnUEtS5wxqSercWE94SbIJuAG4FbilqnzaiyQtk3GfmQjw5Kq6ZmKVSJLmZdOHJHVu3KAu4EtJNiZZO98ESdYm2ZBkw8zMzJIUN73upEVPv9h5JKl34wb1E6rqAOCZwB8keeLcCapqfVWtqao1U1NTS1qkJG3NxgrqqrqivV8NnAAcOMmiJEm/smBQJ9kpyS6z3cAzgHMmXZgkaTDOWR/3B05IMjv9J6rqCxOtSpJ0uwWDuqouAR61DLVIkubh6XmS1DmDWpI6Z1BLUucMaknqnEEtSZ0zqCWpcwa1JHXOoJakzhnUktQ5g1qSOmdQS1LnDGpJ6pxBLUmdM6glqXMGtSR1zqCWpM4Z1JLUubGDOsmqJGckOXGSBUmS7mgxe9SvB86fVCGSpPmNFdRJ9gKeDXxosuVIkuYad4/6L4E3A7dtboIka5NsSLJhZmZmKWob2/S6k5hed9JElrsU00jS3bFgUCd5DnB1VW3c0nRVtb6q1lTVmqmpqSUrUJK2duPsUT8eeF6STcAngack+dhEq5Ik3W7BoK6qo6pqr6qaBg4DvlJVL5l4ZZIkwPOoJal7qxczcVV9DfjaRCqRJM3LPWpJ6pxBLUmdM6glqXMGtSR1zqCWpM4Z1JLUOYNakjpnUEtS5wxqSeqcQS1JnTOoJalzBrUkdc6glqTOGdSS1DmDWpI6Z1BLUucMaknq3DhPId8+ybeSnJXk3CTvWI7CJEmDcR7FdRPwlKq6Mck2wKlJPl9V35xwbZIkxgjqqirgxta7TXvVJIuSJP3KWA+3TbIK2Ag8BPibqjptnmnWAmsB9tlnn6WscYum1520qGk2Hf3sOw3fdPSz5+3e3HK2NI0kLbWxDiZW1a1V9WhgL+DAJI+cZ5r1VbWmqtZMTU0tcZmStPVa1FkfVXUd8FXgkIlUI0m6k3HO+phKslvr3gF4OvDdCdclSWrGaaPeE/j71k59H+BTVXXiZMuSJM0a56yPs4HHLEMtkqR5eGWiJHXOoJakzhnUktQ5g1qSOmdQS1LnDGpJ6pxBLUmdM6glqXMGtSR1zqCWpM4Z1JLUOYNakjpnUEtS5wxqSeqcQS1JnTOoJalzBrUkdW6cZybuneSrSc5Lcm6S1y9HYZKkwTjPTLwFeGNVnZ5kF2BjkpOr6rwJ1yZJYow96qr6YVWd3rpvAM4HHjDpwiRJg3H2qG+XZJrhQbenzTNuLbAWYJ999rnLBU2vO2mL/QsNX2jcUthSTZuOfvbdXvbdXYake5exDyYm2Rn4DHBkVV0/d3xVra+qNVW1ZmpqailrlKSt2lhBnWQbhpD+eFUdP9mSJEmjxjnrI8CHgfOr6j2TL0mSNGqcPerHAy8FnpLkzPZ61oTrkiQ1Cx5MrKpTgSxDLZKkeXhloiR1zqCWpM4Z1JLUOYNakjpnUEtS5wxqSeqcQS1JnTOoJalzBrUkdc6glqTOGdSS1DmDWpI6Z1BLUucMaknqnEEtSZ0zqCWpcwa1JHVunGcmfiTJ1UnOWY6CJEl3NM4e9d8Bh0y4DknSZiwY1FV1CvCjZahFkjSPJWujTrI2yYYkG2ZmZpZqsXfb9LqT7tQ/+9rcNOMs566sf0vrnFvTfLXON8186xt32UtlnLrmm2cpppm0u7Jt2jpN8nuyZEFdVeurak1VrZmamlqqxUrSVs+zPiSpcwa1JHVunNPzjgG+AeyX5PIkr5x8WZKkWasXmqCqDl+OQiRJ87PpQ5I6Z1BLUucMaknqnEEtSZ0zqCWpcwa1JHXOoJakzhnUktQ5g1qSOmdQS1LnDGpJ6pxBLUmdM6glqXMGtSR1zqCWpM4Z1JLUOYNakjo3VlAnOSTJBUkuSrJu0kVJkn5lnGcmrgL+BngmsD9weJL9J12YJGkwzh71gcBFVXVJVf0S+CTw/MmWJUmalara8gTJocAhVfWq1v9S4LFV9Zo5060F1rbe/YALFlHHHsA1i5h+JVnrZFjr5NyT6t2aa31gVU3NN2LBp5CPq6rWA+vvyrxJNlTVmqWqZZKsdTKsdXLuSfVa6/zGafq4Ath7pH+vNkyStAzGCepvAw9Nsm+SbYHDgM9NtixJ0qwFmz6q6pYkrwG+CKwCPlJV5y5xHXepyWSFWOtkWOvk3JPqtdZ5LHgwUZK0srwyUZI6Z1BLUudWPKh7uDw9yUeSXJ3knJFh90tycpIL2/t92/Ak+d+t3rOTHDAyz8va9BcmedkE6tw7yVeTnJfk3CSv77XWto7tk3wryVmt3ne04fsmOa3VdWw7SE2S7Vr/RW389MiyjmrDL0jynydU76okZyQ5sec623o2JflOkjOTbGjDev0e7JbkuCTfTXJ+koN7rDXJfu3znH1dn+TILmqtqhV7MRycvBh4ELAtcBaw/wrU8UTgAOCckWF/Dqxr3euAd7buZwGfBwIcBJzWht8PuKS937d133eJ69wTOKB17wL8P4bL+rurta0nwM6texvgtFbHp4DD2vD3A7/ful8NvL91HwYc27r3b9+N7YB923dm1QTq/e/AJ4ATW3+XdbZ1bQL2mDOs1+/B3wOvat3bArv1WutIzauAK4EH9lDrRDZyER/GwcAXR/qPAo5aoVqmuWNQXwDs2br3BC5o3R8ADp87HXA48IGR4XeYbkI1/wPw9HtIrTsCpwOPZbiaa/Xc7wDDmUUHt+7VbbrM/V6MTreE9e0FfBl4CnBiW293dY4sexN3DuruvgfArsCltBMXeq51Tn3PAL7eS60r3fTxAOCykf7L27Ae3L+qfti6rwTu37o3V/Oybkv7uf0Yhr3UbmttzQlnAlcDJzPsZV5XVbfMs+7b62rjfwLsvkz1/iXwZuC21r97p3XOKuBLSTZmuH0D9Pk92BeYAf62NSt9KMlOndY66jDgmNa94rWudFDfI9TwZ7Gb8xiT7Ax8Bjiyqq4fHddbrVV1a1U9mmGP9UDg4Stb0Z0leQ5wdVVtXOlaFuEJVXUAw10t/yDJE0dHdvQ9WM3QrPi+qnoM8FOG5oPbdVQrAO1YxPOAT88dt1K1rnRQ93x5+lVJ9gRo71e34ZureVm2Jck2DCH98ao6vudaR1XVdcBXGZoQdksye7HV6Lpvr6uN3xW4dhnqfTzwvCSbGO4O+RTgvR3WebuquqK9Xw2cwPBHsMfvweXA5VV1Wus/jiG4e6x11jOB06vqqta/4rWudFD3fHn654DZo7UvY2gPnh3+e+2I70HAT9rPoi8Cz0hy33ZU+Blt2JJJEuDDwPlV9Z6ea231TiXZrXXvwNCefj5DYB+6mXpnt+NQ4CttD+ZzwGHtbIt9gYcC31qqOqvqqKraq6qmGb6DX6mq3+2tzllJdkqyy2w3w7/fOXT4PaiqK4HLkuzXBj0VOK/HWkcczq+aPWZrWtlaJ9UYv4hG+2cxnL1wMfDWFarhGOCHwM0MewCvZGhz/DJwIfDPwP3atGF4kMLFwHeANSPLeQVwUXu9fAJ1PoHhZ9fZwJnt9awea23r+C3gjFbvOcAfteEPYgiwixh+Xm7Xhm/f+i9q4x80sqy3tu24AHjmBL8LT+JXZ310WWer66z2Onf2/03H34NHAxva9+CzDGdC9FrrTgy/jnYdGbbitXoJuSR1bqWbPiRJCzCoJalzBrUkdc6glqTOGdSS1DmDWhORZPeRu5BdmeSKkf5tx1zGWxa5zrcnedNdq3j5JXlSksetdB3qn0Gtiaiqa6vq0TVcPv5+4C9m+6vql2MuZlFBfXclWbWc62M4Z9ug1oIMai2bJL+d5F/ajYS+mGTPJLtmuHfzfm2aY5L81yRHAzu0PfCPz7OsQ5KcnuFe118eGbV/kq8luSTJ60am/2xb77kjNzEiyY1J3p3kLIbL20fX8ZAk/9zWcXqSB7er0P5XknMy3A/6RW3aJ6Xdx7r1/3WSI1r3piTvaMv4TpKHZ7ip1n8D3tC28T8uwUese6kFH24rLZEAfwU8v6pmWsD9aVW9IsPDk/8uyXsZ7tv7QYAkr2l75HdcUDIFfBB4YlVdmuR+I6MfDjyZ4X7dFyR5X1XdDLyiqn7ULmX/dpLPVNW1DFeinVZVb5yn5o8DR1fVCUm2Z9ix+S8MV9o9CtijLeuUMbb/mqo6IMmrgTdV1auSvB+4sareNcb82ooZ1Fou2wGPBE4eblnCKobL9qmqk5P8DsPluI8aY1kHAadU1aVt/h+NjDupqm4CbkpyNcMtKS8HXpfkhW2avRnuw3EtcCvDTa7uoN1L4wFVdUJbxy/a8CcAx1TVrQw36/kX4D8A189dxhyzN9DayBD20tgMai2XAOdW1cF3GpHcB/hN4GcM94G4/G6s56aR7luB1UmeBDyN4Sb+P0vyNYb7dQD8ooXu3XULd2xK3H7O+Nm6bsX/d1ok26i1XG4CppIcDMPtWpM8oo17A8Nd9V7McIP5bdrwm0e6R30TeGK7Qx1zmj7msyvw4xbSD2fYI9+iqroBuDzJC9o6tkuyI/CvwIsyPBBhiuExbt8CvsfQPr5dhjsGPnWhdQA3MDTRSFtkUGu53MZwS9B3tgN3ZwKPawcRXwW8sar+FTgFeFubZz1w9tyDiVU1A6wFjm/LOnaBdX+BYc/6fOBohqAfx0sZmkzOBv4N+HWGez+fzXDnuq8Ab66qK6vqMoZnLJ7T3s8YY/n/CLzQg4laiHfPk6TOuUctSZ0zqCWpcwa1JHXOoJakzhnUktQ5g1qSOmdQS1Ln/j/JK91cljAqVAAAAABJRU5ErkJggg==\n"
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%pylab inline\n",
    "import matplotlib.pyplot as plt\n",
    "_ = plt.hist(train_df['text_len'], bins=200)\n",
    "plt.xlabel('Text char count')\n",
    "plt.title(\"Histogram of char count\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **新闻类别分布**\n",
    "\n",
    "接下来可以对数据集的类别进行分布统计，具体统计每类新闻的样本个数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    },
    {
     "data": {
      "text/plain": "Text(0.5, 0, 'category')"
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": "<Figure size 432x288 with 1 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEZCAYAAACZwO5kAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAZR0lEQVR4nO3de7gddX3v8feH65H7bRtuCeEIIqiF0n2CAhaoQAOkohYVsAoKRlGqnNrTw0ErSE9btKdwTqHKEwFRlGsVm8o1igSwCrmQQBCQi8EkQBKu4dZK4HP+mNk+y81ayd5r1t5h//J5Pc961sxvZr7zy+ydz571W7NmyTYREVGuddZ0ByIiYmQl6CMiCpegj4goXII+IqJwCfqIiMIl6CMiCpegj2ghyZJ2WdP9iOilBH2MOEkLJS2TtHFL24mSbl6D3VorSTpD0nfWdD9idCXoY7SsC3xuTXciYm2UoI/R8g/AX0raot1CSW+RNEPSU5Lul/TBun1nSc9IWqee/4akZS3bXSLplHr6eEkPS3pO0q8kfbjDvtaVdJqkh+p150ga32a9IyTdKWmFpEWSzmhZ9l8kfUfSk3X/Zkka16t+SNq3rvls/bxvy3YLJR3cMv/bs3RJE+vhp+Mk/VrSE5K+UC+bDJwGfEjS85Lmt+tXlCdBH6NlNnAz8JeDF9RDOjOAS4E3AkcDX5O0h+1fASuA369X/0PgeUm71/MHADPrGv8EHGZ7U2BfYF6HvvwFcAxwOLAZ8HHgxTbrvQB8FNgCOAI4SdJ762XHAZsD44GtgU8BL/WiH5K2Aq6p62wNnA1cI2nrDnXa2R/YDXg38CVJu9u+Hvg74Arbm9jecxj1YgxL0Mdo+hLw55L6BrVPARba/qbtlbbvBL4HfKBePhM4QNK29fy/1PM7UwXkwJnpq8DbJL3B9mO27+nQjxOBL9q+35X5tp8cvJLtm23fbftV23cBl1H9YQF4mSqEd7H9iu05tlf0qB9HAA/YvqQ+HpcB9wF/0qFOO1+2/ZLt+fXxSaivxRL0MWpsLwB+CJw6aNFOwD71EMgzkp4BPgwMBPtM4ECqs/lbqF4ZHFA/bq2D+AXgQ1Rn1o9JukbSWzp0ZTzw0Or6K2kfST+RtFzSs3XtberFlwA3AJdLelTSVyWt36N+bA88MqjtEWCH1fW5xeMt0y8Cmwxj2yhMgj5G2+nAJ/jd0FoEzLS9RctjE9sn1ctnAu+iCvuZwG3AftTDNgNFbN9g+xBgO6oz4G906MMi4E1D6OulwHRgvO3NgfMB1ft62faXbe9BNTwzhWqYpxf9eJTqj1+rCcCSevoFYKOWZdsydLld7VooQR+jyvaDwBXAZ1uafwi8WdJHJK1fP/7bwDi87QeAl4A/o/qDsAJYCvwpddBLGifpyHqM/D+B56mGUNq5APgbSbuq8nsdxr83BZ6y/R+SJgHHDiyQdJCkt0tal+o9hJeBV3vUj2vr43GspPUkfQjYoz5OUI35H10fp37gqA7121kKTBx4czvWDvlhx5pwJvDba+ptPwccSvUm7KNUww5fATZs2WYm8KTtRS3zAubW8+tQvbn5KPAU1dn+SbR3NnAlcCNVSF8IvKHNep8GzpT0HNX7C1e2LNuW6r2CFcC9dX8u6UU/6nH6KcDngSeBvwKm2H6i3u6vqV4JPA18meqVx1BdVT8/KWnuKteMYihfPBIRUbac0UdEFC5BHxFRuAR9REThEvQREYVL0EdEFG69Nd2BdrbZZhtPnDhxTXcjImLMmDNnzhO2B99eBHidBv3EiROZPXv2mu5GRMSYIWnwbTN+K0M3ERGFS9BHRBQuQR8RUbgEfURE4RL0ERGFS9BHRBQuQR8RUbgEfURE4V6XH5jqZOKp1wxr/YVnHTFCPYmIGDtyRh8RUbgEfURE4RL0ERGFS9BHRBQuQR8RUbgEfURE4RL0ERGFS9BHRBQuQR8RUbgEfURE4cbULRBGUm6vEBGlyhl9REThVntGL+kiYAqwzPbb6rYrgN3qVbYAnrG9V5ttFwLPAa8AK23396TXERExZEMZurkYOA/49kCD7Q8NTEv6R+DZVWx/kO0nuu1gREQ0s9qgt32LpIntlkkS8EHgj3rcr4iI6JGmY/TvApbafqDDcgM3SpojaeqqCkmaKmm2pNnLly9v2K2IiBjQNOiPAS5bxfL9be8NHAZ8RtIfdlrR9jTb/bb7+/r6GnYrIiIGdB30ktYD3g9c0Wkd20vq52XA1cCkbvcXERHdaXJGfzBwn+3F7RZK2ljSpgPTwKHAggb7i4iILqw26CVdBvwM2E3SYkkn1IuOZtCwjaTtJV1bz44DbpM0H7gDuMb29b3rekREDMVQrro5pkP78W3aHgUOr6cfBvZs2L+IiGgon4yNiChcgj4ionAJ+oiIwiXoIyIKl6CPiChcgj4ionAJ+oiIwiXoIyIKl6CPiChcgj4ionAJ+oiIwiXoIyIKl6CPiChcgj4ionAJ+oiIwiXoIyIKl6CPiChcgj4ionBD+c7YiyQtk7Sgpe0MSUskzasfh3fYdrKk+yU9KOnUXnY8IiKGZihn9BcDk9u0n2N7r/px7eCFktYF/hk4DNgDOEbSHk06GxERw7faoLd9C/BUF7UnAQ/aftj2b4DLgSO7qBMREQ00GaM/WdJd9dDOlm2W7wAsaplfXLdFRMQo6jbovw68CdgLeAz4x6YdkTRV0mxJs5cvX960XERE1LoKettLbb9i+1XgG1TDNIMtAca3zO9Yt3WqOc12v+3+vr6+broVERFtdBX0krZrmX0fsKDNarOAXSXtLGkD4Ghgejf7i4iI7q23uhUkXQYcCGwjaTFwOnCgpL0AAwuBT9brbg9cYPtw2yslnQzcAKwLXGT7npH4R0RERGerDXrbx7RpvrDDuo8Ch7fMXwu85tLLiIgYPflkbERE4RL0ERGFS9BHRBQuQR8RUbgEfURE4RL0ERGFS9BHRBQuQR8RUbgEfURE4RL0ERGFS9BHRBQuQR8RUbgEfURE4RL0ERGFS9BHRBQuQR8RUbjVfvFI9MbEU68Z1voLzzpihHoSEWubnNFHRBRutUEv6SJJyyQtaGn7B0n3SbpL0tWStuiw7UJJd0uaJ2l2D/sdERFDNJQz+ouByYPaZgBvs/17wC+B/7WK7Q+yvZft/u66GBERTaw26G3fAjw1qO1G2yvr2Z8DO45A3yIiogd6MUb/ceC6DssM3ChpjqSpPdhXREQMU6OrbiR9AVgJfLfDKvvbXiLpjcAMSffVrxDa1ZoKTAWYMGFCk25FRESLrs/oJR0PTAE+bNvt1rG9pH5eBlwNTOpUz/Y02/22+/v6+rrtVkREDNJV0EuaDPwV8B7bL3ZYZ2NJmw5MA4cCC9qtGxERI2col1deBvwM2E3SYkknAOcBm1INx8yTdH697vaSrq03HQfcJmk+cAdwje3rR+RfERERHa12jN72MW2aL+yw7qPA4fX0w8CejXoXERGN5RYIhcgtFiKik9wCISKicAn6iIjCJegjIgqXoI+IKFyCPiKicAn6iIjCJegjIgqXoI+IKFyCPiKicAn6iIjCJegjIgqXoI+IKFyCPiKicAn6iIjCJegjIgqXoI+IKFyCPiKicAn6iIjCDSnoJV0kaZmkBS1tW0maIemB+nnLDtseV6/zgKTjetXxiIgYmqGe0V8MTB7UdirwY9u7Aj+u53+HpK2A04F9gEnA6Z3+IERExMgYUtDbvgV4alDzkcC36ulvAe9ts+kfAzNsP2X7aWAGr/2DERERI6jJGP0424/V048D49qsswOwqGV+cd32GpKmSpotafby5csbdCsiIlr15M1Y2wbcsMY02/22+/v6+nrRrYiIoFnQL5W0HUD9vKzNOkuA8S3zO9ZtERExSpoE/XRg4Cqa44B/bbPODcChkras34Q9tG6LiIhRMtTLKy8DfgbsJmmxpBOAs4BDJD0AHFzPI6lf0gUAtp8C/gaYVT/OrNsiImKUrDeUlWwf02HRu9usOxs4sWX+IuCirnoXERGN5ZOxERGFS9BHRBQuQR8RUbgEfURE4RL0ERGFS9BHRBQuQR8RUbgEfURE4RL0ERGFS9BHRBQuQR8RUbgEfURE4RL0ERGFS9BHRBQuQR8RUbgEfURE4Yb0xSMRE0+9ZljrLzzriBHqSUQMV87oIyIK13XQS9pN0ryWxwpJpwxa50BJz7as86XGPY6IiGHpeujG9v3AXgCS1gWWAFe3WfVW21O63U9ERDTTq6GbdwMP2X6kR/UiIqJHehX0RwOXdVj2TknzJV0n6a2dCkiaKmm2pNnLly/vUbciIqJx0EvaAHgPcFWbxXOBnWzvCZwL/KBTHdvTbPfb7u/r62varYiIqPXijP4wYK7tpYMX2F5h+/l6+lpgfUnb9GCfERExRL0I+mPoMGwjaVtJqqcn1ft7sgf7jIiIIWr0gSlJGwOHAJ9safsUgO3zgaOAkyStBF4CjrbtJvuMiIjhaRT0tl8Ath7Udn7L9HnAeU32ERERzeSTsRERhUvQR0QULkEfEVG4BH1EROES9BERhUvQR0QULkEfEVG4BH1EROES9BERhUvQR0QULkEfEVG4BH1EROES9BERhUvQR0QULkEfEVG4BH1EROES9BERhUvQR0QUrnHQS1oo6W5J8yTNbrNckv5J0oOS7pK0d9N9RkTE0DX6ztgWB9l+osOyw4Bd68c+wNfr54iIGAWjMXRzJPBtV34ObCFpu1HYb0RE0JugN3CjpDmSprZZvgOwqGV+cd0WERGjoBdDN/vbXiLpjcAMSffZvmW4Reo/ElMBJkyY0INuxVgy8dRrhrX+wrOOWKvqRzTR+Ize9pL6eRlwNTBp0CpLgPEt8zvWbYPrTLPdb7u/r6+vabciIqLWKOglbSxp04Fp4FBgwaDVpgMfra++eQfwrO3Hmuw3IiKGrunQzTjgakkDtS61fb2kTwHYPh+4FjgceBB4EfhYw31GRMQwNAp62w8De7ZpP79l2sBnmuwnIiK6l0/GRkQULkEfEVG4BH1EROES9BERhUvQR0QULkEfEVG4BH1EROES9BERhUvQR0QULkEfEVG4BH1EROES9BERhUvQR0QULkEfEVG4BH1EROES9BERhevFl4NHxAgb619uPtbrj3U5o4+IKFzXQS9pvKSfSPqFpHskfa7NOgdKelbSvPrxpWbdjYiI4WoydLMS+LztuZI2BeZImmH7F4PWu9X2lAb7iYiIBro+o7f9mO259fRzwL3ADr3qWERE9EZPxuglTQR+H7i9zeJ3Spov6TpJb+3F/iIiYugaX3UjaRPge8AptlcMWjwX2Mn285IOB34A7NqhzlRgKsCECROadisiImqNzuglrU8V8t+1/f3By22vsP18PX0tsL6kbdrVsj3Ndr/t/r6+vibdioiIFk2uuhFwIXCv7bM7rLNtvR6SJtX7e7LbfUZExPA1GbrZD/gIcLekeXXbacAEANvnA0cBJ0laCbwEHG3bDfYZERHD1HXQ274N0GrWOQ84r9t9REREc7kFQkTEagznFguvx9sr5BYIERGFS9BHRBQuQR8RUbgEfURE4RL0ERGFS9BHRBQuQR8RUbgEfURE4RL0ERGFS9BHRBQut0CIiFiDhnN7BejuFgs5o4+IKFyCPiKicAn6iIjCJegjIgqXoI+IKFyCPiKicAn6iIjCNQp6SZMl3S/pQUmntlm+oaQr6uW3S5rYZH8RETF8XQe9pHWBfwYOA/YAjpG0x6DVTgCetr0LcA7wlW73FxER3WlyRj8JeND2w7Z/A1wOHDlonSOBb9XT/wK8W5Ia7DMiIoZJtrvbUDoKmGz7xHr+I8A+tk9uWWdBvc7iev6hep0n2tSbCkytZ3cD7h9Gd7YBXlOzR0ayduqnfuqnfq9q72S7r92C1829bmxPA6Z1s62k2bb7e9ylEa+d+qmf+qk/GrWbDN0sAca3zO9Yt7VdR9J6wObAkw32GRERw9Qk6GcBu0raWdIGwNHA9EHrTAeOq6ePAm5yt2NFERHRla6HbmyvlHQycAOwLnCR7XsknQnMtj0duBC4RNKDwFNUfwxGQldDPq+D2qmf+qmf+iNeu+s3YyMiYmzIJ2MjIgqXoI+IKFyCPiKicK+b6+iHStJbgB2A220/39I+2fb1a65nQ1P3/0iqfwNUl6BOt31vj+pPAmx7Vn1LisnAfbav7UX9Nvv7tu2PjkTtXpP0WeBq24tGqP4+wL22V0h6A3AqsDfwC+DvbD/bsP5/Bd5PdcnyK8AvgUttr2jW89EnaX+qT9cvsH1jD+oNXPn3qO0fSToW2Be4F5hm++Wm+xjLxtSbsfV/1M9Q/fD2Aj5n+1/rZXNt7z3C+/+Y7W822P5/AsdQ3S5icd28I9Uv6OW2z2rYv9Op7j20HjAD2Af4CXAIcIPtv21Yf/DlswIOAm4CsP2eJvVHmqRngReAh4DLgKtsL+9h/XuAPesr0qYBL1Lf+qNuf3+D2p8FpgC3AIcDdwLPAO8DPm375ma9H1mS7rA9qZ7+BNX/46uBQ4F/68Hv/nepfu83ojoumwDfpzr2sn1c563XArbHzAO4G9iknp4IzKYKe4A7R2H/v264/S+B9du0bwA80KPjsy7VL/sKYLO6/Q3AXT2oPxf4DnAgcED9/Fg9fcAIH/vrelDjTqrhykOpLv1dDlxP9VmPTXtQ/97WYzVo2bxe/Gzr6Y2Am+vpCb363af6QONZwH1Ul0M/SXVSdRawRdNj3zI9C+irpzcG7u5B3++qn9cDlrYcK/Xid7+utRnw98AlwLGDln2tB/W3Bb5OdbPIrYEz6p/7lcB2TWqPtTH6dVwP19heSBU0h0k6m+oH2pikuzo87gbGNSz/KrB9m/bt6mVNrbT9iu0XgYdcv6S3/VKP6vcDc4AvAM+6Oot8yfZM2zObFpe0d4fHH1C9gmvKtl+1faPtE6h+Fl+jGt56uAf1F0j6WD09X1I/gKQ3A70YOhgYat2Q6owV278G1u9BbagC5WngQNtb2d6a6hXb0/WyJtaRtKWkranOsJcD2H4BWNmw9kD9DYBNqf4Qbl63b0jvjs83qXLme8DRkr4nacN62Tt6UP9iqmG+RVSvxF+ievV2K3B+k8JjbYx+qaS9bM8DsP28pCnARcDbe7SPccAfU/1ytxLw7w1rnwL8WNIDVD9MqM7IdgFO7rTRMPxG0kZ10P/BQKOkzelB0Nt+FThH0lX181J6+zs0C5hJ+z/aW/Sg/u/UdTVuOx2YLmmjHtQ/Efh/kr5IdTOqn0laRPWzPrFh7QuAWZJuB95FfctvSX1UZ9+9MNH279xK3PbjwFckfbxh7c2pThIEWNJ2th+TtAm9OUm7kOqVyLpUJyJXSXqYKoAv70F9gDfZ/tN6+geSvgDcJKlXQ5bjbJ8LIOnTLT+LcyWd0KTwWBuj35HqrPXxNsv2s/3THuzjQuCbtm9rs+xS28c2rL8O1ZtQrW/GzrL9SpO6de0Nbf9nm/ZtqF763d10H4PqHgHsZ/u0HtVbALzP9gNtli2yPb7NZsOp/2bbv2xSY4j72QzYmeqP4GLbS3tU963A7lRvYN7Xi5qD6t8I/Aj41kCfJY0DjgcOsX3wCOxzI6qA+1UPam0PYPtRSVsAB1MNt97RtHZd/17grfUJz0Db8cD/oBpS3qlh/fm296yn/7ftL7Ysu9t21yezYyroo2z1ra/vtv2aW1RLeq/tH4x+r9YekrakulLoSOCNdfNSqlc9Z9ke/Cp3rSLpq8CNtn80qH0ycK7tXRvWPxP4qluuJqzbd6E6/kd1XTtBH2NB0yueopkc/1Ub6ePT+Iq/BH2MBZJ+bXvCmu7H2irHf9VG+vg0rT/W3oyNgkm6q9Miml/xFKuR479qI318RrJ+gj5eT0byiqdYvRz/VRvp4zNi9RP08XryQ6qrF+YNXiDp5lHvzdonx3/VRvr4jFj9jNFHRBRurH0yNiIihilBHxFRuAR9rPUkHShp3zXdj4iRkqCPqG6ON6JBr0r+v8UakV+8KJakj9Z3Hp0v6RJJfyLpdkl3SvqRpHGSJgKfAv67pHmS3iWpr74z4az6sV9dr0/SDEn3SLpA0iP1fYSQ9BeSFtSPU+q2iZLul/RtYAHw15L+b0v/PiHpnFE+LLEWylU3UaT6BmBXA/vafkLSVoCBZ2xb0onA7rY/L+kM4Hnb/6fe9lKq+4vfJmkC1Ze27C7pPGCJ7b+v729yHdAH7ER1i9l3UF3zfDvwZ1TXQz9c9+Hn9Z0a5wNvsf2ypH8HPtnrm81FDJbr6KNUf0T1DVJPANh+StLbgSskbUf1ZS+d7ph4MLCH9Nu7525Wh/T+VN/ohO3rJQ18sGV/qq8ofAFA0vepbiU8HXjE9s/rbZ6XdBMwpb4T4voJ+RgNCfpYm5wLnG17uqQDqb7Bp511gHfY/o/WxpbgH44XBs1fAJxGde/03CQsRkXG6KNUNwEfUPWNRtRDN5tT3f8fqq8PHPAc1TcTDbgR+POBGUl71ZM/BT5Ytx0KbFm33wq8V9JGkjamOuu/tV2nbN9O9eXex1J9b23EiEvQR5Fs3wP8LTBT0nzgbKoz+KskzaH6BqgB/wa8b+DNWOCzQH/9Ru4vqN6sBfgycGj9BSkfAB4HnrM9l2qM/g6q8fkLbN+5iu5dCfx0bb+/e4yevBkbMUSqvh/0FdsrJb0T+Lrtvbqo80PgHNs/7nUfI9rJGH3E0E0Arqyvh/8N8InhbFx/vd0dwPyEfIymnNFHRBQuY/QREYVL0EdEFC5BHxFRuAR9REThEvQREYVL0EdEFO7/A4GqyViWxxqOAAAAAElFTkSuQmCC\n"
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%pylab inline\n",
    "train_df['label'].value_counts().plot(kind='bar')\n",
    "plt.title('News class count')\n",
    "plt.xlabel(\"category\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在数据集中标签的对应的关系如下：{'科技': 0, '股票': 1, '体育': 2, '娱乐': 3, '时政': 4, '社会': 5, '教育': 6, '财经': 7, '家居': 8, '游戏': 9, '房产': 10, '时尚': 11, '彩票': 12, '星座': 13}\n",
    "\n",
    "从统计结果可以看出，赛题的数据集类别分布存在较为不均匀的情况。在训练集中科技类新闻最多，其次是股票类新闻，最少的新闻是星座新闻。\n",
    "\n",
    "#### **字符分布统计**\n",
    "\n",
    "接下来可以统计每个字符出现的次数，首先可以将训练集中所有的句子进行拼接进而划分为字符，并统计每个字符的个数。\n",
    "\n",
    "从统计结果中可以看出，在训练集中总共包括6869个字，其中编号3750的字出现的次数最多，编号3133的字出现的次数最少。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2405\n",
      "('3750', 3702)\n",
      "('5034', 1)\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "all_lines = ' '.join(list(train_df['text']))\n",
    "word_count = Counter(all_lines.split(\" \"))\n",
    "word_count = sorted(word_count.items(), key=lambda d:d[1], reverse = True)\n",
    "\n",
    "print(len(word_count))\n",
    "\n",
    "print(word_count[0])\n",
    "\n",
    "print(word_count[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里还可以根据字在每个句子的出现情况，反推出标点符号。下面代码统计了不同字符在句子中出现的次数，其中字符3750，字符900和字符648在20w新闻的覆盖率接近99%，很有可能是标点符号。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('3750', 99)\n",
      "('900', 99)\n",
      "('648', 96)\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "train_df['text_unique'] = train_df['text'].apply(lambda x: ' '.join(list(set(x.split(' ')))))\n",
    "all_lines = ' '.join(list(train_df['text_unique']))\n",
    "word_count = Counter(all_lines.split(\" \"))\n",
    "word_count = sorted(word_count.items(), key=lambda d:int(d[1]), reverse = True)\n",
    "\n",
    "print(word_count[0])\n",
    "\n",
    "print(word_count[1])\n",
    "\n",
    "print(word_count[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### **数据分析的结论**\n",
    "\n",
    "通过上述分析我们可以得出以下结论：\n",
    "\n",
    "1. 赛题中每个新闻包含的字符个数平均为1000个，还有一些新闻字符较长；\n",
    "2. 赛题中新闻类别分布不均匀，科技类新闻样本量接近4w，星座类新闻样本量不到1k；\n",
    "3. 赛题总共包括7000-8000个字符；\n",
    "\n",
    "通过数据分析，我们还可以得出以下结论：\n",
    "\n",
    "1. 每个新闻平均字符个数较多，可能需要截断；\n",
    "\n",
    "2. 由于类别不均衡，会严重影响模型的精度；\n",
    "\n",
    "### **本章小结**\n",
    "\n",
    "本章对赛题数据进行读取，并新闻句子长度、类别和字符进行了可视化分析。\n",
    "\n",
    "### **本章作业**\n",
    "\n",
    "1. 假设字符3750，字符900和字符648是句子的标点符号，请分析赛题每篇新闻平均由多少个句子构成？\n",
    "2. 统计每类新闻中出现次数对多的字符\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}