Multi-Perspective Relevance Matching with Hierarchical ConvNets for Social Media Search

This repo contains code and data for our neural tweet search paper published in AAAI'19.

Given a query, we aim to return the most relevant documents(tweets) by ranking their relevency. In social media search, the scenario is different as standard ad-hoc retrieval: shorter document length, less formal languages and multiple relevance source signals (e.g., URL, hashtag). We propose a hierarchical convolutional model to approach the hetergeneous relevance signals (tweet, URL, hashtag) at multiple perspectives, including character-, word-, phrase- and sentence-level modeling. Our model demonstrated significant gains on multiple twitter datasets against state-of-the-art neural ranking models. More details can be found in our paper.

Requirements

• Python 2.7
• Tensorflow or Theano (tested on TF 1.4.1)
• Keras (tested on 2.0.5)

Install

• Download our repo:

git clone https://github.com/Jeffyrao/neural-tweet-search.git
cd neural-tweet-search

• Install gdrive
• Download required data and word2vec:

$ chmod +x download.sh; ./download.sh

• Install Tensorflow and Keras dependency:

$ pip install -r requirements.txt

Run

• Train and test on GPU:

CUDA_VISIBLE_DEVICES=0 python -u train.py -t trec-2013

The path of best model and output predictions will be shown in the log. Default parameters should work reasonably well.

• Note: you might need around ~40GB memory to create the dataset (because of the large size of IDF weights). Please file a issue if you have any problem in creating the dataset.

• Parameter sweep to find the best parameter set:

chmod +x param_sweep.sh; ./param_sweep.sh trec-2013 &

This command will save all the outputs under tune-logs folder.

Evaluate with trec_eval

$ ./trec_eval.8.1/trec_eval data/twitter-v0/qrels.microblog2011-2014.txt \
                            best_run/mphcnn_trec_2013_pred.txt

This should return the exact MPHCNN score on TREC 2013 dataset (MAP: 0.2818, P30: 0.5222) we reported in our paper.

Command line parameters

option	input format	default	description
-t	[trec-2011, trec-2012, trec-2013, trec-2014]	trec-2011	test set
-l	[true, false]	false	whether to load pre-created dataset (set to true when data is ready)
--load_model	[true, false]	false	whether to load pre-trained model
-b	[1, n)	64	batch size
-n	[1, n)	256	number of convolutional filters
-d	[0, 1]	0.1	dropout rate
-o	[sgd, adam, rmsprop]	sgd	optimization method
--lr	[0, 1]	0.05	learning rate
--epochs	[1, n)	15	number of training epochs
--trainable	[true, false]	true	whether to train word embeddings
--val_split	(0, 1)	0.15	percentage of validation set sampled from training set
-v	[0, 1, 2]	1	verbose (for logging), 0 for silent, 1 for interactive, 2 for per-epoch logging
--conv_option	[normal, ResNet]	normal	convolutional model, normal or ResNet
--model_option	[complete, word-url]	complete	what input sources to use, complete for MP-HCNN, word-url for only modeling query-tweet (word) and query-url (char)

Reference

If you are using this code or dataset, please kindly cite the paper below:

@article{rao2019multi,
  title={Multi-Perspective Relevance Matching with Hierarchical ConvNets for Social Media Search},
  author={Rao, Jinfeng and Yang, Wei and Zhang, Yuhao and Ture, Ferhan and Lin, Jimmy},
  journal={Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI)},
  year={2019}
}