notebooks/00_quick_start/lstur_synthetic.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LSTUR: Neural News Recommendation with Long- and Short-term User Representations\n",
    "LSTUR \\[1\\] is a news recommendation approach capturing users' both long-term preferences and short-term interests. The core of LSTUR is a news encoder and a user encoder.  In the news encoder, we learn representations of news from their titles. In user encoder, we propose to learn long-term\n",
    "user representations from the embeddings of their IDs. In addition, we propose to learn short-term user representations from their recently browsed news via GRU network. Besides, we propose two methods to combine\n",
    "long-term and short-term user representations. The first one is using the long-term user representation to initialize the hidden state of the GRU network in short-term user representation. The second one is concatenating both\n",
    "long- and short-term user representations as a unified user vector.\n",
    "\n",
    "## Properties of LSTUR:\n",
    "- LSTUR captures users' both long-term and short term preference.\n",
    "- It uses embeddings of users' IDs to learn long-term user representations.\n",
    "- It uses users' recently browsed news via GRU network to learn short-term user representations.\n",
    "\n",
    "## Data format:\n",
    "\n",
    "### train data\n",
    "One simple example: <br>\n",
    "\n",
    "`1 0 0 0 0 Impression:0 User:2903 CandidateNews0:27006,11901,21668,9856,16156,21390,1741,2003,16983,8164 CandidateNews1:8377,10423,9960,5485,20494,7553,1251,17232,4745,9178 CandidateNews2:1607,26414,25830,16156,15337,16461,4004,6230,17841,10704 CandidateNews3:17323,20324,27855,16156,2934,14673,551,0,0,0 CandidateNews4:7172,3596,25442,21596,26195,4745,17988,16461,1741,76 ClickedNews0:11362,8205,22501,9349,12911,20324,1238,11362,26422,19185 ...`\n",
    "<br>\n",
    "\n",
    "In general, each line in data file represents one positive instance and n negative instances in a same impression. The format is like: <br>\n",
    "\n",
    "`[label0] ... [labeln] [Impression:i] [User:u] [CandidateNews0:w1,w2,w3,...] ... [CandidateNewsn:w1,w2,w3,...] [ClickedNews0:w1,w2,w3,...] ...`\n",
    "\n",
    "<br>\n",
    "\n",
    "It contains several parts seperated by space, i.e. label part, Impression part `<impresison id>`, User part `<user id>`, CandidateNews part, ClickedHistory part. CandidateNews part describes the target news article we are going to score in this instance, it is represented by (aligned) title words. To take a quick example, a news title may be : `Trump to deliver State of the Union address next week` , then the title words value may be `CandidateNewsi:34,45,334,23,12,987,3456,111,456,432`. <br>\n",
    "ClickedNewsk describe the k-th news article the user ever clicked and the format is the same as candidate news. Words are aligned in news title. We use a fixed length to describe an article, if the title is less than the fixed length, just pad it with zeros.\n",
    "\n",
    "### test data\n",
    "One simple example: <br>\n",
    "`1 Impression:0 User:6446 CandidateNews0:18707,23848,13490,10948,21385,11606,1251,16591,827,28081 ClickedNews0:27838,7376,16567,28518,119,21248,7598,9349,20324,9349 ClickedNews1:7969,9783,1741,2549,27104,14669,14777,21343,7667,20324 ...`\n",
    "<br>\n",
    "\n",
    "In general, each line in data file represents one instance. The format is like: <br>\n",
    "\n",
    "`[label] [Impression:i] [User:u] [CandidateNews0:w1,w2,w3,...] [ClickedNews0:w1,w2,w3,...] ...`\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Global settings and imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/data/anaconda/envs/reco_gpu/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/data/anaconda/envs/reco_gpu/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/data/anaconda/envs/reco_gpu/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/data/anaconda/envs/reco_gpu/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/data/anaconda/envs/reco_gpu/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/data/anaconda/envs/reco_gpu/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.6.10 |Anaconda, Inc.| (default, Mar 25 2020, 23:51:54) \n",
      "[GCC 7.3.0]\n",
      "Tensorflow version: 1.12.0\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.append(\"../../\")\n",
    "import os\n",
    "from reco_utils.recommender.deeprec.deeprec_utils import download_deeprec_resources \n",
    "from reco_utils.recommender.newsrec.newsrec_utils import prepare_hparams\n",
    "from reco_utils.recommender.newsrec.models.lstur import LSTURModel\n",
    "from reco_utils.recommender.newsrec.io.news_iterator import NewsIterator\n",
    "import papermill as pm\n",
    "from tempfile import TemporaryDirectory\n",
    "import tensorflow as tf\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"Tensorflow version: {}\".format(tf.__version__))\n",
    "\n",
    "tmpdir = TemporaryDirectory()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download and load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 21.2k/21.2k [00:01<00:00, 12.4kKB/s]\n"
     ]
    }
   ],
   "source": [
    "data_path = tmpdir.name\n",
    "yaml_file = os.path.join(data_path, r'lstur.yaml')\n",
    "train_file = os.path.join(data_path, r'train.txt')\n",
    "valid_file = os.path.join(data_path, r'test.txt')\n",
    "wordEmb_file = os.path.join(data_path, r'embedding.npy')\n",
    "if not os.path.exists(yaml_file):\n",
    "    download_deeprec_resources(r'https://recodatasets.blob.core.windows.net/newsrec/', data_path, 'lstur.zip')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create hyper-parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "epochs=5\n",
    "seed=42"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('attention_hidden_dim', 200), ('batch_size', 64), ('body_size', None), ('cnn_activation', 'relu'), ('data_format', 'news'), ('dense_activation', None), ('doc_size', 10), ('dropout', 0.2), ('epochs', 5), ('filter_num', 400), ('gru_unit', 400), ('head_dim', 100), ('head_num', 4), ('his_size', 50), ('iterator_type', None), ('learning_rate', 0.0001), ('loss', 'cross_entropy_loss'), ('metrics', ['group_auc', 'mean_mrr', 'ndcg@5;10']), ('npratio', 4), ('optimizer', 'adam'), ('show_step', 100000), ('subvert_emb_dim', 100), ('subvert_num', None), ('title_size', None), ('type', 'ini'), ('user_emb_dim', 50), ('user_num', 10338), ('vert_emb_dim', 100), ('vert_num', None), ('window_size', 3), ('wordEmb_file', '/tmp/tmp2e1es2so/embedding.npy'), ('word_emb_dim', 100), ('word_size', 28929)]\n"
     ]
    }
   ],
   "source": [
    "hparams = prepare_hparams(yaml_file, wordEmb_file=wordEmb_file, epochs=epochs)\n",
    "print(hparams)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "iterator = NewsIterator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the LSTUR model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LSTURModel(hparams, iterator, seed=seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'group_auc': 0.5094, 'mean_mrr': 0.1628, 'ndcg@5': 0.1497, 'ndcg@10': 0.2118}\n"
     ]
    }
   ],
   "source": [
    "print(model.run_eval(valid_file))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/data/anaconda/envs/reco_gpu/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:112: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "at epoch 1\n",
      "train info: logloss loss:1.6093932487526719\n",
      "eval info: group_auc:0.5476, mean_mrr:0.1704, ndcg@10:0.236, ndcg@5:0.1733\n",
      "at epoch 1 , train time: 12.5 eval time: 8.1\n",
      "at epoch 2\n",
      "train info: logloss loss:1.5449624518958889\n",
      "eval info: group_auc:0.5537, mean_mrr:0.1816, ndcg@10:0.2515, ndcg@5:0.1843\n",
      "at epoch 2 , train time: 10.2 eval time: 8.1\n",
      "at epoch 3\n",
      "train info: logloss loss:1.5040684374011293\n",
      "eval info: group_auc:0.561, mean_mrr:0.1816, ndcg@10:0.2515, ndcg@5:0.1821\n",
      "at epoch 3 , train time: 10.3 eval time: 8.0\n",
      "at epoch 4\n",
      "train info: logloss loss:1.4703012485893405\n",
      "eval info: group_auc:0.5649, mean_mrr:0.1823, ndcg@10:0.2511, ndcg@5:0.1866\n",
      "at epoch 4 , train time: 10.3 eval time: 7.9\n",
      "at epoch 5\n",
      "train info: logloss loss:1.4386503496948553\n",
      "eval info: group_auc:0.5667, mean_mrr:0.1827, ndcg@10:0.2465, ndcg@5:0.1898\n",
      "at epoch 5 , train time: 10.3 eval time: 8.1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<reco_utils.recommender.newsrec.models.lstur.LSTURModel at 0x7f5478722b38>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(train_file, valid_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'group_auc': 0.5667, 'mean_mrr': 0.1827, 'ndcg@5': 0.1898, 'ndcg@10': 0.2465}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/data/anaconda/envs/reco_gpu/lib/python3.6/site-packages/ipykernel_launcher.py:3: DeprecationWarning: Function record is deprecated and will be removed in verison 1.0.0 (current version 0.19.1). Please see `scrapbook.glue` (nteract-scrapbook) as a replacement for this functionality.\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "data": {
      "application/papermill.record+json": {
       "res_syn": {
        "group_auc": 0.5667,
        "mean_mrr": 0.1827,
        "ndcg@10": 0.2465,
        "ndcg@5": 0.1898
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "res_syn = model.run_eval(valid_file)\n",
    "print(res_syn)\n",
    "pm.record(\"res_syn\", res_syn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reference\n",
    "\\[1\\] Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu and Xing Xie: Neural News Recommendation with Long- and Short-term User Representations, ACL 2019<br>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}