notebooks/training_with_hdfs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0e814db9",
   "metadata": {},
   "source": [
    "<img src=\"http://developer.download.nvidia.com/compute/machine-learning/frameworks/nvidia_logo.png\" style=\"width: 90px; float: right;\">\n",
    "\n",
    "# HugeCTR training with HDFS example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfa44b99",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7166fb3",
   "metadata": {},
   "source": [
    "In version v3.4, we introduced the support for HDFS. Users can now move their data and model files from HDFS to local filesystem through our API to do HugeCTR training. And after training, users can choose to dump the trained parameters and optimizer states into HDFS. In this example notebook, we are going to demonstrate the end to end procedure of training with HDFS."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "025dd2e5",
   "metadata": {},
   "source": [
    "## Get HugeCTR from NGC\n",
    "The HugeCTR Python module is preinstalled in the 22.07 and later [Merlin Training Container](https://ngc.nvidia.com/catalog/containers/nvidia:merlin:merlin-hugectr): `nvcr.io/nvidia/merlin/merlin-hugectr:22.07`.\n",
    "\n",
    "You can check the existence of required libraries by running the following Python code after launching the container.\n",
    "\n",
    "```bash\n",
    "$ python3 -c \"import hugectr\"\n",
    "```\n",
    "\n",
    "> If you prefer to build HugeCTR from the source code instead of using the NGC container, refer to the \n",
    "> [How to Start Your Development](https://nvidia-merlin.github.io/HugeCTR/master/hugectr_contributor_guide.html#how-to-start-your-development)\n",
    "> documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "479ba8e7",
   "metadata": {},
   "source": [
    "## Hadoop Installation and Configuration\n",
    "\n",
    "### Download and Install Hadoop\n",
    "\n",
    "1. Download a JDK:\n",
    "\n",
    "   ```bash\n",
    "   wget https://download.java.net/java/GA/jdk16.0.2/d4a915d82b4c4fbb9bde534da945d746/7/GPL/openjdk-16.0.2_linux-x64_bin.tar.gz\n",
    "   tar -zxvf openjdk-16.0.2_linux-x64_bin.tar.gz\n",
    "   mv jdk-16.0.2 /usr/local\n",
    "   ```\n",
    "\n",
    "2. Set Java environmental variables:\n",
    "\n",
    "   ```bash\n",
    "   export JAVA_HOME=/usr/local/jdk-16.0.2\n",
    "   export JRE_HOME=${JAVA_HOME}/jre\n",
    "   export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib\n",
    "   export PATH=.:${JAVA_HOME}/bin:$PATH\n",
    "   ```\n",
    "\n",
    "3. Download and install Hadoop:\n",
    "\n",
    "   ```bash\n",
    "   wget https://dlcdn.apache.org/hadoop/common/hadoop-3.3.1/hadoop-3.3.1.tar.gz\n",
    "   tar -zxvf hadoop-3.3.1.tar.gz\n",
    "   mv hadoop-3.3.1 /usr/local\n",
    "   ```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d6d48b7",
   "metadata": {},
   "source": [
    "### Hadoop configuration\n",
    "\n",
    "Set Hadoop environment variables:\n",
    "\n",
    "```bash\n",
    "export HDFS_NAMENODE_USER=root\n",
    "export HDFS_DATANODE_USER=root\n",
    "export HDFS_SECONDARYNAMENODE_USER=root\n",
    "export YARN_RESOURCEMANAGER_USER=root\n",
    "export YARN_NODEMANAGER_USER=root\n",
    "echo ‘export JAVA_HOME=/usr/local/jdk-16.0.2’ >> /usr/local/hadoop/etc/hadoop/hadoop-env.sh\n",
    "```\n",
    "\n",
    "`core-site.xml` config:\n",
    "\n",
    "```xml\n",
    "<property>\n",
    "    <name>fs.defaultFS</name>\n",
    "    <value>hdfs://namenode:9000</value>\n",
    "</property>\n",
    "<property>\n",
    "    <name>hadoop.tmp.dir</name>\n",
    "    <value>/usr/local/hadoop/tmp</value>\n",
    "</property>\n",
    "```\n",
    "\n",
    "`hdfs-site.xml` for name node:\n",
    "\n",
    "```xml\n",
    "<property>\n",
    "     <name>dfs.replication</name>\n",
    "     <value>4</value>\n",
    "</property>\n",
    "<property>\n",
    "     <name>dfs.namenode.name.dir</name>\n",
    "     <value>file:/usr/local/hadoop/hdfs/name</value>\n",
    "</property>\n",
    "<property>\n",
    "     <name>dfs.client.block.write.replace-datanode-on-failure.enable</name>         \n",
    "     <value>true</value>\n",
    "</property>\n",
    "<property>\n",
    "    <name>dfs.client.block.write.replace-datanode-on-failure.policy</name>\n",
    "    <value>NEVER</value>\n",
    "</property>\n",
    "```\n",
    "\n",
    "`hdfs-site.xml` for data node:\n",
    "\n",
    "```xml\n",
    "<property>\n",
    "     <name>dfs.replication</name>\n",
    "     <value>4</value>\n",
    "</property>\n",
    "<property>\n",
    "     <name>dfs.datanode.data.dir</name>\n",
    "     <value>file:/usr/local/hadoop/hdfs/data</value>\n",
    "</property>\n",
    "<property>\n",
    "     <name>dfs.client.block.write.replace-datanode-on-failure.enable</name>         \n",
    "     <value>true</value>\n",
    "</property>\n",
    "<property>\n",
    "    <name>dfs.client.block.write.replace-datanode-on-failure.policy</name>\n",
    "    <value>NEVER</value>\n",
    "</property>\n",
    "```\n",
    "\n",
    "`workers` for all node:\n",
    "\n",
    "```bash\n",
    "worker1\n",
    "worker2\n",
    "worker3\n",
    "worker4\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e4e0b59",
   "metadata": {},
   "source": [
    "### Start HDFS\n",
    "\n",
    "1. Enable ssh connection:\n",
    "\n",
    "   ```bash\n",
    "   ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa\n",
    "   cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys\n",
    "   /etc/init.d/ssh start\n",
    "   ```\n",
    "\n",
    "2. Format the NameNode:\n",
    "\n",
    "   ```bash\n",
    "   /usr/local/hadoop/bin/hdfs namenode -format\n",
    "   ```\n",
    "\n",
    "3. Format the DataNodes:\n",
    "\n",
    "   ```bash\n",
    "   /usr/local/hadoop/bin/hdfs datanode -format\n",
    "   ```\n",
    "\n",
    "4. Start HDFS from the NameNode:\n",
    "\n",
    "   ```bash\n",
    "   /usr/local/hadoop/sbin/start-dfs.sh\n",
    "   ```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "399717d1",
   "metadata": {},
   "source": [
    "## Wide and Deep Model\n",
    "\n",
<<<<<<< HEAD
    "In the Docker container, `nvcr.io/nvidia/merlin/merlin-hugectr:22.07`, \n",
=======
    "In the Docker container, `nvcr.io/nvidia/merlin/merlin-hugectr:22.06`, \n",
>>>>>>> master
    "make sure that you installed Hadoop and set the proper environment variables as instructed in the preceding sections.\n",
    "\n",
    "If you chose to compile HugeCTR, make sure you that you set `DENABLE_HDFS` to `ON`.\n",
    "\n",
    "* Run `export CLASSPATH=$(hadoop classpath --glob)` first to link the required JAR file.\n",
    "* Make sure that we have the model files your Hadoop cluster and provide the correct links to the model files.\n",
    "\n",
    "Now you can run the following sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "02510bae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting train_with_hdfs.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile train_with_hdfs.py\n",
    "import hugectr\n",
    "from mpi4py import MPI\n",
    "from hugectr.data import DataSource, DataSourceParams\n",
    "\n",
    "data_source_params = DataSourceParams(\n",
    "    use_hdfs = True, #whether use HDFS to save model files\n",
    "    namenode = 'localhost', #HDFS namenode IP\n",
    "    port = 9000, #HDFS port\n",
    ")\n",
    "\n",
    "solver = hugectr.CreateSolver(max_eval_batches = 1280,\n",
    "                              batchsize_eval = 1024,\n",
    "                              batchsize = 1024,\n",
    "                              lr = 0.001,\n",
    "                              vvgpu = [[0]],\n",
    "                              repeat_dataset = True,\n",
    "                              data_source_params = data_source_params)\n",
    "reader = hugectr.DataReaderParams(data_reader_type = hugectr.DataReaderType_t.Norm,\n",
    "                                  source = ['./wdl_norm/file_list.txt'],\n",
    "                                  eval_source = './wdl_norm/file_list_test.txt',\n",
    "                                  check_type = hugectr.Check_t.Sum)\n",
    "optimizer = hugectr.CreateOptimizer(optimizer_type = hugectr.Optimizer_t.Adam,\n",
    "                                    update_type = hugectr.Update_t.Global,\n",
    "                                    beta1 = 0.9,\n",
    "                                    beta2 = 0.999,\n",
    "                                    epsilon = 0.0000001)\n",
    "model = hugectr.Model(solver, reader, optimizer)\n",
    "model.add(hugectr.Input(label_dim = 1, label_name = \"label\",\n",
    "                        dense_dim = 13, dense_name = \"dense\",\n",
    "                        data_reader_sparse_param_array = \n",
    "                        # the total number of slots should be equal to data_generator_params.num_slot\n",
    "                        [hugectr.DataReaderSparseParam(\"wide_data\", 2, True, 1),\n",
    "                        hugectr.DataReaderSparseParam(\"deep_data\", 1, True, 26)]))\n",
    "model.add(hugectr.SparseEmbedding(embedding_type = hugectr.Embedding_t.DistributedSlotSparseEmbeddingHash, \n",
    "                            workspace_size_per_gpu_in_mb = 69,\n",
    "                            embedding_vec_size = 1,\n",
    "                            combiner = \"sum\",\n",
    "                            sparse_embedding_name = \"sparse_embedding2\",\n",
    "                            bottom_name = \"wide_data\",\n",
    "                            optimizer = optimizer))\n",
    "model.add(hugectr.SparseEmbedding(embedding_type = hugectr.Embedding_t.DistributedSlotSparseEmbeddingHash, \n",
    "                            workspace_size_per_gpu_in_mb = 1074,\n",
    "                            embedding_vec_size = 16,\n",
    "                            combiner = \"sum\",\n",
    "                            sparse_embedding_name = \"sparse_embedding1\",\n",
    "                            bottom_name = \"deep_data\",\n",
    "                            optimizer = optimizer))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.Reshape,\n",
    "                            bottom_names = [\"sparse_embedding1\"],\n",
    "                            top_names = [\"reshape1\"],\n",
    "                            leading_dim=416))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.Reshape,\n",
    "                            bottom_names = [\"sparse_embedding2\"],\n",
    "                            top_names = [\"reshape2\"],\n",
    "                            leading_dim=1))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.Concat,\n",
    "                            bottom_names = [\"reshape1\", \"dense\"],\n",
    "                            top_names = [\"concat1\"]))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.InnerProduct,\n",
    "                            bottom_names = [\"concat1\"],\n",
    "                            top_names = [\"fc1\"],\n",
    "                            num_output=1024))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.ReLU,\n",
    "                            bottom_names = [\"fc1\"],\n",
    "                            top_names = [\"relu1\"]))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.Dropout,\n",
    "                            bottom_names = [\"relu1\"],\n",
    "                            top_names = [\"dropout1\"],\n",
    "                            dropout_rate=0.5))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.InnerProduct,\n",
    "                            bottom_names = [\"dropout1\"],\n",
    "                            top_names = [\"fc2\"],\n",
    "                            num_output=1024))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.ReLU,\n",
    "                            bottom_names = [\"fc2\"],\n",
    "                            top_names = [\"relu2\"]))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.Dropout,\n",
    "                            bottom_names = [\"relu2\"],\n",
    "                            top_names = [\"dropout2\"],\n",
    "                            dropout_rate=0.5))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.InnerProduct,\n",
    "                            bottom_names = [\"dropout2\"],\n",
    "                            top_names = [\"fc3\"],\n",
    "                            num_output=1))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.Add,\n",
    "                            bottom_names = [\"fc3\", \"reshape2\"],\n",
    "                            top_names = [\"add1\"]))\n",
    "model.add(hugectr.DenseLayer(layer_type = hugectr.Layer_t.BinaryCrossEntropyLoss,\n",
    "                            bottom_names = [\"add1\", \"label\"],\n",
    "                            top_names = [\"loss\"]))\n",
    "model.compile()\n",
    "model.summary()\n",
    "\n",
    "model.load_dense_weights('/model/wdl/_dense_1000.model')\n",
    "model.load_dense_optimizer_states('/model/wdl/_opt_dense_1000.model')\n",
    "model.load_sparse_weights(['/model/wdl/0_sparse_1000.model', '/model/wdl/1_sparse_1000.model'])\n",
    "model.load_sparse_optimizer_states(['/model/wdl/0_opt_sparse_1000.model', '/model/wdl/1_opt_sparse_1000.model'])\n",
    "\n",
    "model.fit(max_iter = 1020, display = 200, eval_interval = 500, snapshot = 1000, snapshot_prefix = \"/model/wdl/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b29f1042",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HugeCTR Version: 3.3\n",
      "====================================================Model Init=====================================================\n",
      "[HCTR][09:00:54][WARNING][RK0][main]: The model name is not specified when creating the solver.\n",
      "[HCTR][09:00:54][INFO][RK0][main]: Global seed is 1285686508\n",
      "[HCTR][09:00:55][INFO][RK0][main]: Device to NUMA mapping:\n",
      "  GPU 0 ->  node 0\n",
      "[HCTR][09:00:56][WARNING][RK0][main]: Peer-to-peer access cannot be fully enabled.\n",
      "[HCTR][09:00:56][INFO][RK0][main]: Start all2all warmup\n",
      "[HCTR][09:00:56][INFO][RK0][main]: End all2all warmup\n",
      "[HCTR][09:00:56][INFO][RK0][main]: Using All-reduce algorithm: NCCL\n",
      "[HCTR][09:00:56][INFO][RK0][main]: Device 0: Tesla V100-PCIE-32GB\n",
      "[HCTR][09:00:56][INFO][RK0][main]: num of DataReader workers: 12\n",
      "[HCTR][09:00:56][INFO][RK0][main]: max_vocabulary_size_per_gpu_=6029312\n",
      "[HCTR][09:00:56][INFO][RK0][main]: max_vocabulary_size_per_gpu_=5865472\n",
      "[HCTR][09:00:56][INFO][RK0][main]: Graph analysis to resolve tensor dependency\n",
      "===================================================Model Compile===================================================\n",
      "[HCTR][09:01:00][INFO][RK0][main]: gpu0 start to init embedding\n",
      "[HCTR][09:01:00][INFO][RK0][main]: gpu0 init embedding done\n",
      "[HCTR][09:01:00][INFO][RK0][main]: gpu0 start to init embedding\n",
      "[HCTR][09:01:00][INFO][RK0][main]: gpu0 init embedding done\n",
      "[HCTR][09:01:00][INFO][RK0][main]: Starting AUC NCCL warm-up\n",
      "[HCTR][09:01:00][INFO][RK0][main]: Warm-up done\n",
      "[HCTR][09:01:00][INFO][RK0][main]: ===================================================Model Summary===================================================\n",
      "label                                   Dense                         Sparse                        \n",
      "label                                   dense                          wide_data,deep_data           \n",
      "(None, 1)                               (None, 13)                              \n",
      "——————————————————————————————————————————————————————————————————————————————————————————————————————————————————\n",
      "Layer Type                              Input Name                    Output Name                   Output Shape                  \n",
      "——————————————————————————————————————————————————————————————————————————————————————————————————————————————————\n",
      "LocalizedSlotSparseEmbeddingHash        wide_data                     sparse_embedding2             (None, 1, 1)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "LocalizedSlotSparseEmbeddingHash        deep_data                     sparse_embedding1             (None, 26, 16)                \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Reshape                                 sparse_embedding1             reshape1                      (None, 416)                   \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Reshape                                 sparse_embedding2             reshape2                      (None, 1)                     \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Concat                                  reshape1                      concat1                       (None, 429)                   \n",
      "                                        dense                                                                                     \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            concat1                       fc1                           (None, 1024)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "ReLU                                    fc1                           relu1                         (None, 1024)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Dropout                                 relu1                         dropout1                      (None, 1024)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            dropout1                      fc2                           (None, 1024)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "ReLU                                    fc2                           relu2                         (None, 1024)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Dropout                                 relu2                         dropout2                      (None, 1024)                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "InnerProduct                            dropout2                      fc3                           (None, 1)                     \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "Add                                     fc3                           add1                          (None, 1)                     \n",
      "                                        reshape2                                                                                  \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "BinaryCrossEntropyLoss                  add1                          loss                                                        \n",
      "                                        label                                                                                     \n",
      "------------------------------------------------------------------------------------------------------------------\n",
      "2022-02-23 09:01:00,548 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "[HDFS][INFO]: Read file /model/wdl/_dense_1000.model successfully!\n",
      "[HDFS][INFO]: Read file /model/wdl/_opt_dense_1000.model successfully!\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Loading dense opt states: \n",
      "[HCTR][09:01:01][INFO][RK0][main]: Loading sparse model: /model/wdl/0_sparse_1000.model\n",
      "[HDFS][INFO]: Read file /model/wdl/0_sparse_1000.model/key successfully!\n",
      "[HDFS][INFO]: Read file /model/wdl/0_sparse_1000.model/slot_id successfully!\n",
      "[HDFS][INFO]: Read file /model/wdl/0_sparse_1000.model/emb_vector successfully!\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Start to upload embedding table file to GPUs, total loop_num: 128\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Loading sparse model: /model/wdl/1_sparse_1000.model\n",
      "[HDFS][INFO]: Read file /model/wdl/1_sparse_1000.model/key successfully!\n",
      "[HDFS][INFO]: Read file /model/wdl/1_sparse_1000.model/slot_id successfully!\n",
      "[HDFS][INFO]: Read file /model/wdl/1_sparse_1000.model/emb_vector successfully!\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Start to upload embedding table file to GPUs, total loop_num: 518\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Loading sparse optimizer states: /model/wdl/0_opt_sparse_1000.model\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Rank0: Read optimzer state from file\n",
      "[HDFS][INFO]: Read file /model/wdl/0_opt_sparse_1000.model successfully!\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Rank0: Read optimzer state from file\n",
      "[HDFS][INFO]: Read file /model/wdl/0_opt_sparse_1000.model successfully!\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Loading sparse optimizer states: /model/wdl/1_opt_sparse_1000.model\n",
      "[HCTR][09:01:01][INFO][RK0][main]: Rank0: Read optimzer state from file\n",
      "[HDFS][INFO]: Read file /model/wdl/1_opt_sparse_1000.model successfully!\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Rank0: Read optimzer state from file\n",
      "[HDFS][INFO]: Read file /model/wdl/1_opt_sparse_1000.model successfully!\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Done\n",
      "=====================================================Model Fit=====================================================\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Use non-epoch mode with number of iterations: 1020\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Training batchsize: 1024, evaluation batchsize: 1024\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Evaluation interval: 500, snapshot interval: 1000\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Dense network trainable: True\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Sparse embedding sparse_embedding1 trainable: True\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Sparse embedding sparse_embedding2 trainable: True\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Use mixed precision: False, scaler: 1.000000, use cuda graph: True\n",
      "[HCTR][09:01:02][INFO][RK0][main]: lr: 0.001000, warmup_steps: 1, end_lr: 0.000000\n",
      "[HCTR][09:01:02][INFO][RK0][main]: decay_start: 0, decay_steps: 1, decay_power: 2.000000\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Training source file: ./wdl_norm/file_list.txt\n",
      "[HCTR][09:01:02][INFO][RK0][main]: Evaluation source file: ./wdl_norm/file_list_test.txt\n",
      "[HCTR][09:01:04][INFO][RK0][main]: Iter: 200 Time(200 iters): 1.12465s Loss: 0.632464 lr:0.001\n",
      "[HCTR][09:01:05][INFO][RK0][main]: Iter: 400 Time(200 iters): 1.03567s Loss: 0.612515 lr:0.001\n",
      "[HCTR][09:01:06][INFO][RK0][main]: Evaluation, AUC: 0.499877\n",
      "[HCTR][09:01:06][INFO][RK0][main]: Eval Time for 1280 iters: 0.647875s\n",
      "[HCTR][09:01:06][INFO][RK0][main]: Iter: 600 Time(200 iters): 1.68717s Loss: 0.625102 lr:0.001\n",
      "[HCTR][09:01:07][INFO][RK0][main]: Iter: 800 Time(200 iters): 1.03752s Loss: 0.608092 lr:0.001\n",
      "[HCTR][09:01:08][INFO][RK0][main]: Iter: 1000 Time(200 iters): 1.03691s Loss: 0.688194 lr:0.001\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Evaluation, AUC: 0.500383\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Eval Time for 1280 iters: 0.650671s\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Rank0: Dump hash table from GPU0\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Rank0: Write hash table <key,value> pairs to file\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/0_sparse_1000.model/key successfully!\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/0_sparse_1000.model/slot_id successfully!\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/0_sparse_1000.model/emb_vector successfully!\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Rank0: Dump hash table from GPU0\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Rank0: Write hash table <key,value> pairs to file\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/1_sparse_1000.model/key successfully!\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/1_sparse_1000.model/slot_id successfully!\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/1_sparse_1000.model/emb_vector successfully!\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Dumping sparse weights to files, successful\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Rank0: Write optimzer state to file\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/0_opt_sparse_1000.model successfully!\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:09][INFO][RK0][main]: Rank0: Write optimzer state to file\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/0_opt_sparse_1000.model successfully!\n",
      "[HCTR][09:01:10][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:10][INFO][RK0][main]: Rank0: Write optimzer state to file\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/1_opt_sparse_1000.model successfully!\n",
      "[HCTR][09:01:11][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:11][INFO][RK0][main]: Rank0: Write optimzer state to file\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/1_opt_sparse_1000.model successfully!\n",
      "[HCTR][09:01:12][INFO][RK0][main]: Done\n",
      "[HCTR][09:01:12][INFO][RK0][main]: Dumping sparse optimzer states to files, successful\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/_dense_1000.model successfully!\n",
      "[HCTR][09:01:12][INFO][RK0][main]: Dumping dense weights to HDFS, successful\n",
      "[HDFS][INFO]: Write to HDFS /model/wdl/_opt_dense_1000.model successfully!\n",
      "[HCTR][09:01:12][INFO][RK0][main]: Dumping dense optimizer states to HDFS, successful\n",
      "[HCTR][09:01:12][INFO][RK0][main]: Finish 1020 iterations with batchsize: 1024 in 9.82s.\n"
     ]
    }
   ],
   "source": [
    "!python train_with_hdfs.py"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}