Small fixes, refactorings and updates (#15)

* fix pylint_notebook: markdown cells processing * Fix bug, change colors / resource names, change separator * Cache, vertical scroll, add help * Update doc, README, cli help * Update pylintrc * Fix width in readme * Update test-install * add changes in run_notebook and rewrite some docs * Speed up formatting * small fixes for exec_notebook * Change separators to thin * Update NBstat tutorial * Self-review; more separator controls * More F-keys; change table elements * Make pylint happy * update pylint_notebook * text prettifyings * Fix empty table handling * Update version --------- Co-authored-by: Sergey Tsimfer <[email protected]> Co-authored-by: Sergey Tsimfer <[email protected]>
analysiscenter · May 23, 2024 · 819668b · 819668b
1 parent 2f6adf9
commit 819668b
Show file tree

Hide file tree

Showing 16 changed files with 513 additions and 255 deletions.
diff --git a/.github/workflows/test-install.yml b/.github/workflows/test-install.yml
@@ -16,12 +16,12 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, macos-latest, windows-latest]
-        python-version: [3.8, 3.9]
+        python-version: [3.8, 3.9, '3.10', 3.11]
 
     runs-on: ${{ matrix.os }}
 
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v4
 
     - name: Set up Python ${{ matrix.python-version }} on ${{ matrix.os }}
       uses: actions/setup-python@v2
@@ -47,7 +47,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, macos-latest, windows-latest]
-        python-version: [3.8, 3.9]
+        python-version: [3.8, 3.9, '3.10', 3.11]
 
     runs-on: ${{ matrix.os }}
 

diff --git a/README.md b/README.md
@@ -13,21 +13,12 @@ The main tool of this package is **nbstat** / **nbwatch** command line utility.
 
 <img src="images/nbwatch.gif" width="90%"/>
 
-While in the `watch` mode, you can hit buttons to modify the displayed view:
-
-* `tab` — swaps views, from `nbwatch` to `devicewatch` and back.
-* `b` — toggles bar representation for some of the resources: in addition to its value, show colored bar.
-* `m` — toggles moving average column for some of the resources: values are averaged across the latest iterations.
-* `s` — toggles table separators.
-
-We also add the **devicestat** and **devicewatch** commands that show transposed view with the same information and parameters.
-
 For more information, check out the full [user documentation:](nbtools/nbstat/README.md) explanation of different table views, command line options and ready-to-use snippets.
 
 
 ### Troubleshooting: PID namespaces, user permissions and zombie processes
 A [known problem](https://github.com/NVIDIA/nvidia-docker/issues/179) of NVIDIA drivers is that **nvidia-smi** reports PIDs of processes on devices in the global namespace, not in the container namespace, which does not allow to match PIDs of container processes to their device PIDs. There are a few workarounds:
-* pass `--pid=host` flag to `docker run`.
+* [recommended] pass `--pid=host` flag to `docker run`.
 * patch NVIDIA driver to handle PID namespaces correctly.
 * [Linux only] fallback on manually inspecting */proc/PID/* files to identify the host PID for each process inside of the container.
 
@@ -42,8 +33,11 @@ In order to inspect certain properties of processes, we rely on having all neces
 ### Contribute
 If you are interested to contribute, check out the [developer/contributor page.](nbtools/nbstat/DEV.md) It contains detailed description about inner workings of the library, my design choices and motivation behind them, as well as discussion of complexities along the way.
 
+## Library
+Other than `nbstat / nbwatch` monitoring utilities, this library provides a few useful tools for working with notebooks and GPUs.
+
 
-## **pylint_notebook**
+### **pylint_notebook**
 Shamelessly taken from [pylint page:](https://pylint.pycqa.org/en/latest/)
 
 Function that checks for errors in Jupyter Notebooks with Python code, tries to enforce a coding standard and looks for code smells. It can also look for certain type errors, it can recommend suggestions about how particular blocks can be refactored and can offer you details about the code's complexity.
@@ -58,37 +52,38 @@ pylint_notebook(path_to_ipynb,             # If not provided, use path to the cu
 
 Under the hood, it converts `.ipynb` notebook to `.py` script, creates a custom `.pylintrc` configuration, runs the `pylint` and removes all temporary files. Learn more about its usage in the [tutorial.](tutorials/NBstat.ipynb)
 
-## **run_notebook**
+### **exec_notebook**
 Provides a `eval`-like interface for running Jupyter Notebooks programmatically. We use it for running interactive tests, that are easy to work with: in case of any failures, one can jump right into fixing it with an already set-up environment.
 
 ```python
-from nbtools import run_notebook
-run_notebook(path_to_ipynb,                       # Which notebook to run
-             out_path_ipynb,                      # Where to save result
-             inputs={'learning_rate': 0.05,},     # Pass variables to notebook
-             outputs=['accuracy'])                # Extract variables from notebook
+from nbtools import exec_notebook
+exec_notebook(path_to_ipynb,                       # Which notebook to run
+              out_path_ipynb,                      # Where to save result
+              inputs={'learning_rate': 0.05,},     # Pass variables to notebook
+              outputs=['accuracy'])                # Extract variables from notebook
 ```
 
 
-## **set_gpus**
+### **set_gpus, free_gpus**
 Select free device(s) and set `CUDA_VISIBLE_DEVICES` environment variable so that the current process sees only them.
 
 Eliminates an enormous amount of bugs and unexpected behaviors related to GPU usage.
 
 ```python
-from nbtools import set_gpus
-set_gpus(n=2,                              # Number of devices to set.
-         min_free_memory=0.7,              # Minimum amount of free memory on device to consider it free.
-         max_processes=3)                  # Maximum amount of  processes  on device to consider it free.
+from nbtools import set_gpus, free_gpus
+used_gpus = set_gpus(n=2,                # Number of devices to set.
+                     min_free_memory=0.7,# Minimum amount of free memory on device to consider free.
+                     max_processes=3)    # Maximum amount of  processes  on device to consider free.
+free_gpus(used_gpus)                     # Kill all processes on selected GPUs. Useful at teardown.
 ```
 
-## Other functions
+### Other functions
 ```python
-from nbtools import (in_notebook,          # Return True if executed inside of Jupyter Notebook
-                     get_notebook_path,    # If executed in Jupyter Notebook, return its absolute path
-                     get_notebook_name,    # If executed in Jupyter Notebook, return its name
-                     notebook_to_script)   # Convert Jupyter Notebook to an executable Python script.
-                                           # Works well with magics and command line executions.
+from nbtools import (in_notebook,         # Return True if executed inside of Jupyter Notebook
+                     get_notebook_path,   # If executed in Jupyter Notebook, return its absolute path
+                     get_notebook_name,   # If executed in Jupyter Notebook, return its name
+                     notebook_to_script)  # Convert Jupyter Notebook to an executable Python script.
+                                          # Works well with magics and command line executions.
 ```
 
 

diff --git a/nbtools/__init__.py b/nbtools/__init__.py
@@ -1,7 +1,7 @@
 """ Init file. """
 #pylint: disable=wildcard-import
 from .core import *
-from .run_notebook import run_notebook
+from .exec_notebook import exec_notebook, run_notebook
 from .pylint_notebook import pylint_notebook
 
-__version__ = '0.9.12'
+__version__ = '0.9.13'
diff --git a/nbtools/core.py b/nbtools/core.py
@@ -90,19 +90,23 @@ def notebook_to_script(path_script, path_notebook=None, ignore_markdown=True, re
             continue
 
         cell_lines = cell['source'].split('\n')
-        cell_lines.insert(0, f'\n### [{cell_number}] cell')
+
+        if cell['cell_type'] == 'code':
+            cell_lines.insert(0, f'\n### [{cell_number}] cell')
 
         # Comment cell/line magics
         for j, line in enumerate(cell_lines):
-            if line.startswith('%') or line.startswith('!'):
+            if line.startswith('%') or line.startswith('!') or cell['cell_type'] != 'code':
                 cell_lines[j] = '### ' + line
 
         code_line_number = len(code_lines) + 1
         cell_line_numbers[cell_number] = range(code_line_number, code_line_number + len(cell_lines))
 
         code_lines.extend([line.strip('\n') for line in cell_lines])
         code_lines.append('')
-        cell_number += 1
+
+        if cell['cell_type'] == 'code':
+            cell_number += 1
 
     code = '\n'.join(code_lines).strip()
 
@@ -135,7 +139,8 @@ def get_available_gpus(n=1, min_free_memory=0.9, max_processes=2, verbose=False,
 
     Returns
     -------
-    List with indices of available GPUs
+    available_devices : list
+        Indices of available GPUs.
     """
     try:
         import nvidia_smi
@@ -211,6 +216,11 @@ def set_gpus(n=1, min_free_memory=0.9, max_processes=2, verbose=False, raise_err
         If 2, then display memory and process information for each device.
     raise_error : bool
         Whether to raise an exception if not enough devices are available.
+
+    Returns
+    -------
+    devices : list
+        Indices of selected and reserved GPUs.
     """
     #pylint: disable=consider-iterating-dictionary
     if 'CUDA_VISIBLE_DEVICES' in os.environ.keys():