Skip to content

Commit

Permalink
Add a file watcher with watchdog (ipython#29)
Browse files Browse the repository at this point in the history 
* Create a watchdog trick for modules that is sensitive to notebook paths Fixes ipython#28
* Catch an error when a file is created in the watcher
* Use the watcher while developing importnb
* Rename the unittests to something more canonical.
* Add the github pages deployment to travis.
* Fix the watcher instructions in the readme.
* Add the watchdog requirement for the setup
* Manually install pyyaml from to pass 3.7dev
* Add context to the readme
* Deploy pages on any travis_branch
* Escape the profile read error int he ipython extension
* add pip to the cache on travis
  • Loading branch information
@tonyfast
tonyfast authored May 8, 2018
1 parent 7fbda28 commit 0eb26a6
Show file tree
Hide file tree
Showing 20 changed files with 822 additions and 290 deletions.
209 changes: 162 additions & 47 deletions readme.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"__importnb__ supports the ability to use Jupyter notebooks as python source.\n",
"__importnb__ imports notebooks as modules & packages.\n",
"\n",
"[![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/deathbeds/importnb/master?urlpath=lab/tree/readme.ipynb)[![Build Status](https://travis-ci.org/deathbeds/importnb.svg?branch=master)](https://travis-ci.org/deathbeds/importnb)[![PyPI version](https://badge.fury.io/py/importnb.svg)](https://badge.fury.io/py/importnb)![PyPI - Python Version](https://img.shields.io/pypi/pyversions/importnb.svg)![PyPI - Format](https://img.shields.io/pypi/format/importnb.svg)![PyPI - Format](https://img.shields.io/pypi/l/importnb.svg)\n",
"\n",
"\n",
"[![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/deathbeds/importnb/master?urlpath=lab/tree/readme.ipynb)[![Build Status](https://travis-ci.org/deathbeds/importnb.svg?branch=master)](https://travis-ci.org/deathbeds/importnb)\n",
"\n",
" pip install importnb"
]
Expand All @@ -15,9 +17,16 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Jupyter Extension\n",
"# `importnb` works in Python and IPython\n",
"\n",
" %load_ext importnb "
"Use the `Notebook` context manager."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### For brevity"
]
},
{
Expand All @@ -26,17 +35,15 @@
"metadata": {},
"outputs": [],
"source": [
" foo = 42\n",
" import readme\n",
" assert readme.foo is 42\n",
" assert readme.__file__.endswith('.ipynb')"
" with __import__('importnb').Notebook(): \n",
" import readme"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Notebooks maybe reloaded with the standard Python Import machinery."
"#### or explicity "
]
},
{
Expand All @@ -45,44 +52,48 @@
"metadata": {},
"outputs": [],
"source": [
" from importnb import Notebook, reload\n",
" reload(readme);"
" from importnb import Notebook\n",
" with Notebook(): \n",
" import readme"
]
},
{
"cell_type": "markdown",
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"## Unload the extension\n",
"\n",
" %unload_ext importnb"
" foo = 42\n",
" import readme\n",
" assert readme.foo is 42\n",
" assert readme.__file__.endswith('.ipynb')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Context Manager"
"### Modules may be reloaded \n",
"\n",
"The context manager is required to `reload` a module."
]
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
" with Notebook(): \n",
" import readme\n",
" from importlib import reload\n",
" with Notebook():\n",
" reload(readme)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Integrations\n",
"\n",
"`importnb` integrates with IPython, py.test, and setuptools.\n"
"## Integrations\n"
]
},
{
Expand All @@ -91,24 +102,31 @@
"source": [
"### IPython\n",
"\n",
"`importnb` may allow notebooks to import by default with \n",
"#### Extension\n",
"\n",
" ipython -c \"__import__('importnb').utils.ipython.install()\"\n",
"Avoid the use of the context manager using loading importnb as IPython extension.\n",
"\n",
" %load_ext importnb\n",
" \n",
"This extension will install a script into the default IPython profile startup that is called each time an IPython session is created. \n",
"`%unload_ext importnb` will unload the extension.\n",
"\n",
"#### Default Extension\n",
"\n",
"#### Command\n",
"`importnb` may allow notebooks to import by default with \n",
"\n",
"After the `importnb` extension is created notebooks can be execute from the command line.\n",
" importnb-install\n",
" \n",
"This extension will install a script into the default IPython profile startup that is called each time an IPython session is created. \n",
"\n",
" ipython -m readme\n",
"Uninstall the extension with `importnb-install`.\n",
"\n",
"### Unloading the Extension\n",
"##### Run a notebook as a module\n",
"\n",
"The default settings may be discarded temporarily with\n",
"When the default extension is loaded any notebook can be run from the command line. After the `importnb` extension is created notebooks can be execute from the command line.\n",
"\n",
" %unload_ext importnb\n",
" "
" ipython -m readme\n",
" \n",
"> See the [deploy step in the travis build](https://github.com/deathbeds/importnb/blob/docs/.travis.yml#L19)."
]
},
{
Expand All @@ -135,33 +153,89 @@
" ...,)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [Watchdog](https://github.com/gorakhargosh/watchdog/tree/master/src/watchdog/tricks)\n",
"\n",
"`importnb` exports a watchdog trick to watch files and apply command like operations on their module path.\n",
"\n",
"#### Tricks File\n",
"\n",
"For example, create a file called `tricks.yaml` containing\n",
"\n",
" tricks:\n",
" - importnb.utils.watch.ModuleTrick:\n",
" patterns: ['*.ipynb']\n",
" shell_command: ipython -m ${watch_dest_path}\n",
" \n",
"#### Run the watcher in a terminal\n",
"\n",
" watchmedo tricks tricks.yaml\n",
" \n",
"> [`tricks.yaml`](tricks.yaml) is a concrete implementation of `tricks.yaml`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Developer"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"* [Tests](tests/test_importnb.ipynb)\n",
"* [Source Notebooks](src/notebooks/)\n",
"* [Transpiled Python Source](src/importnb/)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Format and test the Source Code"
]
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"src/notebooks/compiler_ipython.ipynb\n",
"src/notebooks/compiler_python.ipynb\n",
"src/notebooks/decoder.ipynb\n",
"src/notebooks/exporter.ipynb\n",
"src/notebooks/loader.ipynb\n",
"src/notebooks/utils/__init__.ipynb\n",
"src/notebooks/utils/ipython.ipynb\n",
"src/notebooks/utils/pytest_plugin.ipynb\n",
"src/notebooks/utils/setup.ipynb\n",
"src/notebooks/utils/watch.ipynb\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"test_import (tests.test_.TestContext) ... ok\n",
"test_reload_with_context (tests.test_.TestContext) ... ok\n",
"test_reload_without_context (tests.test_.TestContext) ... skipped 'importnb is probably installed'\n",
"test_failure (tests.test_.TestExtension) ... expected failure\n",
"test_import (tests.test_.TestExtension) ... ok\n",
"test_exception (tests.test_.TestPartial) ... ok\n",
"test_traceback (tests.test_.TestPartial) ... ok\n",
"test_imports (tests.test_.TestRemote) ... skipped 'requires IP'\n",
"test_import (tests.test_unittests.TestContext) ... ok\n",
"test_reload_with_context (tests.test_unittests.TestContext) ... ok\n",
"test_reload_without_context (tests.test_unittests.TestContext) ... skipped 'importnb is probably installed'\n",
"test_failure (tests.test_unittests.TestExtension) ... expected failure\n",
"test_import (tests.test_unittests.TestExtension) ... ok\n",
"test_exception (tests.test_unittests.TestPartial) ... ok\n",
"test_traceback (tests.test_unittests.TestPartial) ... ok\n",
"test_imports (tests.test_unittests.TestRemote) ... skipped 'requires IP'\n",
"\n",
"----------------------------------------------------------------------\n",
"Ran 8 tests in 2.021s\n",
"Ran 8 tests in 2.023s\n",
"\n",
"OK (skipped=2, expected failures=1)\n"
]
Expand All @@ -171,16 +245,57 @@
" if __name__ == '__main__':\n",
" from pathlib import Path\n",
" import black\n",
" from nbconvert.exporters.markdown import MarkdownExporter\n",
" from importnb.compiler_python import ScriptExporter\n",
" for path in Path('src/notebooks/').rglob(\"\"\"*.ipynb\"\"\"):\n",
" \n",
" 'checkpoint' not in str(path) and (Path('src/importnb') / path.with_suffix('.py').relative_to('src/notebooks')).write_text(\n",
" if 'checkpoint' not in str(path):\n",
" print(path)\n",
" (Path('src/importnb') / path.with_suffix('.py').relative_to('src/notebooks')).write_text(\n",
" black.format_str(ScriptExporter().from_filename(path)[0], 100))\n",
" \n",
" __import__('unittest').main(module='tests', argv=\"discover --verbose\".split(), exit=False) \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Format the Github markdown files"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
" if __name__ == '__main__':\n",
" from nbconvert.exporters.markdown import MarkdownExporter\n",
" for path in map(Path, ('readme.ipynb', 'changelog.ipynb')):\n",
" path.with_suffix('.md').write_text(MarkdownExporter().from_filename(path)[0])\n",
" path.with_suffix('.md').write_text(MarkdownExporter().from_filename(path)[0])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Format the Github Pages documentation\n",
"\n",
" __import__('unittest').main(module='tests', argv=\"discover --verbose\".split(), exit=False)\n"
"We use `/docs` as the `local_dir`."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
" if __name__ == '__main__':\n",
" from nbconvert.exporters.markdown import MarkdownExporter\n",
" files = 'readme.ipynb', 'changelog.ipynb', 'tests/test_importnb.ipynb'\n",
" for doc in map(Path, files):\n",
" to = ('docs' / doc.with_suffix('.md'))\n",
" to.parent.mkdir(exist_ok=True)\n",
" to.write_text(MarkdownExporter().from_filename(doc)[0])"
]
},
{
Expand Down
Loading

0 comments on commit 0eb26a6

Please sign in to comment.