[IPython] Source inspection dispatcher for better IDLE compatibility #1222

archibate · 2020-06-11T16:23:52Z

Related issue = close #1221 close #1136

Inspired by https://stackoverflow.com/questions/60797338/get-ast-from-python-object-in-interpreter.

Working in python native shell:

(dill) [bate@archit taichi]$ python
Python 3.8.2 (default, Feb 26 2020, 22:21:03) 
[GCC 9.2.1 20200130] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import taichi as ti
[Taichi] mode=development
[Taichi] preparing sandbox at /tmp/taichi-440b8jj2
[Taichi] <dev mode>, supported archs: [cpu, cuda, opengl], commit fbdb7ece, python 3.8.2
>>> @ti.kernel
... def f(x: ti.f32) -> ti.f32: return x ** 2
... 
>>> f(1.414)
1.9993960857391357
>>>

~~Not working in ipython shell:~~ working now.

WANT_VER = v0.6.10

archibate · 2020-06-11T16:34:28Z

Not sure if this helps: https://ipython.readthedocs.io/en/stable/api/generated/IPython.core.oinspect.html?highlight=getsource

In [1]: import IPython                                                                                                                                                                       

In [2]: def f(): return 233                                                                                                                                                                  

In [3]: IPython.core.oinspect.getsource(f)                                                                                                                                                   
Out[3]: 'def f(): return 233\n'

Sleepy now to prevent class drop, will fix this tmr.

archibate · 2020-06-12T02:55:20Z

I can now confirm that Jupyter is working with Taichi happily :)

Eydcao · 2020-06-12T03:54:41Z

Hi @archibate , good morning!

I will try to do some simple tests and review it once I am free this evening. May you have enjoyed your course :>

archibate · 2020-06-12T05:25:27Z

Python 3.8.2 (default, Feb 26 2020, 22:21:03) 
[GCC 9.2.1 20200130] on linux
Type "help", "copyright", "credits" or "license()" for more information.
>>> import taichi as ti
[Taichi] mode=development
[Taichi] preparing sandbox at /tmp/taichi-xffbpj30
[Taichi] <dev mode>, supported archs: [cpu, cuda, opengl], commit fbdb7ece, python 3.8.2
[Taichi] Interactive shell detected: Python IDLE shell
[Taichi] File ".tmp_idle_source" cleaned
>>> @ti.kernel
def func(): pass

>>> func()
>>> ti.reset()
[Taichi] File ".tmp_idle_source" cleaned
>>> @ti.kernel
def func()->ti.f32: return 233

>>> func()
233.0
>>>

IDLE OK.

archibate · 2020-06-12T06:50:43Z

Hello? Let's try to merge this before v0.6.10, IDLE & IPython fans would like my PR :)

archibate · 2020-06-12T07:04:54Z

Do you have IPython or IDLE on your windows machine? I would appreciate very much if you could test this PR for me ❤️

archibate · 2020-06-23T00:38:50Z

Should I separate IDLE to another PR?

yuanming-hu · 2020-06-23T00:45:55Z

In fact, Taichi works in
IDLE file mode

not work in:
IDLE interactive mode
IPython interactive mode
IPython file mode
Jupyter notebook
Python native shell

This PR fix all these non-works.

On my end, it seems to me that Taichi works in IPython interactive mode, IPython file mode, Jupyter notebook. What do your IPython and Jupyter notebook report when Taichi doesn't work there?

archibate · 2020-06-23T01:17:00Z

On my end, it seems to me that Taichi works in IPython interactive mode, IPython file mode, Jupyter notebook. What do your IPython and Jupyter notebook report when Taichi doesn't work there?

Could not get source code IIRC... will test later.

archibate · 2020-06-23T01:22:24Z

Oh sorry! It did work:

[bate@archit ~]$ ipython
iPython 3.8.2 (default, Feb 26 2020, 22:21:03) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.15.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import taichi as ti                                                                  
[Taichi] mode=release
[Taichi] version 0.6.11, supported archs: [cpu, cuda, opengl], commit 762aca58, python 3.8.2

In [2]: @ti.kernel 
   ...: def func(): 
   ...:     pass 
   ...:                                                                                      

In [3]: func()

My previous experience of IPython failure is because I used dill.source to replace inspect.
Seems inspect can deal with IPython while dill.source not.
But dill.source can deal with Python native shell while inspect not.

[bate@archit ~]$ python
Python 3.8.2 (default, Feb 26 2020, 22:21:03) 
[GCC 9.2.1 20200130] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import taichi as ti
[Taichi] mode=release
[Taichi] version 0.6.11, supported archs: [cpu, cuda, opengl], commit 762aca58, python 3.8.2
>>> @ti.kernel
... def func(): pass
... 
>>> func()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/bate/.local/lib/python3.8/site-packages/taichi/lang/kernel.py", line 527, in wrapped
    return primal(*args, **kwargs)
  File "/home/bate/.local/lib/python3.8/site-packages/taichi/lang/kernel.py", line 458, in __call__
    self.materialize(key=key, args=args, arg_features=arg_features)
  File "/home/bate/.local/lib/python3.8/site-packages/taichi/lang/kernel.py", line 263, in materialize
    src = remove_indent(inspect.getsource(self.func))
  File "/usr/lib/python3.8/inspect.py", line 985, in getsource
    lines, lnum = getsourcelines(object)
  File "/usr/lib/python3.8/inspect.py", line 967, in getsourcelines
    lines, lnum = findsource(object)
  File "/usr/lib/python3.8/inspect.py", line 798, in findsource
    raise OSError('could not get source code')
OSError: could not get source code

So we still need both of them.

archibate · 2020-06-23T08:39:46Z

Ok, now I reverted IDLE part and here's only Python native shell and IPython shell part now. Hope this make your review faster.

yuanming-hu

Thanks! I think the code mostly looks good. However, I'm still confused: given that IPython and Jupyter notebook already works well, do we need to do anything about them? Also maybe we should update the title and remove "IPython" there?

archibate · 2020-06-23T15:24:46Z

I know, but it's wise to use the IPython builtin inspector for better compatibility, also the framework may apply to IDLE and other shells.
Also note that kernel print's, since they're print to cout, not recognized by IPython GUI and IDLE at all, and many GAMES201 user is considering this as a bug...
So this PR, the ShellInspectorWrapper is not to serve IPython only, but a general framework for our future work about these interactive shells.

yuanming-hu

I see. SGTM! Thank you for adding this feature. Maybe change the PR title into something like "Source inspection dispatcher for better IDLE compatibility"?

python/taichi/lang/shell.py

Co-authored-by: Yuanming Hu <[email protected]>

@rexwangcc

…aichi-dev#1222) * [Misc] Use 'dill.source' of 'inspect' to run codes in interactive shell * [skip ci] Add dill to CI * _ShellInspectorWrapper * Hack IDLE to make it happy * Fix IDLE * [skip ci] cache: we do care user experience!! * improve stability * [skip ci] fix example exit * [skip ci] fix exit in IPython * [skip ci] improve comment * fix exec risk in ti debug (@rexwangcc) * [skip ci] Fix ti debug too verbose * Better line * fix test_cli * [skip ci] idle_hacker.py * fix typo and improve hacker * [skip ci] reimprov * improve idle inspector * [skip ci] add tag [IPython] * revert idle * [skip ci] revert off-topic cuda debug * [skip ci] clean * [skip ci] enforce code format * [skip ci] no atexit * fix * [skip ci] Update python/taichi/lang/shell.py Co-authored-by: Yuanming Hu <[email protected]> * add dill to jenkins Co-authored-by: Taichi Gardener <[email protected]> Co-authored-by: Yuanming Hu <[email protected]> Co-authored-by: Yuanming Hu <[email protected]>

tyoc213 · 2020-06-29T05:43:11Z

Hi there, just tried this (I want to test convert an image readed from disk to gray scale, but just starting)

$ python
Python 3.7.6 | packaged by conda-forge | (default, Jun  1 2020, 18:57:50) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> import taichi
[Taichi] mode=release
[Taichi] version 0.6.12, supported archs: [cpu, cuda, opengl], commit 42139736, python 3.7.6
>>>                   title=title, ctx=ctx)
  File "<stdin>", line 1
    title=title, ctx=ctx)
    ^
IndentationError: unexpected indent
>>> 
>>> @taichi.kernel
... def paint(img:numpy.ndarray):
...     #for I in ti.grouped(pixel):
...     #    pixel[I] = ti.random() * 255
...     res = img.dot([0.2989, 0.5870, 0.1140])
...     return res
... 
>>> paint(numpy.ndarray([1,1,1])
... 
... )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tyoc213/miniconda3/envs/fastai2/lib/python3.7/site-packages/taichi/lang/kernel.py", line 527, in wrapped
    return primal(*args, **kwargs)
  File "/home/tyoc213/miniconda3/envs/fastai2/lib/python3.7/site-packages/taichi/lang/kernel.py", line 458, in __call__
    self.materialize(key=key, args=args, arg_features=arg_features)
  File "/home/tyoc213/miniconda3/envs/fastai2/lib/python3.7/site-packages/taichi/lang/kernel.py", line 263, in materialize
    src = remove_indent(inspect.getsource(self.func))
  File "/home/tyoc213/miniconda3/envs/fastai2/lib/python3.7/inspect.py", line 973, in getsource
    lines, lnum = getsourcelines(object)
  File "/home/tyoc213/miniconda3/envs/fastai2/lib/python3.7/inspect.py", line 955, in getsourcelines
    lines, lnum = findsource(object)
  File "/home/tyoc213/miniconda3/envs/fastai2/lib/python3.7/inspect.py", line 786, in findsource
    raise OSError('could not get source code')
OSError: could not get source code

I got this also on my jupyter notebook with almost same code.

I uninstalled and installed nightly and it istalled 5.11.
I uninstalled nighlty and installed again normal package, it is updated to 5.14

$ python
Python 3.7.6 | packaged by conda-forge | (default, Jun  1 2020, 18:57:50) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import taichi
[Taichi] mode=release
[Taichi] version 0.6.14, llvm 10.0.0, commit c83a7e3d, python 3.7.6
[Taichi] Interactive shell detected: Python shell
>>> import numpy
>>> @taichi.kernel
... def paint(img:numpy.ndarray):
...   res = img.dot([0.2989, 0.5870, 0.1140])
...   return res
... 
>>> paint(numpy.ndarray([1,1,1])
... 
... )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tyoc213/miniconda3/envs/fastai2/lib/python3.7/site-packages/taichi/lang/kernel.py", line 528, in wrapped
    return primal(*args, **kwargs)
  File "/home/tyoc213/miniconda3/envs/fastai2/lib/python3.7/site-packages/taichi/lang/kernel.py", line 459, in __call__
    self.materialize(key=key, args=args, arg_features=arg_features)
  File "/home/tyoc213/miniconda3/envs/fastai2/lib/python3.7/site-packages/taichi/lang/kernel.py", line 296, in materialize
    visitor.visit(tree)
  File "/home/tyoc213/miniconda3/envs/fastai2/lib/python3.7/ast.py", line 271, in visit
    return visitor(node)
  File "/home/tyoc213/miniconda3/envs/fastai2/lib/python3.7/site-packages/taichi/lang/transformer.py", line 568, in visit_Module
    self.generic_visit(node)
  File "/home/tyoc213/miniconda3/envs/fastai2/lib/python3.7/site-packages/taichi/lang/transformer.py", line 74, in generic_visit
    value = self.visit(value)
  File "/home/tyoc213/miniconda3/envs/fastai2/lib/python3.7/ast.py", line 271, in visit
    return visitor(node)
  File "/home/tyoc213/miniconda3/envs/fastai2/lib/python3.7/site-packages/taichi/lang/transformer.py", line 666, in visit_FunctionDef
    self.generic_visit(node)
  File "/home/tyoc213/miniconda3/envs/fastai2/lib/python3.7/site-packages/taichi/lang/transformer.py", line 74, in generic_visit
    value = self.visit(value)
  File "/home/tyoc213/miniconda3/envs/fastai2/lib/python3.7/ast.py", line 271, in visit
    return visitor(node)
  File "/home/tyoc213/miniconda3/envs/fastai2/lib/python3.7/site-packages/taichi/lang/transformer.py", line 766, in visit_Return
    'kernel with return value must be '
taichi.lang.exception.TaichiSyntaxError: kernel with return value must be annotated with a return type, e.g. def func() -> ti.f32

So it runs finds it... now I need to learn to use taichi.

Sorry for the "updates" I runned this latest update on jupyter notebook

With the cell

import numpy
@taichi.kernel
def taichi_paint(img:numpy.ndarray) -> taichi.f32:
    res = img.dot([0.1,0.1,0.1])
    return res

def doit():
    fname = str(files[0])
    infile = taichi.imread(fname, 3)
    i = taichi_paint(infile)
    return i

doit()

The output

---------------------------------------------------------------------------
TaichiSyntaxError                         Traceback (most recent call last)
<ipython-input-10-b457d520650c> in <module>
     11     return i
     12 
---> 13 doit()

<ipython-input-10-b457d520650c> in doit()
      8     fname = str(files[0])
      9     infile = taichi.imread(fname, 3)
---> 10     i = taichi_paint(infile)
     11     return i
     12 

~/miniconda3/envs/fastai2/lib/python3.7/site-packages/taichi/lang/kernel.py in wrapped(*args, **kwargs)
    526         @functools.wraps(func)
    527         def wrapped(*args, **kwargs):
--> 528             return primal(*args, **kwargs)
    529 
    530         wrapped.grad = adjoint

~/miniconda3/envs/fastai2/lib/python3.7/site-packages/taichi/lang/kernel.py in __call__(self, *args, **kwargs)
    457         instance_id, arg_features = self.mapper.lookup(args)
    458         key = (self.func, instance_id)
--> 459         self.materialize(key=key, args=args, arg_features=arg_features)
    460         return self.compiled_functions[key](*args)
    461 

~/miniconda3/envs/fastai2/lib/python3.7/site-packages/taichi/lang/kernel.py in materialize(self, key, args, arg_features)
    337             self.runtime.inside_kernel = False
    338 
--> 339         taichi_kernel = taichi_kernel.define(taichi_ast_generator)
    340 
    341         assert key not in self.compiled_functions

~/miniconda3/envs/fastai2/lib/python3.7/site-packages/taichi/lang/kernel.py in taichi_ast_generator()
    331                 import taichi as ti
    332                 raise ti.TaichiSyntaxError(
--> 333                     "Kernels cannot call other kernels. I.e., nested kernels are not allowed. Please check if you have direct/indirect invocation of kernels within kernels. Note that some methods provided by the Taichi standard library may invoke kernels, and please move their invocations to Python-scope."
    334                 )
    335             self.runtime.inside_kernel = True

TaichiSyntaxError: Kernels cannot call other kernels. I.e., nested kernels are not allowed. Please check if you have direct/indirect invocation of kernels within kernels. Note that some methods provided by the Taichi standard library may invoke kernels, and please move their invocations to Python-scope.

[Misc] Use 'dill.source' of 'inspect' to run codes in interactive shell

80fd302

archibate requested review from yuanming-hu, k-ye and taichi-gardener June 11, 2020 16:23

merge master

6ee33d0

archibate changed the title ~~[Misc] Use 'dill.source' of 'inspect' to run codes in interactive shell~~ [Misc] Use 'dill.source' instead of 'inspect' to run in IPython Jun 11, 2020

archibate changed the title ~~[Misc] Use 'dill.source' instead of 'inspect' to run in IPython~~ [Misc] Use 'dill.source' instead of 'inspect' to run in python interactive shell Jun 11, 2020

archibate changed the title ~~[Misc] Use 'dill.source' instead of 'inspect' to run in python interactive shell~~ [Misc] Use 'dill.source.getsource' to run in Python interactive shell Jun 11, 2020

archibate added 2 commits June 12, 2020 00:35

[skip ci] Add dill to CI

4da77a6

_ShellInspectorWrapper

d64c2c2

archibate changed the title ~~[Misc] Use 'dill.source.getsource' to run in Python interactive shell~~ [Misc] Support IPython and Python interactive shell Jun 12, 2020

archibate requested a review from Eydcao June 12, 2020 03:42

archibate added 2 commits June 12, 2020 13:10

Hack IDLE to make it happy

3d629bf

Fix IDLE

c495d70

archibate changed the title ~~[Misc] Support IPython and Python interactive shell~~ [Misc] Support IPython and IDLE as interactive shell Jun 12, 2020

archibate requested a review from rexwangcc June 12, 2020 05:26

archibate added 2 commits June 12, 2020 13:31

[skip ci] cache: we do care user experience!!

742ca63

improve stability

d36c58b

[skip ci] fix example exit

17785f8

archibate requested review from xumingkuan and removed request for rexwangcc June 12, 2020 07:03

archibate added the skip ci label Jun 12, 2020

[skip ci] Merge branch 'master' into dill

26011ee

This comment has been minimized.

Sign in to view

archibate added 2 commits June 23, 2020 16:29

[skip ci] Merge branch 'master' into dill

411af8b

revert idle

794bf05

archibate changed the title ~~[IPython] Support IPython and IDLE as interactive shell~~ [IPython] Support Python and IPython interactive shell Jun 23, 2020

archibate mentioned this pull request Jun 23, 2020

[IPython] Support IDLE and Blender scripting module as interactive shell #1308

Closed

archibate and others added 4 commits June 23, 2020 16:41

[skip ci] revert off-topic cuda debug

44a194d

[skip ci] clean

4a5c37a

[skip ci] enforce code format

bfaeb84

[skip ci] no atexit

343feca

archibate assigned yuanming-hu Jun 23, 2020

fix

7b759f0

yuanming-hu reviewed Jun 23, 2020

View reviewed changes

yuanming-hu approved these changes Jun 23, 2020

View reviewed changes

python/taichi/lang/shell.py Outdated Show resolved Hide resolved

archibate changed the title ~~[IPython] Support Python and IPython interactive shell~~ [IPython] Source inspection dispatcher for better IDLE compatibility Jun 23, 2020

[skip ci] Update python/taichi/lang/shell.py

f12b8d9

Co-authored-by: Yuanming Hu <[email protected]>

archibate added LGTM and removed skip ci labels Jun 23, 2020

yuanming-hu and others added 3 commits June 23, 2020 18:47

add dill to jenkins

e4f1900

Merge branch 'master' into dill

fcbba02

Merge branch 'dill' of github.com:archibate/taichi into dill

0e685ad

archibate merged commit 37976e7 into taichi-dev:master Jun 24, 2020

yuanming-hu mentioned this pull request Jun 28, 2020

[release] v0.6.14 #1346

Merged

EaryChow mentioned this pull request Jul 13, 2020

IDLE still doesn't work for me #1483

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[IPython] Source inspection dispatcher for better IDLE compatibility #1222

[IPython] Source inspection dispatcher for better IDLE compatibility #1222

archibate commented Jun 11, 2020 •

edited

Loading

archibate commented Jun 11, 2020 •

edited

Loading

archibate commented Jun 12, 2020

Eydcao commented Jun 12, 2020

archibate commented Jun 12, 2020

archibate commented Jun 12, 2020

archibate commented Jun 12, 2020

This comment has been minimized.

archibate commented Jun 23, 2020

yuanming-hu commented Jun 23, 2020

archibate commented Jun 23, 2020

archibate commented Jun 23, 2020

archibate commented Jun 23, 2020

yuanming-hu left a comment

archibate commented Jun 23, 2020

yuanming-hu left a comment

tyoc213 commented Jun 29, 2020 •

edited

Loading

[IPython] Source inspection dispatcher for better IDLE compatibility #1222

[IPython] Source inspection dispatcher for better IDLE compatibility #1222

Conversation

archibate commented Jun 11, 2020 • edited Loading

archibate commented Jun 11, 2020 • edited Loading

archibate commented Jun 12, 2020

Eydcao commented Jun 12, 2020

archibate commented Jun 12, 2020

archibate commented Jun 12, 2020

archibate commented Jun 12, 2020

This comment has been minimized.

archibate commented Jun 23, 2020

yuanming-hu commented Jun 23, 2020

archibate commented Jun 23, 2020

archibate commented Jun 23, 2020

archibate commented Jun 23, 2020

yuanming-hu left a comment

Choose a reason for hiding this comment

archibate commented Jun 23, 2020

yuanming-hu left a comment

Choose a reason for hiding this comment

tyoc213 commented Jun 29, 2020 • edited Loading

archibate commented Jun 11, 2020 •

edited

Loading

archibate commented Jun 11, 2020 •

edited

Loading

tyoc213 commented Jun 29, 2020 •

edited

Loading