[RELEASE] Merge pull request #94 from rapidsai/develop

gQuant 0.5 release
NVIDIA · Jul 10, 2020 · c22c12f · c22c12f
2 parents 63616ba + 5d53970
commit c22c12f
Show file tree

Hide file tree

Showing 48 changed files with 1,682 additions and 767 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -0,0 +1,148 @@
+# Changelog
+
+## [0.5](https://github.com/rapidsai/gQuant/tree/0.5) (2020-07-10)
+
+[Full Changelog](https://github.com/rapidsai/gQuant/compare/0.4.1...0.5)
+
+**Implemented enhancements:**
+
+- \[FEA\] csvStockLoader.py and stockNameLoader.py - Use cudf.read\_csv\(\) insteand of pandas.read\_csv\(\) [\#24](https://github.com/rapidsai/gQuant/issues/24)
+
+**Fixed bugs:**
+
+- \[BUG\] Using a UDF via Series.rolling.apply\(\) results in KeyError in numba [\#88](https://github.com/rapidsai/gQuant/issues/88)
+- \[BUG\] download\_data.sh seems to do not be in containers anymore [\#66](https://github.com/rapidsai/gQuant/issues/66)
+
+**Closed issues:**
+
+- \[FEA\] Conda resolves too slow in the latest versions of the container [\#67](https://github.com/rapidsai/gQuant/issues/67)
+- \[FEA\] Comprehensive refactoring of indicator\_demo.ipynb notebook [\#46](https://github.com/rapidsai/gQuant/issues/46)
+- \[FEA\] Rename viz\_graph\(\) to viz\(\), save\_taskgraph\(\) to save\(\) [\#34](https://github.com/rapidsai/gQuant/issues/34)
+
+**Merged pull requests:**
+
+- \[REVIEW\] Fix mortgage e2e example for rapids 0.14. [\#93](https://github.com/rapidsai/gQuant/pull/93) ([avolkov1](https://github.com/avolkov1))
+- \[REVIEW\] Update RAPIDS to version 0.14 [\#92](https://github.com/rapidsai/gQuant/pull/92) ([yidong72](https://github.com/yidong72))
+- \[REVIEW\]Multiple gpu xgboost - Dask performance fix [\#91](https://github.com/rapidsai/gQuant/pull/91) ([yidong72](https://github.com/yidong72))
+- \[REVIEW\]Mutliple GPU xgboost [\#90](https://github.com/rapidsai/gQuant/pull/90) ([yidong72](https://github.com/yidong72))
+
+## [0.4.1](https://github.com/rapidsai/gQuant/tree/0.4.1) (2020-05-26)
+
+[Full Changelog](https://github.com/rapidsai/gQuant/compare/0.4...0.4.1)
+
+**Merged pull requests:**
+
+- \[REVIEW\] hot fix for 0.4 release [\#86](https://github.com/rapidsai/gQuant/pull/86) ([yidong72](https://github.com/yidong72))
+- \[REVIEW\] fix the cuIndicator notebook and RSI perf notebook [\#85](https://github.com/rapidsai/gQuant/pull/85) ([yidong72](https://github.com/yidong72))
+- Add cuda102 docker support and update version against development branch [\#84](https://github.com/rapidsai/gQuant/pull/84) ([jbaron](https://github.com/jbaron))
+
+## [0.4](https://github.com/rapidsai/gQuant/tree/0.4) (2020-05-19)
+
+[Full Changelog](https://github.com/rapidsai/gQuant/compare/v0.2...0.4)
+
+**Implemented enhancements:**
+
+- \[REVIEW\]Feature adding fractional differencing computation [\#56](https://github.com/rapidsai/gQuant/pull/56) ([yidong72](https://github.com/yidong72))
+
+**Fixed bugs:**
+
+- \[BUG\] Dask computation fails with 0.8 build script [\#28](https://github.com/rapidsai/gQuant/issues/28)
+
+**Closed issues:**
+
+- \[FEA\] Add cuda 10.1.2 support [\#64](https://github.com/rapidsai/gQuant/issues/64)
+- \[FEA\] Use RAPIDS 0.9 container in build.sh [\#54](https://github.com/rapidsai/gQuant/issues/54)
+- \[FEA\] Rename notebook to notebooks [\#50](https://github.com/rapidsai/gQuant/issues/50)
+- \[FEA\] Add Jupyterlab extension to display GPU usage [\#49](https://github.com/rapidsai/gQuant/issues/49)
+- \[FEA\] Merge develop branch to master [\#47](https://github.com/rapidsai/gQuant/issues/47)
+- \[FEA\] implement the fractional difference operation [\#42](https://github.com/rapidsai/gQuant/issues/42)
+
+**Merged pull requests:**
+
+- \[REVIEW\] merge develop to master and release it as 0.4 [\#82](https://github.com/rapidsai/gQuant/pull/82) ([yidong72](https://github.com/yidong72))
+- \[REVIEW\]update to latest version of RAPIDS 0.13 [\#81](https://github.com/rapidsai/gQuant/pull/81) ([yidong72](https://github.com/yidong72))
+- fixed the gamma computation error [\#79](https://github.com/rapidsai/gQuant/pull/79) ([doyend](https://github.com/doyend))
+- \[REVIEW\]asian barrier option  tutorial [\#77](https://github.com/rapidsai/gQuant/pull/77) ([yidong72](https://github.com/yidong72))
+- \[REVIEW\] upgrade to RAPIDS 0.11 [\#76](https://github.com/rapidsai/gQuant/pull/76) ([yidong72](https://github.com/yidong72))
+- \[skip ci\] Merge CI Scripts [\#75](https://github.com/rapidsai/gQuant/pull/75) ([avolkov1](https://github.com/avolkov1))
+- \[REVIEW\] Add CI scripts and conda recipe [\#74](https://github.com/rapidsai/gQuant/pull/74) ([raydouglass](https://github.com/raydouglass))
+- \[WIP\] CUQ-36: fix typechecking nodes multi input dataframes [\#68](https://github.com/rapidsai/gQuant/pull/68) ([avolkov1](https://github.com/avolkov1))
+- \[REVIEW\] Upgrade to RAPIDS 0.10 [\#63](https://github.com/rapidsai/gQuant/pull/63) ([yidong72](https://github.com/yidong72))
+- \[REVIEW\] stable master merge [\#62](https://github.com/rapidsai/gQuant/pull/62) ([yidong72](https://github.com/yidong72))
+- \[REVIEW\]upgrade to RAPIDS 0.9, FIX the rebase problem [\#61](https://github.com/rapidsai/gQuant/pull/61) ([yidong72](https://github.com/yidong72))
+- Revert "\[REVIEW\]upgrade to RAPIDS 0.9" [\#59](https://github.com/rapidsai/gQuant/pull/59) ([yidong72](https://github.com/yidong72))
+- Revert "\[REVIEW\]upgrade to RAPIDS 0.9" [\#58](https://github.com/rapidsai/gQuant/pull/58) ([avolkov1](https://github.com/avolkov1))
+- \[REVIEW\]upgrade to RAPIDS 0.9 [\#57](https://github.com/rapidsai/gQuant/pull/57) ([yidong72](https://github.com/yidong72))
+- \[REVIEW\] change the text for notebook 05 [\#55](https://github.com/rapidsai/gQuant/pull/55) ([yidong72](https://github.com/yidong72))
+- Fix \#50b - Rename notebook folder to notebooks [\#52](https://github.com/rapidsai/gQuant/pull/52) ([miguelusque](https://github.com/miguelusque))
+- Fix \#50 - Rename notebook folder to notebooks [\#51](https://github.com/rapidsai/gQuant/pull/51) ([miguelusque](https://github.com/miguelusque))
+
+## [v0.2](https://github.com/rapidsai/gQuant/tree/v0.2) (2019-08-16)
+
+[Full Changelog](https://github.com/rapidsai/gQuant/compare/v0.1...v0.2)
+
+**Implemented enhancements:**
+
+- \[FEA\] Refactor 04\_portfolio\_trade.ipynb notebook [\#39](https://github.com/rapidsai/gQuant/issues/39)
+- \[FEA\] Refactor notebook 01\_tutorial.ipynb [\#35](https://github.com/rapidsai/gQuant/issues/35)
+- \[FEA\] Add error message \(or warning\) if replace node does not exist [\#32](https://github.com/rapidsai/gQuant/issues/32)
+- \[FEA\] Add new issue templates [\#26](https://github.com/rapidsai/gQuant/issues/26)
+- \[FEA\] cuIndicator notebook plot widget is too complicated [\#17](https://github.com/rapidsai/gQuant/issues/17)
+
+**Fixed bugs:**
+
+- \[BUG\] Remove debug info from barPlotNode.py and cumReturnNode.py [\#40](https://github.com/rapidsai/gQuant/issues/40)
+- \[BUG\] 04\_portfolio\_trade.ipynb - Number of filtered stocks differs from text [\#23](https://github.com/rapidsai/gQuant/issues/23)
+
+**Merged pull requests:**
+
+- Fix \#17 - cuIndicator notebook plot widget is too complicated \(WIP\) [\#45](https://github.com/rapidsai/gQuant/pull/45) ([miguelusque](https://github.com/miguelusque))
+- Fix \#39 - Refactor 04\_portfolio\_trade.ipynb notebook [\#44](https://github.com/rapidsai/gQuant/pull/44) ([miguelusque](https://github.com/miguelusque))
+- Merge develop to master [\#43](https://github.com/rapidsai/gQuant/pull/43) ([yidong72](https://github.com/yidong72))
+- Fix \#40 - Remove debug info [\#41](https://github.com/rapidsai/gQuant/pull/41) ([miguelusque](https://github.com/miguelusque))
+- Update mortgage example using TaskGraph API. [\#38](https://github.com/rapidsai/gQuant/pull/38) ([avolkov1](https://github.com/avolkov1))
+- fixed the issue 32 [\#37](https://github.com/rapidsai/gQuant/pull/37) ([yidong72](https://github.com/yidong72))
+- Fix \#35 - Refactor 01\_tutorial.ipynb notebook [\#36](https://github.com/rapidsai/gQuant/pull/36) ([miguelusque](https://github.com/miguelusque))
+- Fix \#26b - Add new issue templates [\#30](https://github.com/rapidsai/gQuant/pull/30) ([miguelusque](https://github.com/miguelusque))
+- Revert "fix \#26 - Add new issues template" [\#29](https://github.com/rapidsai/gQuant/pull/29) ([yidong72](https://github.com/yidong72))
+- Fix \#26 - Add new issues template [\#27](https://github.com/rapidsai/gQuant/pull/27) ([miguelusque](https://github.com/miguelusque))
+- added workflow class [\#22](https://github.com/rapidsai/gQuant/pull/22) ([yidong72](https://github.com/yidong72))
+- Fix \#19b - Combine OS/Cuda versions user input [\#21](https://github.com/rapidsai/gQuant/pull/21) ([miguelusque](https://github.com/miguelusque))
+- Fix \#19 - build.sh - Move pip dependencies to conda dependencies [\#20](https://github.com/rapidsai/gQuant/pull/20) ([miguelusque](https://github.com/miguelusque))
+- Fix \#13, \#14, \#16 in cuIndicator.ipynb notebook [\#18](https://github.com/rapidsai/gQuant/pull/18) ([miguelusque](https://github.com/miguelusque))
+- update the build.sh [\#15](https://github.com/rapidsai/gQuant/pull/15) ([yidong72](https://github.com/yidong72))
+- Feature xgb notebook [\#11](https://github.com/rapidsai/gQuant/pull/11) ([yidong72](https://github.com/yidong72))
+- CUQ-5: Mortgage example using gQuant. [\#10](https://github.com/rapidsai/gQuant/pull/10) ([avolkov1](https://github.com/avolkov1))
+- CUQ-5: Mortgage example using  gQuant. [\#9](https://github.com/rapidsai/gQuant/pull/9) ([avolkov1](https://github.com/avolkov1))
+- Feature indicator node [\#8](https://github.com/rapidsai/gQuant/pull/8) ([yidong72](https://github.com/yidong72))
+- Feature mulit assets indicator [\#7](https://github.com/rapidsai/gQuant/pull/7) ([yidong72](https://github.com/yidong72))
+- Update build.sh [\#6](https://github.com/rapidsai/gQuant/pull/6) ([phogan-nvidia](https://github.com/phogan-nvidia))
+- Feature environment [\#5](https://github.com/rapidsai/gQuant/pull/5) ([yidong72](https://github.com/yidong72))
+
+## [v0.1](https://github.com/rapidsai/gQuant/tree/v0.1) (2019-08-13)
+
+[Full Changelog](https://github.com/rapidsai/gQuant/compare/e4a967fc9e3289fdbfa37e7a7b84887579332b42...v0.1)
+
+**Implemented enhancements:**
+
+- \[FEA\] build.sh - Move pip dependencies to conda dependencies [\#19](https://github.com/rapidsai/gQuant/issues/19)
+
+**Fixed bugs:**
+
+- \[BUG\] Update build.sh to 0.7 until issue \#28 is fixed [\#31](https://github.com/rapidsai/gQuant/issues/31)
+- \[BUG\] cuIndicator.ipyng - Wrong series names [\#16](https://github.com/rapidsai/gQuant/issues/16)
+- \[BUG\] cuIndicator.ipynb - Runtime error in cell \#3 - Missing file [\#14](https://github.com/rapidsai/gQuant/issues/14)
+- \[BUG\] cuIndicator.ipynb - Incorrect path to dataset [\#13](https://github.com/rapidsai/gQuant/issues/13)
+
+**Merged pull requests:**
+
+- Revert "gQuant34 - Update build.sh to make use of RAPIDS v0.8 container" [\#33](https://github.com/rapidsai/gQuant/pull/33) ([yidong72](https://github.com/yidong72))
+- gQuant34 - Update build.sh to make use of RAPIDS v0.8 container [\#12](https://github.com/rapidsai/gQuant/pull/12) ([miguelusque](https://github.com/miguelusque))
+- Synch master with develop [\#4](https://github.com/rapidsai/gQuant/pull/4) ([avolkov1](https://github.com/avolkov1))
+- added unit tests for the cuindicator [\#3](https://github.com/rapidsai/gQuant/pull/3) ([yidong72](https://github.com/yidong72))
+- CUQ-21: Improving tutorials for gQuant [\#2](https://github.com/rapidsai/gQuant/pull/2) ([avolkov1](https://github.com/avolkov1))
+- Add download script and instructions in the readme [\#1](https://github.com/rapidsai/gQuant/pull/1) ([yidong72](https://github.com/yidong72))
+
+
+
+\* *This Changelog was automatically generated by [github_changelog_generator](https://github.com/github-changelog-generator/github-changelog-generator)*
diff --git a/docker/build.sh b/docker/build.sh
@@ -33,7 +33,7 @@ echo -e "\nPlease, select your cuda version:\n" \
 
 read -p "Enter your option and hit return [1]-4: " CUDA_VERSION
 
-RAPIDS_VERSION="0.13"
+RAPIDS_VERSION="0.14"
 
 CUDA_VERSION=${CUDA_VERSION:-1}
 case $CUDA_VERSION in

diff --git a/gquant/dataframe_flow/_node_flow.py b/gquant/dataframe_flow/_node_flow.py
@@ -124,6 +124,41 @@ def __translate_column(self, columns):
 
         return output
 
+    def __delayed_call_noports(self, inputs):
+
+        def get_pout(df_out):
+            '''Used for delayed unpacking.'''
+            if isinstance(df_out, cudf.DataFrame):
+                # Needed for the same reason as __make_copy. To prevent columns
+                # addition in the input data frames. In python everything is
+                # by reference value and dataframes are mutable.
+                # Handle the case when dask_cudf.DataFrames are source frames
+                # which appear as cudf.DataFrame in a dask-delayed function.
+                return df_out.copy(deep=False)
+
+            return df_out
+
+        # handle the dask dataframe automatically
+        # use the to_delayed interface
+        # TODO, currently only handles first input is dask_cudf df
+        i_df = inputs[0]
+        rest = inputs[1:]
+        if isinstance(i_df, dask_cudf.DataFrame):
+            output_df_dly_list = []
+            for input_dly in i_df.to_delayed():
+                inputs_ = [input_dly] + rest
+                output_df_dly = dask.delayed(self.decorate_process())(inputs_)
+                output_df_dly_per = output_df_dly.persist()
+                df_out = dask.delayed(get_pout)(output_df_dly_per)
+                output_df_dly_list.append(df_out.persist())
+
+            output_df = dask_cudf.from_delayed(output_df_dly_list)
+
+        else:
+            output_df = self.decorate_process()(inputs)
+
+        return output_df
+
     def columns_flow(self):
         """
         Flow the graph to determine the input output dataframe column names and
@@ -406,7 +441,7 @@ def __valide(self, node_output, ref_cols):
                 cudf_types_tuple = (cudf.DataFrame, dask_cudf.DataFrame)
 
                 if out_type in cudf_types_tuple:
-                    if len(out_val) == 0 and out_optional:
+                    if len(out_val.columns) == 0 and out_optional:
                         continue
 
                 if out_type in cudf_types_tuple:
@@ -439,6 +474,10 @@ def __input_columns_ready(self):
             if iport not in self.input_columns:
                 return False
 
+        if (self._using_ports() and len(self._get_input_ports(
+        )) != 0 and len(self.inputs) == 0):
+            return False
+
         return True
 
     def __get_input_df(self):
@@ -673,9 +712,9 @@ def get_pout(out_dict, port):
         return output_df
 
     def decorate_process(self):
-        import time
 
         def timer(*argv):
+            import time
             start = time.time()
             result = self.process(*argv)
             end = time.time()
@@ -711,18 +750,7 @@ def __call__(self, inputs_data):
                     else:
                         output_df = self.decorate_process()(inputs)
                 else:
-                    # handle the dask dataframe automatically
-                    # use the to_delayed interface
-                    # TODO, currently only handles first input is dask_cudf df
-                    i_df = inputs[0]
-                    rest = inputs[1:]
-                    if isinstance(i_df, dask_cudf.DataFrame):
-                        d_fun = dask.delayed(self.decorate_process())
-                        output_df = dask_cudf.from_delayed([
-                            d_fun([item] + rest)
-                            for item in i_df.to_delayed()])
-                    else:
-                        output_df = self.decorate_process()(inputs)
+                    output_df = self.__delayed_call_noports(inputs)
 
         if self.uid != OUTPUT_ID and output_df is None:
             raise Exception("None output")

diff --git a/gquant/dataframe_flow/task.py b/gquant/dataframe_flow/task.py
@@ -3,6 +3,7 @@
 import copy
 from .taskSpecSchema import TaskSpecSchema
 from ._node import _Node
+from pathlib import Path
 
 
 __all__ = ['Task']
@@ -62,6 +63,9 @@ def get_node_obj(self, replace=None, profile=False, tgraph_mixin=False):
             if modulepath is not None:
                 spec = importlib.util.spec_from_file_location(node_id,
                                                               modulepath)
+                modulename = Path(modulepath).stem
+                spec = importlib.util.spec_from_file_location(
+                    modulename, modulepath)
                 mod = importlib.util.module_from_spec(spec)
                 spec.loader.exec_module(mod)
                 NodeClass = getattr(mod, node_type)

diff --git a/gquant/plugin_nodes/analysis/barPlotNode.py b/gquant/plugin_nodes/analysis/barPlotNode.py
@@ -1,5 +1,8 @@
 from gquant.dataframe_flow import Node
 from bqplot import Axis, LinearScale, DateScale, Figure, OHLC, Bars, Tooltip
+import cupy as cp
+import cudf
+import dask_cudf
 
 
 class BarPlotNode(Node):
@@ -39,16 +42,33 @@ def process(self, inputs):
         ax_y = Axis(label='Price', scale=sc, orientation='vertical',
                     tick_format='0.0f')
         # Construct the marks
-        ohlc = OHLC(x=stock['datetime'][::stride],
-                    y=stock[['open', 'high', 'low', 'close']]
-                    .as_gpu_matrix()[::stride, :],
-                    marker='candle', scales={'x': dt_scale, 'y': sc},
-                    format='ohlc', stroke='blue',
-                    display_legend=True, labels=[label])
-        bar = Bars(x=stock['datetime'][::stride],
-                   y=stock['volume'][::stride],
-                   scales={'x': dt_scale, 'y': sc2},
-                   padding=0.2)
+        if (isinstance(stock,
+                       cudf.DataFrame) or isinstance(stock,
+                                                     dask_cudf.DataFrame)):
+            ohlc = OHLC(x=stock['datetime'][::stride].to_array(),
+                        y=cp.asnumpy(stock[['open',
+                                            'high',
+                                            'low',
+                                            'close']].values[::stride, :]),
+                        marker='candle', scales={'x': dt_scale, 'y': sc},
+                        format='ohlc', stroke='blue',
+                        display_legend=True, labels=[label])
+            bar = Bars(x=stock['datetime'][::stride].to_array(),
+                       y=stock['volume'][::stride].to_array(),
+                       scales={'x': dt_scale, 'y': sc2},
+                       padding=0.2)
+        else:
+            ohlc = OHLC(x=stock['datetime'][::stride],
+                        y=stock[['open',
+                                 'high',
+                                 'low', 'close']].values[::stride, :],
+                        marker='candle', scales={'x': dt_scale, 'y': sc},
+                        format='ohlc', stroke='blue',
+                        display_legend=True, labels=[label])
+            bar = Bars(x=stock['datetime'][::stride],
+                       y=stock['volume'][::stride],
+                       scales={'x': dt_scale, 'y': sc2},
+                       padding=0.2)
         def_tt = Tooltip(fields=['x', 'y'], formats=['%Y-%m-%d', '.2f'])
         bar.tooltip = def_tt
         bar.interactions = {