Merge pull request #1426 from OceanParcels/v3.0

Towards parcels v3.0

.binder/environment.yml

@@ -1,13 +1,7 @@
-name: py3_parcels
+name: parcels_binder
 channels:
   - conda-forge
   - defaults
 dependencies:
   - python>=3.8
   - parcels
-  - ffmpeg>=3.2.3
-  - mpi4py>=3.0.1
-  - mpich>=3.2.1
-  - py>=1.4.27
-  - scikit-learn
-  - pykdtree

.github/actions/install-parcels/action.yml

@@ -29,14 +29,3 @@ runs:
     - name: Install parcels
       run: pip install .
       shell: bash -el {0}
-    - name: Set env variables for macos
-      run: |
-        if [[ "${{ runner.os }}" == macOS ]]; then
-          echo "Setting CONDA_BUILD_SYSROOT and C_INCLUDE_PATH for macos"
-          echo "CONDA_BUILD_SYSROOT=/" >> $GITHUB_ENV
-          echo "C_INCLUDE_PATH=$C_INCLUDE_PATH:/Applications/Xcode.app/Contents//Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/" >> $GITHUB_ENV
-          echo "CC=gcc" >> $GITHUB_ENV
-        else
-          echo "Platform not macos."
-        fi
-      shell: bash

.github/workflows/integration-tests.yml

@@ -36,7 +36,7 @@ jobs:
       - name: Setup Conda and parcels
         uses: ./.github/actions/install-parcels
         with:
-          environment-file: environment_py3_${{ matrix.os-short }}.yml
+          environment-file: environment.yml
           environment-name: py3_parcels
       - name: Integration test
         run: |

.github/workflows/unit-tests.yml

@@ -22,7 +22,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [macos, ubuntu, windows]
-        include:
+        include:  # TODO check if these can go?
           - os: macos
             os-short: osx
           - os: ubuntu
@@ -35,7 +35,7 @@ jobs:
       - name: Setup Conda and parcels
         uses: ./.github/actions/install-parcels
         with:
-          environment-file: environment_py3_${{ matrix.os-short }}.yml
+          environment-file: environment.yml
           environment-name: py3_parcels
       - name: Unit test
         run: |

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
 -   repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.4.0
+    rev: v4.5.0
     hooks:
     -   id: trailing-whitespace
     -   id: end-of-file-fixer
@@ -10,7 +10,7 @@ repos:
         types: [text]
         files: \.(json|ipynb)$
 -   repo: https://github.com/pycqa/flake8
-    rev: '6.0.0'
+    rev: '6.1.0'
     hooks:
     -   id: flake8
 -   repo: https://github.com/pycqa/pydocstyle

.readthedocs.yaml

@@ -2,7 +2,7 @@ version: 2
 build:
   os: ubuntu-22.04
   tools:
-    python: mambaforge-4.10
+    python: mambaforge-22.9
   jobs:
     pre_build:
       - sphinx-build -b linkcheck docs/ _build/linkcheck
@@ -11,4 +11,4 @@ sphinx:
   configuration: docs/conf.py
 
 conda:
-  environment: environment_py3_linux.yml
+  environment: docs/environment_docs.yml

docs/conf.py

@@ -71,6 +71,7 @@
     r"https://aip\.scitation\.org/doi/10\.1063/1\.4982720",  # Site doesn't allow crawling
     r"https://www\.sciencedirect\.com/.*",  # Site doesn't allow crawling
     r"https://lxml\.de/",  # Crawler occasionally fails to establish connection
+    r"https://linux\.die\.net/",  # Site doesn't allow crawling
 
     # To monitor
     r"http://marine.copernicus.eu/",  # 2023-06-07 Site non-responsive
@@ -361,7 +362,8 @@ def linkcode_resolve(domain, info):
     'examples/tutorial_interaction': '_static/pulled_particles_twoatractors_line.gif',
     'examples/documentation_LargeRunsOutput': '_static/harddrive.png',
     'examples/tutorial_unitconverters': '_static/globe-icon.jpg',
-    'examples/documentation_geospatial': '_images/tutorial_geospatial_google_earth.png'
+    'examples/documentation_geospatial': '_images/tutorial_geospatial_google_earth.png',
+    'examples/tutorial_kernelloop': '_static/loop-icon.jpeg',
 }
 # -- Options for LaTeX output ---------------------------------------------
 

docs/documentation.rst

@@ -21,7 +21,6 @@ Parcels has several documentation and tutorial Jupyter notebooks which go throug
    examples/documentation_indexing.ipynb
    examples/tutorial_nemo_curvilinear.ipynb
    examples/tutorial_nemo_3D.ipynb
-   examples/tutorial_SummedFields.ipynb
    examples/tutorial_NestedFields.ipynb
    examples/tutorial_timevaryingdepthdimensions.ipynb
    examples/tutorial_periodic_boundaries.ipynb
@@ -47,6 +46,7 @@ Parcels has several documentation and tutorial Jupyter notebooks which go throug
    examples/tutorial_particle_field_interaction.ipynb
    examples/tutorial_interaction.ipynb
    examples/tutorial_analyticaladvection.ipynb
+   examples/tutorial_kernelloop.ipynb
 
 
 .. nbgallery::

environment_py3_linux.yml → docs/environment_docs.yml

@@ -6,9 +6,6 @@ dependencies:
   - cgen
   - ffmpeg>=3.2.3
   - git
-  - gcc_linux-64
-  - mpi4py>=3.0.1
-  - mpich>=3.2.1
   - jupyter
   - matplotlib-base>=2.0.2
   - netcdf4>=1.1.9

docs/examples/documentation_MPI.ipynb

@@ -5,7 +5,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# Parallelisation\n"
+    "# Parallelisation with MPI and Field chunking with dask\n"
    ]
   },
   {
@@ -148,25 +148,15 @@
    "source": [
     "Note that if you want, you can save this new DataSet with the `.to_zarr()` or `.to_netcdf()` methods.\n",
     "\n",
-    "When using `.to_zarr()`, then it further analyses may be sped up by first rechunking the DataSet, by using `ds.chunk()`. Note that in some cases, you will first need to remove the chunks encoding information manually, using a code like below\n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "for v in ds.variables:\n",
-    "    del ds[v].encoding[\"chunks\"]"
+    "When using `.to_zarr()`, then further analysis may be sped up by first rechunking the DataSet, by using `ds.chunk()`. Note that in some cases, you will first need to remove the chunks encoding information manually, using a code like below\n"
    ]
   },
   {
    "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "For small projects, the above instructions are sufficient. If your project is large, then it is helpful to combine the `proc*` directories into a single zarr dataset and to optimise the chunking for your analysis. What is \"large\"? If you find yourself running out of memory while doing your analyses, saving the results, or sorting the dataset, or if reading the data is taking longer than you can tolerate, your problem is \"large.\" Another rule of thumb is if the size of your output directory is 1/3 or more of the memory of your machine, your problem is large. Chunking and combining the `proc*` data in order to speed up analysis is discussed [in the documentation on runs with large output](https://docs.oceanparcels.org/en/latest/examples/documentation_LargeRunsOutput.html).\n"
+    "For small projects, the above instructions are sufficient. If your project is large, then it is helpful to combine the `proc*` directories into a single zarr dataset and to optimise the chunking for your analysis. What is \"large\"? If you find yourself running out of memory while doing your analysis, saving the results, or sorting the dataset, or if reading the data is taking longer than you can tolerate, your problem is \"large.\" Another rule of thumb is if the size of your output directory is 1/3 or more of the memory of your machine, your problem is large. Chunking and combining the `proc*` data in order to speed up analysis is discussed [in the documentation on runs with large output](https://docs.oceanparcels.org/en/latest/examples/documentation_LargeRunsOutput.html).\n"
    ]
   },
   {