Assigning multiple files' path in minio to app_path

[ArijitSinghEDA]

The following files are in my minio bucket

|- main.py
|- help_1.py
|- help_2.py

I need to call only main.py, but main.py calls help_1.py and help_2.py files internally. How do I send all these files in the submit_app function's app_path parameter?

[hussein-awala]

The easiest way to use custom modules is to add them to the docker image and configure the Python path to use them (for ex: #73).

If your files need to be in an S3 bucket and for some reason you need to add them to the job at runtime (it is recommended to avoid doing this if possible), then you should store your modules in a zip file and use the argument --py-files in the spark-submit command which is currently not supported by the spark-on-k8s package, but there are two alternatives ways:

• providing the zip path through the config spark.submit.pyFiles
• updating your job script by adding pyspark.SparkContext.addPyFile() and providing the files as arguments

You can check PySpark documentation for more information.