EtlFlow

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs, scheduling those jobs and monitoring those jobs using UI.

Documentation

Library Documentation https://tharwaninitin.github.io/etlflow/site/docs

Examples

You can use this library in different ways mentioned below.

• Core Module:
  Using this module you can use features of Step API into your project.
• Spark Module (spark steps):
  Using this addon module along with core you can use Apache Spark steps into your project.
• GCP Module (spark steps):
  Using this addon module along with core you can use GCP steps into your project.
• Server Module:
  Using this module along with job you can use additional features of Server API into your project.

Modules Dependency Graph

Scala Version Compatibility Matrix

Main Modules

Module Name	Scala 2.12	Scala 2.13	Scala 3.1
Core	✅	✅	✅
Server	✅	✅	❌

Server Internal Modules

Module Name	Scala 2.12	Scala 2.13	Scala 3.1
Json	✅	✅	✅

Addon Modules For Core

Module Name	Scala 2.12	Scala 2.13	Scala 3.1
Db	✅	✅	✅
Http	✅	✅	✅
Cloud	✅	✅	✅
Email	✅	✅	✅
Aws	✅	✅	✅
Gcp	✅	✅	✅
Redis	✅	✅	❌
Spark	✅	✅	❌

Requirements and Installation

This project is compiled with scala versions 2.12.15, 2.13.7, 3.1.0 Available via maven central. Add the latest release as a dependency to your project

SBT

libraryDependencies += "com.github.tharwaninitin" %% "etlflow-core" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-server" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-spark" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-cloud" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-http" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-redis" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-aws" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-gcp" % "x.x.x"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-email" % "x.x.x"

Maven

<dependency>
    <groupId>com.github.tharwaninitin</groupId>
    <artifactId>etlflow-core_2.12</artifactId>
    <version>x.x.x</version>
</dependency>

Contributions

Please feel free to add issues to report any bugs or to propose new features.