Baseline IRs for translating ADF pipelines to workflows

[ghanse]

Summary

This PR adds intermediate representations of ADF pipelines, activities, and linked services from the APIs. This is a required baseline for translating ADF pipelines to Databricks workflows.

Details

• Added the orchestrators.adf package
• Created PipelineNode for any nodes coming from ADF
• Created classes to represent the ADF API objects (e.g. Pipelines, Activities, and Linked Services)

Notes

• This is a larger PR than I would like, but minimally implements the IRs needed to translate 1 pipeline
• Can you validate my pattern for implementing the children method? (e.g. in DatabricksNotebookActivity)

[vil1]

This needs to be refactored as to avoid Option fields as much as possible.

[vil1]

Case classes like this one where all fields are Seq or Option almost always indicate a design problem. As a rule of thumb, one should ask themselves "what would be the meaning of Activity(Seq(), None, None, None, None, None, None, None, None, Seq())?".

Option in a case class field should only be used when there is a meaningful situation where the field could be absent.

For example, is it possible for an Activity to have no state? Note that isn't exactly the same question as "can an Activity be neither Active nor Inactive?". If the answer to the latter question is "yes", then we should add a case object Unknown extends Activity for example and define the field as state: ActivityState = Unknown.

[ghanse]

I've made the name, activityType, activityProperties, and state fields required. Every activity should have at minimum a name, type, set of type-specific properties, and state.

Many of the other fields are configurations which will be optionally returned by the API client.

[vil1]

Following the comment above, here it is quite clear that we have three mutually exclusive cases mashed together.
We should rather define it as

sealed trait LibraryDefinition
case class MavenLibrary(jarFilePath: String, mavenSpecification: MavenSpecification) extends LibraryDefinition
case class PyPiLibrary(eggFilePath: String, piPySpecification: PyPiSpecification) extends LibraryDefinition
case class CranLibrary(whlFilePath: String, cranSpecification: CranSpecification) extends LibraryDefinition

[ghanse]

This is done.

[vil1]

Suggested change

	abstract class ActivityProperties(name: Option[String]) extends PipelineNode {
	override def children: Seq[PipelineNode] = Seq()
	}
	trait ActivityProperties extends PipelineNode {
	def name: String
	}

[ghanse]

This is done. I will note that there are no common fields across ActivityProperties of different types. Each activity type will have a unique set of properties. Is it OK to use a trait in this way?

[vil1]

Suggested change

	case class DatabricksNotebookActivity(
	name: Option[String],
	baseParameters: Map[String, String],
	libraries: Seq[LibraryDefinition],
	notebookPath: Option[String])
	extends ActivityProperties(name) {
	override def children: Seq[PipelineNode] = super.children ++ libraries
	}
	case class DatabricksNotebookActivity(
	override val name: String,
	baseParameters: Map[String, String],
	libraries: Seq[LibraryDefinition],
	notebookPath: Option[String])
	extends ActivityProperties {
	override def children: Seq[PipelineNode] = libraries
	}

[ghanse]

This is done.

[vil1]

If we are sure that linkedServiceName, policy and userProperties are optional

Suggested change

	override def children: Seq[PipelineNode] = Seq() ++ dependsOn ++ linkedServiceName ++
	policy ++ userProperties
	}
	override def children: Seq[PipelineNode] = dependsOn ++ linkedServiceName.toSeq ++
	policy.toSeq ++ userProperties.toSeq
	}

otherwise, we should go for

override def children: Seq[PipelineNode] = dependsOn ++ Seq(linkedServiceName, policy, userProperties)

[ghanse]

This is done. I made fields required and used the last pattern.