Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement the FlatMap typeclass on Job. This allows Job to be used as a Kleisli/Reader #336

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

gregnwosu
Copy link

this pull request allows us to make Job into cats Reader/Kleisli monads. It opens up the proper monad stack for Job and enables the potential to use Job in Monad Transformer stacks . Some example code will probably help explain this best.

import frameless.cats.jobber._
import frameless.{Job, TypedDataset}
import org.apache.spark.sql.{Dataset, SparkSession}
import cats.data.Kleisli

class Library extends App {
  val spark = SparkSession
    .builder()
    .appName("Spark SQL basic example")
    .config("spark.some.config.option", "some-value")
    .getOrCreate()
  import spark.implicits._
  implicit val sc = spark
  val ds1: Dataset[Int] = Seq(1, 2, 3, 4).toDS()
  val ds2: Dataset[Int] = TypedDataset.create(1 to 200).dataset

  def countAndTakeSumJob(sumFrom: Double): Job[Int] =
    for {
      count <- ds1.count()
      sample: Array[Int] = ds1.take(count)
    } yield sample.fold(sumFrom)(_ + _)

  def addAndConcat(delta: Int): Job[String] =
    for {
      sample <- ds2.map(_ + delta)
      result <- ds2.map((n: Int) => (n + sample).toString())
    } yield result
  val k1: Kleisli[Job, Double, Int] = Kleisli(countAndTakeSumJob _)
  val k2: Kleisli[Job, Int, String] = Kleisli(addAndConcat _)
  // This PR enables...
  //  I want to be able to be able to compose the Kleisli i.e.
  val composed: Kleisli[Job, Double, String] = k2.compose(k1)
  composed.run(5d)
}```

@codecov-io
Copy link

codecov-io commented Oct 2, 2018

Codecov Report

Merging #336 into master will decrease coverage by 0.25%.
The diff coverage is 69.23%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #336      +/-   ##
==========================================
- Coverage   96.34%   96.08%   -0.26%     
==========================================
  Files          57       60       +3     
  Lines        1012     1048      +36     
  Branches       10        9       -1     
==========================================
+ Hits          975     1007      +32     
- Misses         37       41       +4
Impacted Files Coverage Δ
cats/src/main/scala/frameless/cats/implicits.scala 72.72% <69.23%> (-2.28%) ⬇️
...la/frameless/ml/internals/TreesInputsChecker.scala 100% <0%> (ø) ⬆️
...la/frameless/functions/NonAggregateFunctions.scala 100% <0%> (ø) ⬆️
...la/frameless/ml/internals/UnaryInputsChecker.scala 100% <0%> (ø) ⬆️
...in/scala/frameless/ml/clustering/TypedKMeans.scala 100% <0%> (ø)
...a/frameless/ml/internals/VectorInputsChecker.scala 100% <0%> (ø)
...frameless/ml/clustering/TypedBisectingKMeans.scala 100% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 542c0c8...e0b1752. Read the comment docs.

@imarios
Copy link
Contributor

imarios commented Nov 30, 2018

Hey @gregnwosu, do you want to keep this open? I would love to merge the PR if you can make the tests pass (adding a test may help with coverage). Let me know if you have any questions.

@gregnwosu
Copy link
Author

Yes please I will write the tests this weekend

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants