Generic quantities support

[non]

It would be nice if Squants provided a parallel set of generic types for working with other kinds of numbers besides Double, or supported generic types in some other way.

I'm not sure exactly what this would look like, or if this is a realistic goal for the project, but there are definitely types in Spire (Rational, Real, Complex[_], Interval[_], and so on) that would be really useful for dimensional analysis.

[garyKeorkunian]

Erik, I was recently informed of Spires and have been looking at it. I agree it would be nice to let users choose the type for the underlying value. I will play with the idea in a branch and see what I come up with.

Thanks.

[garyKeorkunian]

This work is in progress.

The current development version (0.3.1-SNAPSHOT) offers support for arbitrary numeric types as the argument for all Quantity factory methods. The underlying value and unit multipliers are still Doubles, but this change is a significant step towards providing generic support there as well.

This change will allow user code to begin using arbitrary numeric types now (when creating Quantities) and benefit from the full generics once it is available in the future.

[garyKeorkunian]

Significant progress has been made on this effort. It is a goal for the 0.5 series to have this fully implemented.

There is currently a wip branch that includes this work in progress within the object model replica. This replica can be found in the experimental package in the test code.

Most of the work has been complete and at this time I am playing a bit of whack-a-mole with Scala's type system.

The general strategy is to create a SquantsNumeric trait that can be implemented for any generic type. Implementations for Int, Long, Double and BigDecimal have been created. Implementations for Spire and other types will be included in a contrib project - once all of this working.

The main sticking point is a type conflict between specific UnitOfMeasure implementations and the type required by a Quantity valueUnit.

Quantities are now typed not only on themselves (as in previous versions) but also on a generic value type.

abstract class Quantity[T <: Quantity[T, N], N] ... {
  implicit val num: SquantsNumeric[N]  // provides required operations on N
  def value: N  // previously Double
  def valueUnit: UnitOfMeasure[T]
  ...
}

Since UnitOfMeasure is typed on Quantity, it's signature needed to change to ...

trait UnitOfMeasure[T <: Quantity[T, _]] {
}

The reason for the placeholder in the Value Type position is that it shouldn't matter to the UOM's what generic number type a quantity's underlying value is using. However, this leads to the following type incompatibility:

Quantity.valueUnit must be a UnitOfMeasure[T] where T <: Quantity[T, N]

however

Implementations of UnitOfMeasure are actually UnitOfMeasure[T] where T <: Quantity[T, _]. That is the UOM's are not fixed to specific underlying Quantity value.

In the end Quantity[T, N] != Quantity[T, _], and that is what I need to work through.

I suspect this could be remedied by applying the correct variance, but I haven't found a solution yet.

Any advice or suggests for solving this would be greatly appreciated.

[garyKeorkunian]

Well after a year of dabbling with this off and on I finally have a functioning prototype, which can be found in the squants.experimental package in the test code.

There is still work to do ...

• update the QuantityRange and TimeDerivative / Integral classes to support the change.
• port all of the Quantity Types (there are currently 55 of these).
• create SquantsNumeric Type Classes for Spire types - likely in a companion project.

[cquiroz]

I got inspired by @zainab-ali to add spire/generic support to squants. Would you think the wip branch is a good place to start?

[garyKeorkunian]

@cquiroz I think so. A good amount of work has been done there. The blocker was around the typing for QuantityRanges. Let me know if you have any questions or want to go over it.

[garyKeorkunian]

@cquiroz Actually, the more complete work is in the shared/test/scala/experimental folder of the master branch.

[derekmorr]

I had tried to grab Erik from the Spire project at NEScala to talk about this, but we never connected. He seemed to think there were issues with the current approach, but he didn't elaborate.

[hunterpayne]

I've created something like what this issue requests here...https://github.com/hunterpayne/terra
Its an entirely different project as it required rewriting most of the source to make it work but enjoy...

[garyKeorkunian]

@hunterpayne Nice. That work in the experimental package needed a whole lot of refactoring to get as far as did, too. I'll take a look at this. It's something we are informally targeting for version 2.0. Thanks!!

[hunterpayne]

Thank you Gary. It should be noted that to make it all work I had to resort to using ClassTags in a couple of places which has drawbacks for native and JS builds. So tread carefully on what you want to bring in from Terra. The simplicity of the Squants code has advantages that are lost when you refactor out the numeric types from Quantities (and the types get significantly more complicated). This is a really good case where the cure might be worse than the disease.

[hunterpayne]

Update, I've successfully removed the ClassTag dependency and gotten Scala-JS and Scala-native versions working. Its still a more complex source base to manage but it now has the same platform support.

[garyKeorkunian]

I've created an alternative model that supports generic numerics in quantities.

If can be found here: https://github.com/garyKeorkunian/squants-generic

This one inverts the type stack so that Dimension is at the "root" and UnitOfMeasure and Quantity have a type-dependency on that. This seems to be more semantically correct for the domain.

The README outlines the goals, current state and roadmap.

Please let me know your thoughts. I'd like to vet this a bit before the major refactoring of classes begins.

[hunterpayne]

So this approach you are taking has pluses and minuses.  I'm going to assume you looked at Terra (https://github.com/hunterpayne/terra) and I'm going to compare this approach to the one I used there: Pros:- This approach is much simpler and will be easier to write and debug- Customizing the data types will be simpler assuming users use the same underlying type for all units of measure Cons:- Generics will force users to change a lot more code to convert to squants-generics: i.e. Power becomes Power[Double} instead of changing the imports like Terra does- The code that uses generics will be more brittle if users need to change underlying types frequently but don't use type parameters otherwise if users need to use type parameters, their code becomes more complex - Users will end up mixing and matching data types i.e. Power[Double] and Time[Long] (or Time[Date]) and just converting to Int or Double might not be enough. Just some things to consider.  I ended up creating a minimum set of dimensions to test all the data type interactions.  It was: Dimensionless, Time, TimeSquared, Frequency, Information, DataRate, and Money.  Those covered all the unique cases for unit types I felt.  You probably want to prove your approach on that minimum set of dimensions to test if all the data type interactions you envision will work the way you want. Hunter On Saturday, September 14, 2019, 4:30:34 PM PDT, garyKeorkunian <[email protected]> wrote: I've created an alternative model that supports generic numerics in quantities. If can be found here: https://github.com/garyKeorkunian/squants-generic This one inverts the type stack so that Dimension is at the "root" and UnitOfMeasure and Quantity have a type-dependency on that. This seems to be more semantically correct for the domain. The README outlines the goals, current state and roadmap. Please let me know your thoughts. I'd like to vet this a bit before the major refactoring of classes begins. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[garyKeorkunian]

@hunterpayne Thanks for taking a look and the feedback.

I have looked at terra. I agree terra will be a bit heavier on the maintenance side, but I like many of the things you did there. I need to study it more as it's not quite clear to me how we can extend it with additional types like spire, although I'm sure you have considered it. I do like what you've done, however, I also want to explore further the idea of Quantity being typed on a Dimension instead of the other way around. It seems more semantically correct to me.

I do like that users don't need to change much about their code besides an import. I do think minimizing that impact is important. One part of terra that I like, that seems to get you there, is the [DimensionName]Like naming you use, with specific implementations getting the normal dimensional name. That inspires me to something like this:

final class MassGen[A: SquantsNumeric] // actual class implementation

then provide different imports like

package {
 
  object SquantsDouble {
      type Mass = MassGen[Double]
  }

  object SquantsGeneric {
     type Mass[A] = MassGen[A]
  }
}

Importing SquantsDouble would provide the same typing as exists now. And of course, more can be created that provide specific numerics for each dimension.

SquantsNumeric does provide for interoperability between numerics. It's the fromSquantsNumeric method that provides this and the implementations can choose how best to do that.

As I continue to look at terra, I expect further inspiration and hopefully we can get to something that is both easy to maintain and use. Thanks again!

Gary

[garyKeorkunian]

@hunterpayne Also, I agree I need to refactor a broader set of Dimensions and other features to validate this approach. I just wanted to get some eyes on it to ensure I wasn't doing anything too far off base before I go further.

[hunterpayne]

So there are a few classes in Terra that you might want to look at to really see what is going on inside of there.

1. TerraOps.scala which is the interface that defines how Terra interacts with types and converts between them. This class is admittedly very messy and could use some clean-up but you get the idea.
2. AbstractDoubleTerraOps.scala which glues together all the scopes (*Ops) for each subclass of Dimension and UnitOfMeasure into one big scope
3. StandardTerraOps.scala which is an example of an implementation of the AbstractDoubleTerraOps. Pay special attention to lines 185-206 which use the type aliases to dynamically generate a package hierarchy containing all the types present in Squants.
4. InformationSymbols.scala which is an example of a package mapping which aliases the type parametrized Dimensions and UnitsOfMeasures into nicer types like Squants currently uses (e.g. Mass, Energy instead of MassLike[Tuple] which is the real type).

[hunterpayne]

@garyKeorkunian Your general approach seems sound. Investigate Quantity being typed on a Dimension further, its a good idea. The thing I discovered as I was working on Terra was that the interactions between values coming from different dimensions happens quite often and you need a good solution for that. Also, I learned that the Scala type inference engine really doesn't do well with multiple type parameters which is why there is a TypeContext which holds types which normally might be their own type parameters. Hope this helps.

[garyKeorkunian]

I've update squants-generic to support better backward compatibility with 1.x.

There's some sample code in the README here: https://github.com/garyKeorkunian/squants-generic#current-state

[hunterpayne]

@garyKeorkunian Looks good so far. For extra types to test on maybe consider sigfigs which is a significant digits library for chemistry and engineering.

[TomasPuverle]

Hi, would mind giving an update on this, please?

[garyKeorkunian]

Unfortunately, not much progress since the last comments above.

Contributors are welcome.

[garyKeorkunian]

Hello, all.  

It's been a while, but I think I might have a solution to this.

I created a branch with a POC inside of a new squants2 package.

You can see a write up here: https://github.com/typelevel/squants/tree/generic-value-poc/shared/src/main/scala/squants2

Most of the core is refactored and working as hoped.  I converted several dimensions to validate it.  Of course, there is more to do.

If this approach seems satisfactory, I will continue on ... with as much help as I can get :-)

[hunterpayne]

Hi Gary,   It has been a while.  Here are some comments I hope are helpful from a quick look at your POC.    I would suggest that you should be returning A instead of this.type in Quantity for operator overloads (or does the compiler not like that?). To do this, make A a self-type like here: https://github.com/hunterpayne/terra/blob/master/src/main/scala/org/terra/Quantity.scala#L23 Or does the compiler team now recommend to use this.type instead of self-types now?    Perhaps you need 2 implementations of subtract, divide and modulo, one that returns A and another that returns B because those operations are not commutative.  Not sure how you make the compiler happy with that as you can't have two otherwise identical methods that return different types in Scala.  I struggled with that a lot with Terra.  I ended up requiring like types and forcing the user to cast before the operation to the desired type.  I feel this gives better control to the user over how different types are combined but it does introduce some complexity for the user.   Another issue I ran into is that there are units that need whole numbers instead of floating point numbers (a number of photons or bits for example).  But perhaps this can safely be ignored.  I allowed TL to be defined as anything you want but the default was to make it a Long.  Consider that there are calculations where Double returns the wrong value and Long returns the correct one and vise-versa.   Support for Spire is most important for the electro-magnatic units of measure as imaginary numbers arise in the physics of electricity frequently.  Perhaps it is best to start testing with Spire there.   Finally, why did you create your own Numeric interface?  Was something missing from the scala implementation? Is this about stability as Numeric in Scala might change with different versions of the compiler?  Perhaps consider making QNumeric an implicit class that accepts a Numeric so users can reuse Spire's Numeric implementations.  I ended up making my own ClassTag in Terra for stability so I understand if that is the reason. Hunter PS in your POC, perhaps put a link to Terra so if your solution doesn't work for a specific user, they know about an option that might work for them.  It is extremely difficult to handle all the possible cases for this problem and Terra takes a heavier approach that could be necessary for some uses cases.  You will have to make trade-offs for how you support things like casting and mixing of types. I would suggest that there is no completely "right" solution for these decisions possible in Scala (or any other language).  For example, providing your own types for Terra is challenging and that's the trade-off I choose but that doesn't mean it is the right one for you. On Friday, May 20, 2022, 10:40:44 AM PDT, garyKeorkunian ***@***.***> wrote: Hello, all.   It's been a while, but I think I might have a solution to this. I created a branch with a POC inside of a new squants2 package. You can see a write up here:  https://github.com/typelevel/squants/tree/generic-value-poc/shared/src/main/scala/squants Most of the core is refactored and working as hoped.  I converted several dimensions to validate it.  Of course, there is more to do. If this approach seems satisfactory, I will continue on ... with as much help as I can get :-) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[garyKeorkunian]

Hi @hunterpayne!

Thanks for the quick feedback!  I'll take a look through it this weekend.

[garyKeorkunian]

@hunterpayne 

Thanks for the comments.

1.  Self typing is used in the current model.  I found it difficult to get the numeric working the way I wanted, but I will revisit it.  Returning this.type seems to work OK, but it is ugly, and also requires some type-casting.  This is mostly because the unit.apply method is returning Quantity[A, Mass.type] instead of Mass[A], which is really just a sub-type of the former.
2. I think for the non-commutative operations, it's OK to return the result using the numeric type of the LHS.  The user can be explicit if necessary.  Is there something I am missing as to why we would want both variants?
3. User code can mix types, so some quantities can be Long and others Double.  These should use appropriate conversions when computed, but some "yet to be written" tests will confirm that.
4. The reason for creating the new QNumeric was primarily to support mixed-type operations.  Numeric has binary operations that require type A on both sides.  I wanted to get around that.  That said, I did follow your advice and created an implicit conversion that creates a QNumeric from any Numeric, which allowed me to eliminate about half that code.  That should allow the use of Spire (and others) out of the box.  Some of the default methods, however, are a bit hacky.  Like the trig and rounding functions do a conversion to and from Double to make use of the math lib.  That could be overwritten for specific Numeric's where necessary.  There's now an example of QBigDecimal that uses a more specific rounded function.  More improvements to come.

I agree, there's never one single solution that works for everyone.  I am happy to share that link.

[garyKeorkunian]

I just pushed an update that eliminates the QNumeric and uses the standard Numeric throughout. Much simpler and more flexible.

I had to supply some default implementations for the rounding stuff, but an alternative could be supplied to the map function if the user code requires something different. The trig functions are only limited places (Angle and SVector) so they return Double (for now). Same thing there, user code could provide alternatives.

@hunterpayne ... and that link is up on the README.

[hunterpayne]

For non-commutative operations, I was worried about usages like Long / Double => Long which is likely not what you want. Forcing the numerator to be a Double would probably be what a user would want there. Not sure which side the compiler would force an upcast for. Most CPUs don't allow mixed type math operations so the compiler would just force casting somewhere. Making that casting explicit is probably for the best as it makes debugging much easier for you.

The trig functions probably can't be well defined for things that are not floating point numbers so that's probably good. I am not sure if sin(i) is defined according to a mathematician.

Glad the implicit QNumeric conversion worked for you. Now that I think about it an implicit conversion there is probably better than an implicit class.

Thanks for the link. And nice progress overall.

[AtelierSnek]

The trig functions probably can't be well defined for things that are not floating point numbers so that's probably good. I am not sure if sin(i) is defined according to a mathematician.

The trig operations are all defined for complex numbers, yes, however there are rules that are not the same as real trig.
sin(i) == i * sinh(1) for example

So complex trig would likely need its own implementation

[hunterpayne]

What does spire do for that example?  Do they have their own trig functions they provide? Hunter On Wednesday, June 15, 2022 at 05:46:21 PM PDT, AtelierFox ***@***.***> wrote: The trig functions probably can't be well defined for things that are not floating point numbers so that's probably good. I am not sure if sin(i) is defined according to a mathematician. The trig operations are all defined for complex numbers, yes, however there are rules that are not the same as real trig. sin(i) == i * sinh(1) for example So complex trig would likely need its own implementation — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[GregoryEssertel]

Hey there,

What is the status of this development?

Thanks,

[hunterpayne]

So I don't know where Gerry is on this bug.  However, if you need a solution right now you can use this. https://github.com/hunterpayne/terra You will have to compile it yourself but it does solve your problem. Hunter On Sunday, August 6, 2023 at 02:28:59 PM PDT, Gregory Essertel ***@***.***> wrote: Hey there, What is the status of this development? Thanks, — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[hunterpayne]

Oops, I meant Gary, sorry typo. On Monday, August 7, 2023 at 09:00:10 PM PDT, Hunter C Payne ***@***.***> wrote: So I don't know where Gerry is on this bug.  However, if you need a solution right now you can use this. https://github.com/hunterpayne/terra You will have to compile it yourself but it does solve your problem. Hunter On Sunday, August 6, 2023 at 02:28:59 PM PDT, Gregory Essertel ***@***.***> wrote: Hey there, What is the status of this development? Thanks, — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>