Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unexpected result converting to canonical type #21

Open
labkey-matthewb opened this issue May 5, 2021 · 1 comment
Open

Unexpected result converting to canonical type #21

labkey-matthewb opened this issue May 5, 2021 · 1 comment

Comments

@labkey-matthewb
Copy link
Contributor

    Decimal m = ucum.convert(new Decimal("1",15), "[mi_i]","m");
    System.out.println("one mile = " + m + "m expect 1609.344");

Internally convert() first correctly converts 1[mi_i] to canonical unit [m] as 1609.344 but the
conversion from canonical [m] to destination type [m] converts 1609.344 to 1609 because
the identity conversion m->m uses Decimal("1",precision=1)

I would expect the unity conversion to be a noop.

@grahamegrieve
Copy link
Contributor

I think this is an error in UCUM itself. The UCUM value for inch is given as 2.54cm, which is a precision of 3, so 1 mile is 1.60e3m - which becomes 1609 in decimal form. The correct conversion value is 2.539999962 - so raise an issue against UCUM

grahamegrieve pushed a commit that referenced this issue Jul 17, 2023
grahamegrieve added a commit that referenced this issue Jul 17, 2023
#21 - test case (but code is not incorrect) + #22 - test case for spe…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants