Decoded Epoch type often has wrong timescale

[ljbade]

The hifitime Epoch struct supports different timescales but the decoder often initialises them with the wrong types.

For reference here is how each type of RINEX file should set the timescale:

• clocks -> UTC
• meteo -> UTC
• observation
  • use the value from the TIME OF FIRST OBS header line
  • mandatory header for RINEX 3+, but in RINEX 2 it is only requires for mixed GPS/GLONASS files
  • so if not present default to GPS for pure GPS file, and GLO for pure GLONASS file
• navigation (EOP/EPH/STO/ION) records
  • for RINEX 3+ the value is whatever constellation the satellite is in, so for each record look at the satellite ID, and use that
  • e.g. if the satellite ID is G01 use GPS timescale, E01 use Galileo etc
  • for RINEX 2 the value is the constellation type of the file (since RINEX 2 uses a different file for GPS or GLONASS)

[ljbade]

While working on this I also found a bug in hifitime relates to the GNSS time scales - nyx-space/hifitime#209 - so be sure to the dependency once thats fixes

Also there isn't support for IRNSS/QZSS yet in hifitime - nyx-space/hifitime#210 - so might need to fall back to something else, for QZSS I would use GPS in the meantime. Seems like the decoder doesn't support IRNSS so not an immediate issue there.

[gwbres]

Hello @ljbade,

I am not surprized by incorrect behaviors, due to the many possible cases, that you thoroughly described.
If you feel like correcting these initial conditions, feel free to submit a PR I'll be happy to merge it

[ljbade]

OK, I will work on a small PR based on the changes I made.

[ljbade]

Started on a PR in https://github.com/ljbade/rinex/tree/time-scale

I still need to check that it handles the different cases and add tests then I'll make a PR.

[ljbade]

I have got stuck fixing up the epoch formatting functions to match the parse functions. I can't figure out how to get the Gregorian format from a hifitime::Epoch in some given time scale. There is only a function for returning the Gregorian values in UTC or TAI, and the Epoch::compute_gregorian function is private.

Basically I want to ensure the output RINEX formatted observation times are in the correct time scale for the type of file (e.g. GPS, Galileo, etc) to match the rules for parsing observation times.

[gwbres]

@ljbade,

I have got stuck fixing up the epoch formatting functions to match the parse functions

I'm sorry I won't be able to provide direct help to your branch until I get back from the week end, I can't access a build machine until then

I can't figure out how to get the Gregorian format from a hifitime::Epoch in some given time scale

From the Hifitime API, I only see these methods to answer your requirement

let epoch = Epoch::default();
let new_epoch = epoch.in_time_scale(TimeScale::GST); // let's you convert into a timescale
// you also have this method, but it does not output an Epoch directly
let  iso8601_formatted_str = epoch.to_gregorian_str(TimeScale::GST);

Basically I want to ensure the output RINEX formatted observation times are in the correct time scale for the type of file (e.g. GPS, Galileo, etc) to match the rules for parsing observation times.

In the end, the best thing to do is to enhance the rinex/tests/parsing.rs::test_parser method to have it test the truth table you gave in this topic, for every type of RINEX file

[gwbres]

I initiated support for QZSST. GLONASST and IRNST will follow, it's easier to have them introduced one at a time

[ljbade]

I did find the in_time_scale and the to_gregorian_str string function but that returns it in the wrong format. I will look to see if there is a generic strftime type function that might work.

Otherwise I might make a feature request on the hifitime page.

[gwbres]

I did find the in_time_scale and the to_gregorian_str string function but that returns it in the wrong format. I will look to see if there is a generic strftime type function that might work.

Otherwise I might make a feature request on the hifitime page.

Another option would be

let epoch = Epoch::default();
let duration = epoch.to_duration();
// you also have this, to translate into another timescale at the same time
let duration = epoch.to_duration_time_scale(TimeScale::GST);
// and then use decompose
let (_sign, y, m, d, hh, mm, ss, nanos) = duration.decompose();

[ChristopherRabotin]

@ljbade yes, hifitime supports generic formatting like strftime. You can find examples here: https://github.com/nyx-space/hifitime/blob/master/tests/efmt.rs . The documentation on this feature is a bit hard to find (it even took me a few wrong clicks and I knew what to look for): https://docs.rs/hifitime/latest/hifitime/efmt/format/struct.Format.html .

If you wish to format an epoch in Gregorian, the simplest should be format!("{my_epoch:?}"): that will print your epoch in the current time scale in Gregorian format (cf. https://docs.rs/hifitime/latest/src/hifitime/epoch.rs.html#2872-2891 ).

Version 4 will provide better support: nyx-space/hifitime#231 .

[gwbres]

Hello @ljbade @ChristopherRabotin,

I spent a little time on this, I'm preparing a gnss-time branch that would solve that issue.

In RINEX we're parsing datetimes like 2023-01-01 00:00:00, in otherwords, what Hifitime refers to as "Gregorian" datetimes, but they're expressed in GNSS time scales, in the case of Observation / Navigation RINEX. The easiest option would be to have from_gregorian_gpst and other time scales to solve that case.

Tweaks that might work in the mean time (diff between current stable & gnss-time) : I interpret the string as Gregorian UTC and add the offset to UTC (basically leap seconds on day 1)

[ChristopherRabotin]

Would the work we've started on version 4 solve this issue? If so, I can prioritize it and split up the (rather large) milestone https://github.com/nyx-space/hifitime/milestone/14 into an alpha and beta version.

One of the key differences in version 4 is that the epoch is kept in its initialization time scale all the time, even during serialization. I think we could also fix the Gregorian date parsing so that it's identical in all time scales. Could this leap second bug cause issues in the GNSS computations, nyx-space/hifitime#255 ?

[gwbres]

Would the work we've started on version 4 solve this issue?

There is no emergency in solving those topics. As you can see, we can find workarounds as of today.
We all have our priorities, you don't need to change your plan.
Hifitime is an important topic though, as it is a core dependencies to so many of our libraries

I think we could also fix the Gregorian date parsing so that it's identical in all time scales

I've been thinking about this for a couple of weeks now and I come to the conclusion this behavior is fine.
(Talking about other timescales than TAI/UTC).
When parsing a Gregorian datetime like 2023-09-09 00:00:00 in GPST, you just need to consider the datetime that timescale was launched (Day 1), expressed as Gregorian. From that difference, you get the inner duration you want to store

[gwbres]

Hello @ljbade @ChristopherRabotin

well this gets confusing and blocks the position solver topic to move forward at the moment.

I thought I had a workaround in gnss-time but that "cannot" work: let me explain.

The work around (gnsstime:src/epoch/mod.rs:parse_in_timescale) provides a solution to "Gregorian" datetime parsing, at the expense of little dirtiness (leap seconds).
My goal is to provide my position solver with all Time of embedded Clocks represented in the correct time scale, as specified in the first post here. But that cannot happen in hifitime v3 - at least easily.

The "problem" that I did not expect, is due to the current (hifitime v3) .eq() implementation.
In RINEX we parse time of embedded Clocks that look like these :

2023-01-01 T00:30:30 (sv E01)
2023-01-01 T00:31:45 (sv G31)

and if you're unlucky, 2023-01-01 T00:31:45:GPST is the same duration in TAI as 2023-01-01 T00:30:30:GAL (do you get it ?).
Then, after parsing, I sort everything in a dictionnary.
The .eq() behavior has one of them dropped out, because they're considered identical.

Bottom line: impossible to have a method that proposes to the position solver the "correct" time representation easily.
Does not mean it's impossible, but you always have to introduce cast everywhere, deal with the leap seconds.. etc..etc..

Yet another behavior that gets smoothed out if we define an Epoch from it's timescale T0

[ChristopherRabotin]

I'm not very familiar with the different GNSS clocks: is "sv E01" and "sv G31" supposed to be the same time scale but on different spacecraft, and exhibit their own internal drift? Or should they actually be the same epoch, even if the Gregorian representation is different?

If both epochs in fact represent the same duration since reference epoch then .eq() should return true. And if so, maybe the dictionary is not the right way to store them. Do you have a link to the solver algorithm and a description of the algorithm? Maybe storing each of the epochs as a sorted list would work: it'll be O(log N) iterations to find the right epoch instead of O(1) but at least the solver would work?

[gwbres]

Hello @ChristopherRabotin,

I'm not very familiar with the different GNSS clocks: is "sv E01" and "sv G31" supposed to be the same time scale but on different spacecraft, and exhibit their own internal drift? Or should they actually be the same epoch, even if the Gregorian representation is different?

E01 and G31 are two satellite examples, with two different free running clocks onboard.
One belongs to the Galileo time system, and the latter to GPST.
Since they're free running and different clocks, they drift differently.
Each clock has its own behavior that we compensate for thanks to parameters that the vehicle broadcasts over radio.

Or should they actually be the same epoch, even if the Gregorian representation is different?

No, it's just a description of a free running clock, in the form of a timestamp/datetime.
And that relates to your other question

Do you have a link to the solver algorithm and a description of the algorithm?

It is important during positioning that we express the time as the number of seconds within the related timescale (past its T0).

The algorithm basically solves [x, y, z, dt] where (x,y,z) are the radio receiver's location and dt its own onboard clock offset.
To solve that, we need 3 or 4 SV for which we must know the location in the sky: we have that by solving Kepler or by using SP3.
And we need the exact instant at which the SV sent the radio message. To get that, you need to grab the datetime we're talking about, and remove the time of travel of the signal (a couple seconds, that you need to calculate). This instant is called the "transmission time", while the datetimes we"re talking about and that we have in RINEX are called the "time of clock".
I pretty much have all of that in my "solver" branch. I completed the positions, I'm working on the transmission time.
To summarize, we need the transmission time expressed as past seconds since GPST in case of GPS.

For better and in depth explanations, you need to download Volume 1 at the bottom of this page.

• page 107 with the models
• page 109 transmission time
• page 116 which instant we're talking about,
• page 152 the actual solving process

we will not compensate for relativistic effects in V1, including ones that affect the clock offsets.
You can summarize relatistivistic effects as restricting the accuracy to 1-5 meters, which is totally fine for a V1, compatible with SPP basic solving method and doing better involves much more compensations than just relativistic effects

And if so, maybe the dictionary is not the right way to store them

I can stick to dictionaries, it's just that we don't have means to parse gregorian datetimes correctly right now. And my "tweak" is not correct either. I can fix that by using Epoch::from_gpst_seconds() for example, which works totally fine, but requires me to first calculate the seconds past GPST_REF

[ChristopherRabotin]

Okay, I think I understand the principle of the algorithm now, thanks.

How do we fix this? Is the core issue that the Gregorian computation may be wrong in GNSS scales? Or is it that .eq() will mark two epochs are equal even if they are in different time scales? Or, is the issue that each spacecraft should be considered to be in its own unique time scale because of the drift?

[gwbres]

Hello @ChristopherRabotin,

How do we fix this? Is the core issue that the Gregorian computation may be wrong in GNSS scales? Or is it that .eq() will mark two epochs are equal even if they are in different time scales?

Well, the current behavior is fine, I mean it simply matches the J1900 referencing. It's just it has this side effect that I did not see coming.

For me it demonstrates the current definition is wrong.
Referencing everything to the timescale T0 will solve most problems: constructing from Gregorian datetimes gives different results. But Eq() must still be discussed.

When you're talking about Epoch in a system, you expect something that is unique. By that I mean the Epoch happened at some point and will not happen again in the case of our free running clocks.

Maybe that proves that the next Eq behavior should simply be :

PartialEq()
     a.duration == b.duration && a.time_scale == b.time_scale

Otherwise this :

PartialEq()
     a.duration == b.into_time_scale(a.time_scale).duration

Will produce this behavior :

• 2010-01-01T00:00:00 (GPST) is roughly 30 years
• 2027-08-01T00:00:00 (GST) is roughly 30 years and they are equal

In the case of this crate, that can be worked around by using another indexing method (like Vectors insteand of Dictionaries).

[ChristopherRabotin]

I quite like this definition:

PartialEq()
     a.duration == b.duration && a.time_scale == b.time_scale

I think it's physically more correct than converting time scales, and it's significantly simpler to implement. I think however that there should be another method that performs the conversion quickly to quickly check if two epochs in different time scales match (or we could leave it up to the user to convert to whichever time scale is correct).

@ljbade , what do you think?

[gwbres]

Hello,

i'm happy to announce the workaround works fine.

Basically I just construct an epoch using Epoch::from_str("Y:M:D HH:MM:SS {TS}"), to have the correct timestamp represented in that timescale. It's kind of cumbersome but all the maths after that are correct, which is want we want and the most important.

For example, the Keplerian solver is not broken, which is quite a statement considering we test for millimeter errors, and a 1ns error means +/- 30cm.

I will propose the gnss-time branch to merge.

[ljbade]

I don't fully follow the discussion but I have always found it useful to convert all the different GNSS timescales to a single one e.g. GPS time and then do the solving with that. But the timescale conversion stuff is needed when computing satellite positions.