[teqc] helpful tip of week 1976

Lou Estey lou at unavco.org
Tue Nov 21 11:21:44 MST 2017


This week's tip: options '+smtt' (default) vs. '-smtt'

[Note: Because of the historical background covered, a draft of this 'tip of the week'
was sent to Jeff Freymueller.  Jeff sent back some comments, which are included at
the end.]

If you weren't a teqc user back in 2000-2004 when all of this was being discussed,
these two options are probably rather obscure.  And this week's tip is that you
probably never want to mess with the '-smtt' option.  The background:

Back in the early days of civilian GPS use, there were various proposals as to
what to store from GPS receivers so that post-processing of the recorded data
could be attempted.  Two of these protocols were FICA ("Floating Integer Character ASCII")
and ARGO ("Automated Reformatter of GPS Observations"), the second of which formed
the basis from which RINEX ("Receiver Independent Exchange") eventually emerged.
What wasn't clear at all back then was what time base to use.  There were arguments
for using the receiver's time base -- which, if not synchronized with a very good
external clock like an atomic clock, could easily drift milliseconds per hour.
And there were arguments for using GPS time for the measurements' time base.

If you look at the RINEX specification, e.g. for RINEX 2.11 --
ftp://igs.org/pub/data/format/rinex211.txt -- in Section 3, you'll find:

---------------
GPS observables include three fundamental quantities that need to be defined:
Time, Phase, and Range.

TIME:

   The time of the measurement is the receiver time of the received signals.
   It is identical for the phase and range measurements and is identical for
   all satellites observed at that epoch. It is expressed in GPS time (not
   Universal Time).
---------------

You'd think this would be straightforward enough, but a complication arises
when the receiver is applying clock adjustments, or clock resets, which it does
to keep its clock in closer alignment to GPS time.  Typically for receivers
starting in that era and continuing even today, this coarse steering was and is
often done with millisecond resets to the receiver clock time base.  In other
words, the receiver clock is adjusted by steps of -1 or +1 millisecond to keep
the difference between receiver time and GPS time typically within one of these
ranges:

a) -0.5 ms to +0.5 ms, or
b) 0 ms to +1 ms, or
c) -1 ms to 0 ms.

(Whether a), b), or c) is selected depends on the manufacturer and sometimes even
receiver firmware.)  The complication is: how does one represent these receiver
clock resets?

Werner Gurtner, who was one of the primary authors of the RINEX format, solved
this in his Berne translators(*) by putting each receiver clock reset as an increment
to the epoch time tags.  This had the benefit that the phase and pseudorange
measurements remained "smooth" with no jumps at each receiver clock reset.  The
style of RINEX from these translators became the norm, and formed the basis for
what teqc would create some years later.

There was a problem though.  This style of RINEX did not work in JPL's Gipsy
post-processing engine.  Another piece of software was developed to convert
this style of RINEX into yet another style of RINEX that Gipsy could deal with,
where in the second style the epoch time tags did not have the receiver's clock
resets -- i.e. the epoch time base was now smooth -- but the resets were put
into the phase and pseudorange data at the point of each reset, as a jump in
each phase and a jump in each pseudorange, each jump equal to plus or minus
1 millisecond in cycles or meters, respectively.  (The software that did this
conversion from the one style of RINEX to the other was 'clockprep', developed
and maintained by Jeff Freymueller and a few others.)

I don't happen to recall the exact sequence of events, but sometime around
2003 many of us began to wonder why the second style of RINEX wasn't the norm.
(There might have been some change to Bernese processing, which made it more
compatible with the second style; one of Jeff's comments below addresses this).  Another
factor was the discovery that the Javad/Topcon translator of JPS/TPS formats, which
were the same format at that time, directly created the second style of RINEX
and there didn't seem to be any difficulties with post-processing in any of
the upper-end engines.)  Anyway, after some experimentation, it was decided
by the teqc user community and UNAVCO to switch over to creating this second
style of RINEX by default in teqc.

Right after this decision, here at UNAVCO, we re-created all the RINEX on our
GPS/GNSS ftp server using teqc so that it was all in the second style of RINEX
and that's what we've been creating ever since.

So currently:

'+smtt' = translate to RINEX with "smooth time tags", with millisecond jumps in
the phase and pseudorange observables; this is the default; this creates what I
refer to as the second style of RINEX in the background discussion above.

'-smtt' = translate to RINEX with time tags with receiver clock resets, but the
phase and pseudorange observables are smooth; this creates what I refer to as the
first style (Werner's original style) of RINEX in the background discussion above.
(Note: Because the phase and pseudorange observables continue to accumulate
the receiver's clock drift in this flavor of RINEX, the phase and pseudorange
values can start to take on some unexpected sizes, e.g. if the RINEX file covers
a long enough span of time, the pseudoranges can become negative or become
several times larger than they were at the beginning of the file, depending
on which way the receiver clock is drifting.)

Basically, you want to stick with '+smtt', the default, and avoid using '-smtt'.
(I sometimes use '-smtt' as a test during some translating debugging, which is
why it's still around.)  Either option, however, only has an effect when converting
raw formats to RINEX, but only if the raw data has receiver resets; neither option
has any effect when reading RINEX itself.

-----------

There is a curious side-story to all of this.  This is a component of the qc
algorithm that looks for millisecond receiver clock resets in the epoch time tags.
If you happen to come across an old RINEX observation file in the first style
of RINEX and qc it, the 'Clk' line at the end of the ASCII time plot might look
something like this:

...
Clk|   + + + + ++ ++ ++ ++ +++ ++ ++ ++ ++ ++ ++ +++ ++ ++ ++ ++ +++ ++ +++ |Clk
    +--------|--------|--------|--------|---------|--------|--------|--------+
00:01:30.000                                                        23:58:30.046
1998 Jun  4                                                          1998 Jun  4
...

which indicates times, in this case, where +1 ms clock resets occurred; a '-'
likewise would indicate where -1 ms clock resets occurred.  There are four later
lines in the report related to finding the clock resets in the epoch time tags.
For the above case:

...
No. of Rx clock offsets : 46
Total Rx clock drift    : +46.000000 ms
Rate of Rx clock drift  : +1.921 ms/hr
Avg time between resets : 31.239 minute(s)
...

However, if qc-ing the second style of RINEX, you will probably never see these.
(But if you do, it is an indication that the receiver hiccuped and took observations
1 millisecond before of after when it should have taken observations, i.e. that
epoch is -1 or +1 ms off from when it should have been.)  So, when qc-ing the
second style of RINEX, you'll more typically see something like this:

...
Clk|                                                                        |Clk
    +---------|--------|--------|--------|--------|--------|--------|--------+
00:00:00.000                                                        23:59:30.000
2013 Apr 28                                                          2013 Apr 28
...
No. of Rx clock offsets : 0
Total Rx clock drift    :  0.000000 ms
Rate of Rx clock drift  :  0.000 ms/hr
Avg time between resets : Inf minute(s)
...

whether there are really any receiver clock resets or not in the data.

It should be noted that one should be able to retrieve the same receiver clock resets,
if they are present, from the second style of RINEX by looking for the equivalent
jumps in the phase and pseudorange observables.  But this little detail was never
successfully implemented into teqc.  Maybe someday ...

Happy teqc-ing!

cheers,
--lou

* Werner's translators were for formats from the early TI 4100 receiver, the early Trimble
.dat format (see 'tip of week 1957' https://postal.unavco.org/pipermail/teqc/2017/002305.html),
for the Ashtech BESD-fileset download format, and for JPL's ConanASCII (which itself was
converted from ConanBinary, which could be downloaded from the JPL and, then later, AOA
receivers).

Jeff Freymueller's comments:

------------
One thing maybe to mention is that measurements at two receivers need to be synchronized within some tolerance in order to do any
differential positioning. Geoff Blewitt computed the tolerance back in the 1980s, and I don’t remember what he got, but it was on the order
of a few tens of milliseconds when Selective Availability clock dithering was on, and maybe as much as 100 or 200 when it was off. So it is
important technically that the time tags in the data file reflect the actual degree of synchronization of the clocks.

In software that does a double difference, the jumps in the observables simply difference out so their presence or absence only matters if
you do something to the data before differencing. But GIPSY never does a double difference, so it mattered a lot. The older flavor did not
work with GIPSY because GIPSY treated the time tags in the file as being the actual time of the measurement (and time tags had to be
synchronized, as above). GIPSY never implemented a “we know this number is fudged, so let’s undo it” step.

One further historical note is that one original reason Werner made that choice was that, at the time, the Bernese software started by doing
a polynomial clock correction fit to the data (the code data, I think), and this could only work if the code and phase were smooth. Bernese
then accounted for the offsets in the time tag in a second step. This practice was changed long ago, I think in response to the Selective
Availability clock dithering, and after that they did a simple clock correction solution using the data, which works regardless of whether
the code and phase are smooth or not. I don’t remember the exact timing of the decisions, but the habit of changing the time tag persisted
for at least a decade after the software workflow that originally mandated it was abandoned.

The difference between these two ‘flavors’ of RINEX was in whether you defined the ‘receiver clock’ based on the internal oscillator only or
the internal oscillator plus software resets. I think the second is the only really self-consistent approach, which is what teqc does now.

A further historical note is that for a while there was also a third flavor of RINEX generated, this one being a direct reformatting of the
values recorded by the receiver. This was not compliant with the RINEX standard, but in the late ‘80s to early ‘90s there were a few
adherents of this approach (I recall Kurt Feigl, then a student at MIT, being one). In this version the time tags were smooth and so was the
phase, but the code had the offsets at the clock resets. This is actually what the Trimble receivers reported in their raw data, and I think
Ashtechs did the same. Basically, the phase was given relative to the internal oscillator only while the code was given relative to the
internal oscillator plus resets. I think the rationale was that it didn’t matter whether the phase had the offsets because you were going to
do a double difference anyway, which was fine unless you were not doing a double difference (like GIPSY!). You could get this flavor by
running Werner’s converter with the tolerance for detecting the millisecond offsets set really high. Thankfully, this version was never
widely [used].
------------

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Louis H. Estey, Ph.D.              office:  [+001] 303-381-7456
UNAVCO, 6350 Nautilus Drive           FAX:  [+001] 303-381-7451
Boulder, CO  80301-5554            e-mail:  lou  unavco.org

"If the universe is the answer, what is the question?"
                                                -- Leon Lederman
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Past helpful tips:

week 1894: using teqc config files - http://postal.unavco.org/pipermail/teqc/2016/002067.html
week 1895: qc of high-rate data - http://postal.unavco.org/pipermail/teqc/2016/002071.html
week 1896: UNIX/Linux shells for Windows - http://postal.unavco.org/pipermail/teqc/2016/002072.html
week 1897: '-' vs. '+' teqc options - http://postal.unavco.org/pipermail/teqc/2016/002076.html
week 1898: auto-identification of formats - http://postal.unavco.org/pipermail/teqc/2016/002092.html
week 1899: auto-identification vs. format flags - http://postal.unavco.org/pipermail/teqc/2016/002096.html
week 1900: square brackets in options - http://postal.unavco.org/pipermail/teqc/2016/002105.html
week 1901: using option '+mds' - http://postal.unavco.org/pipermail/teqc/2016/002108.html
week 1902: qc results w/ problematic nav messages - http://postal.unavco.org/pipermail/teqc/2016/002113.html
week 1903: '-no_orb[it]' and '-no_pos[ition]' options - http://postal.unavco.org/pipermail/teqc/2016/002115.html
week 1904: '-week' option - http://postal.unavco.org/pipermail/teqc/2016/002117.html
week 1905: using '+bcf' for XYZ/geodetic conversion - http://postal.unavco.org/pipermail/teqc/2016/002126.html
week 1906: the '+v[erify]' option - http://postal.unavco.org/pipermail/teqc/2016/002128.html
week 1907: '+C2', '+L5', "+L6', '+L7', '+L8', '+all' options - http://postal.unavco.org/pipermail/teqc/2016/002130.html
week 1908: getting RINEX doppler and L2 - http://postal.unavco.org/pipermail/teqc/2016/002131.html
week 1909: using paths w/ file names - http://postal.unavco.org/pipermail/teqc/2016/002132.html
week 1910: the (un)importance of file names - http://postal.unavco.org/pipermail/teqc/2016/002133.html
week 1911: notices, warnings, and errors - http://postal.unavco.org/pipermail/teqc/2016/002134.html
week 1912: the '-max_rx_SVs' option - http://postal.unavco.org/pipermail/teqc/2016/002137.html
week 1913: the end of '++igs' and '+igs' - http://postal.unavco.org/pipermail/teqc/2016/002140.html
week 1914: splicing together RINEX files - http://postal.unavco.org/pipermail/teqc/2016/002144.html
week 1915: using '-O.int' and '-O.dec' - http://postal.unavco.org/pipermail/teqc/2016/002145.html
week 1916: '+doy' option - http://postal.unavco.org/pipermail/teqc/2016/002146.html
week 1917: '-tbin' and '-ast' options - http://postal.unavco.org/pipermail/teqc/2016/002152.html
week 1918: mp12 RMS before/after Oct 2013 - http://postal.unavco.org/pipermail/teqc/2016/002158.html
week 1919: the global windowing options - http://postal.unavco.org/pipermail/teqc/2016/002159.html
week 1920: '-M.dec' and '-N.dec' options - http://postal.unavco.org/pipermail/teqc/2016/002163.html
week 1921: combining time filtering options - http://postal.unavco.org/pipermail/teqc/2016/002176.html
week 1922: helping me (or someone else on the list) help you - http://postal.unavco.org/pipermail/teqc/2016/002187.html
week 1923: the "build" line - http://postal.unavco.org/pipermail/teqc/2016/002190.html
week 1924: the qc '-w[idth]' option - http://postal.unavco.org/pipermail/teqc/2016/002193.html
week 1925: qc with explicit time windowing - http://postal.unavco.org/pipermail/teqc/2016/002194.html
week 1926: the '+rx_state' option - http://postal.unavco.org/pipermail/teqc/2016/002200.html
week 1927: the '-O.sum' option - http://postal.unavco.org/pipermail/teqc/2016/002204.html
week 1928: the '+meta' and '+mds' options - http://postal.unavco.org/pipermail/teqc/2016/002206.html
week 1930: more on '-O.sum' - http://postal.unavco.org/pipermail/teqc/2017/002207.html
week 1931: the '-O.s[ystem]' option - http://postal.unavco.org/pipermail/teqc/2017/002208.html
week 1932: leap seconds - http://postal.unavco.org/pipermail/teqc/2017/002215.html
week 1936: what you can (and shouldn't) do in a RINEX obs file - http://postal.unavco.org/pipermail/teqc/2017/002229.html
week 1938: the '+psp' option - http://postal.unavco.org/pipermail/teqc/2017/002231.html
week 1939: the '+diag' option - http://postal.unavco.org/pipermail/teqc/2017/002235.html
week 1951: '-n_<system>' and SV filtering options - http://postal.unavco.org/pipermail/teqc/2017/002277.html
week 1953: more with '+diag' option - http://postal.unavco.org/pipermail/teqc/2017/002287.html
week 1954: using '+diag' output to split raw files - http://postal.unavco.org/pipermail/teqc/2017/002290.html
week 1955: current qc notation - http://postal.unavco.org/pipermail/teqc/2017/002302.html
week 1956: the '+qcq' option - http://postal.unavco.org/pipermail/teqc/2017/002304.html
week 1957: using Trimble formats - http://postal.unavco.org/pipermail/teqc/2017/002305.html
week 1958: ToC != ToE messages - http://postal.unavco.org/pipermail/teqc/2017/002310.html
week 1959: receivers vs. formats - http://postal.unavco.org/pipermail/teqc/2017/002311.html
week 1960: when the '-week' option is very wrong to use - https://postal.unavco.org/pipermail/teqc/2017/002314.html
week 1961: "less" is usually best - https://postal.unavco.org/pipermail/teqc/2017/002315.html
week 1962: using GPS L2C with teqc - https://postal.unavco.org/pipermail/teqc/2017/002316.html
week 1964: the '+eds' options - https://postal.unavco.org/pipermail/teqc/2017/002317.html
week 1965: handling RINEX comment lines - https://postal.unavco.org/pipermail/teqc/2017/002324.html
week 1966: the '+dUTC_p' options - https://postal.unavco.org/pipermail/teqc/2017/002331.html
week 1967: the strange position from '+meta' - https://postal.unavco.org/pipermail/teqc/2017/002355.html
week 1972: what shows up as metadata in RINEX headers - https://postal.unavco.org/pipermail/teqc/2017/002362.html
week 1973: GPS L2C navigation messages - https://postal.unavco.org/pipermail/teqc/2017/002363.html
week 1974: the '+ion_p' options - https://postal.unavco.org/pipermail/teqc/2017/002370.html
week 1975: the '+event' options - https://postal.unavco.org/pipermail/teqc/2017/002372.html

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://postal.unavco.org/pipermail/teqc/attachments/20171121/80f324d2/attachment-0001.html>


More information about the teqc mailing list