[teqc] helpful tip of week 1939

Lou Estey lou at unavco.org
Wed Mar 8 07:17:42 MST 2017


This week's tip: the '+diag' option

This is one of the options I use the most because it what tells a person how
teqc is reading an input file (or to a more limited degree, stdin, if being used),
whether the input is the form of a manufacturer's format, BINEX, or even RINEX.
The main idea is to provide teqc's parsing "diagnostics" of the input.  First,
the option use as of the 2016Nov7 version:

[4698] teqc +help | grep diag
         +diag[nostics] .     output parsing and other diagnostics to stdout
         +diag[nostics] ..    output parsing and other diagnostics to stderr
         +diag[nostics] name  output parsing and other diagnostics to file 'name'
         ++diag[nostics] name append parsing and other diagnostics to file 'name'
         -diag[nostics]       don't output parsing and other diagnostics (default)

(Note: Before 12 Aug 2016, the usage was just '+diag' which would send the parsing
diagnostics output only to stderr.  Remember that now you need to use '+diag ..'
to accomplish the same thing, or use '+diag .' to send this output to stdout.)

Let's take a look at a quick example, say, parsing a Leica MDB file, including
'+quiet' (to suppress any Notice or Warning messages; see tip of week 1911) and
'+mds' (to suppress converting any observation data that's read into RINEX):

[4713] teqc +quiet +diag . +mds unip103-104.mdb
Leica MDB frame 0x9cae @ 0o00000000 = 0x00000000 = 00000000   type= 0x65 = 101
Leica MDB frame 0x9cae @ 0o00000212 = 0x0000008a = 00000138   type= 0x66 = 102
Leica MDB frame 0x9cae @ 0o00000740 = 0x000001e0 = 00000480   type= 0x68 = 104
Leica MDB frame 0x9cae @ 0o00001131 = 0x00000259 = 00000601   type= 0x82 = 130
Leica MDB frame 0x9cae @ 0o00001147 = 0x00000267 = 00000615   type= 0x6c = 108
Leica MDB frame 0x9cae @ 0o00001421 = 0x00000311 = 00000785   type= 0x6d = 109
Leica MDB frame 0x9cae @ 0o00001436 = 0x0000031e = 00000798   type= 0x6e = 110
Leica MDB frame 0x9cae @ 0o00001504 = 0x00000344 = 00000836   type= 0x80 = 128
Leica MDB frame 0x9cae @ 0o00001524 = 0x00000354 = 00000852   type= 0x7d = 125
Leica MDB frame 0x9cae @ 0o00001540 = 0x00000360 = 00000864   type= 0x69 = 105
Leica MDB frame 0x9cae @ 0o00001575 = 0x0000037d = 00000893   type= 0x73 = 115
Leica MDB frame 0x9cae @ 0o00002004 = 0x00000404 = 00001028   type= 0x73 = 115
Leica MDB frame 0x9cae @ 0o00002213 = 0x0000048b = 00001163   type= 0x73 = 115
Leica MDB frame 0x9cae @ 0o00002422 = 0x00000512 = 00001298   type= 0x73 = 115
Leica MDB frame 0x9cae @ 0o00002631 = 0x00000599 = 00001433   type= 0x73 = 115
Leica MDB frame 0x9cae @ 0o00003040 = 0x00000620 = 00001568   type= 0x73 = 115
Leica MDB frame 0x9cae @ 0o00003247 = 0x000006a7 = 00001703   type= 0x73 = 115
...

On each line, the initial part about 'Leica MDB frame 0x9cae' is telling you what
teqc thinks the format is and what it is looking for to determine the start of each
Leica MDB record, which is the byte pair 0x9cae (in hexadecimal) for MDB.

Next, teqc is telling you the location, byte-wise, where the first byte of each
record is located in octal, hexadecimal, and decimal notation, noting that this
uses the convention that the first byte in a file is called "byte 0", which makes
teqc's positioning consistent with the UNIX/Linux command `od` ("octal dump"),
which is a standard low-level tool that can be used to look at the byte-by-byte
makeup of a file.

The last part is telling you what the record "type" or ID is, shown in both
hexadecimal and decimal for MDB.

This is, in a general way, what the output of the '+diag' option should look like
for normal data, assuming teqc can parse the format.

In the above example, teqc is also correctly auto-identifying the file (see tip of
week 1898 and week 1899) as Leica MDB.  Let's suppose we mistakenly told teqc
that this file was a Trimble .dat/.tgd file:

[4713] teqc -tr d +quiet +diag . +mds unip103-104.mdb | more
Trimble .dat frame t @ 0o00000046 = 0x00000026 = 00000038   type= 0x11 = 17
Trimble .dat frame t @ 0o00074327 = 0x000078d7 = 00030935   type= 0x00 = 0
Trimble .dat frame t @ 0o00076723 = 0x00007dd3 = 00032211   type= 0x01 = 1
Trimble .dat frame t @ 0o00077302 = 0x00007ec2 = 00032450   type= 0x44 = 68
next_Trimble_dat_record(): fseek() reposition in '/ap/src/teqc/data/leica/mdb/week_rollover_mdb/unip_201304/unip103-104.mdb' to 0x00007ec3
Trimble .dat frame t @ 0o00077621 = 0x00007f91 = 00032657   type= 0x61 = 97
next_Trimble_dat_record(): fseek() reposition in '/ap/src/teqc/data/leica/mdb/week_rollover_mdb/unip_201304/unip103-104.mdb' to 0x00007f92
Trimble .dat frame t @ 0o00100022 = 0x00008012 = 00032786   type= 0x88 = 136
next_Trimble_dat_record(): fseek() reposition in '/ap/src/teqc/data/leica/mdb/week_rollover_mdb/unip_201304/unip103-104.mdb' to 0x00008013
Trimble .dat frame t @ 0o00100423 = 0x00008113 = 00033043   type= 0x07 = 7
Trimble .dat frame t @ 0o00101157 = 0x0000826f = 00033391   type= 0xff = 255
next_Trimble_dat_record(): fseek() reposition in '/ap/src/teqc/data/leica/mdb/week_rollover_mdb/unip_201304/unip103-104.mdb' to 0x00008270
Trimble .dat frame t @ 0o00101501 = 0x00008341 = 00033601   type= 0x33 = 51
next_Trimble_dat_record(): fseek() reposition in '/ap/src/teqc/data/leica/mdb/week_rollover_mdb/unip_201304/unip103-104.mdb' to 0x00008342
Trimble .dat frame t @ 0o00102600 = 0x00008580 = 00034176   type= 0x8d = 141
next_Trimble_dat_record(): fseek() reposition in '/ap/src/teqc/data/leica/mdb/week_rollover_mdb/unip_201304/unip103-104.mdb' to 0x00008581
Trimble .dat frame t @ 0o00106647 = 0x00008da7 = 00036263   type= 0x06 = 6
Trimble .dat frame t @ 0o00107675 = 0x00008fbd = 00036797   type= 0x14 = 20
Trimble .dat frame t @ 0o00120714 = 0x0000a1cc = 00041420   type= 0x7c = 124
...

and you may get a segmentation or other fault, or if teqc gets all the way through
the input, at the end '+mds' will report:

1980-01-01 00:00:00  1980-01-01 00:00:00  <file size>  <file name>

i.e. it found no usable epochs of observation data in the input.

Do some experiments yourself on your own data files.

If you leave '+mds' and '+meta' out of the command line, then teqc will also output
the conversion of anything it reads into RINEX, which by default goes to stdout.
Thus using '+diag .', which also outputs to stdout, you can see epoch-by-epoch where
your data is located in the input.  For example:

[4716] teqc +quiet +diag . unip103-104.mdb
Leica MDB frame 0x9cae @ 0o00000000 = 0x00000000 = 00000000   type= 0x65 = 101
Leica MDB frame 0x9cae @ 0o00000212 = 0x0000008a = 00000138   type= 0x66 = 102
...
Leica MDB frame 0x9cae @ 0o00015123 = 0x00001a53 = 00006739   type= 0x71 = 113
Leica MDB frame 0x9cae @ 0o00015326 = 0x00001ad6 = 00006870   type= 0x78 = 120
      2.11           OBSERVATION DATA    G (GPS)             RINEX VERSION / TYPE
teqc  2016Nov7      Lou Estey           20170307 20:48:44UTCPGM / RUN BY / DATE
Solaris 5.10|UltraSparc IIIi|cc -m64 SS12.1|=+|*Sparc       COMMENT
BIT 2 OF LLI FLAGS DATA COLLECTED UNDER A/S CONDITION       COMMENT
-Unknown-                                                   MARKER NAME
-Unknown-           -Unknown-                               OBSERVER / AGENCY
462069              LEICA GRX1200       4.03/2.127          REC # / TYPE / VERS
1205                LEIAT504        NONE                    ANT # / TYPE
   -961120.9898 -5946434.9958  2096811.8942                  APPROX POSITION XYZ
         0.0000        0.0000        0.0000                  ANTENNA: DELTA H/E/N
      1     1                                                WAVELENGTH FACT L1/2
      6    L1    L2    C1    P2    S1    S2                  # / TYPES OF OBSERV
      1.0000                                                 INTERVAL
SPIDER                                                      COMMENT
Made by SPIDER v2.0                                         COMMENT
Project creator:                                            COMMENT
JOSE SANTIAGO                                               COMMENT
  SNR is mapped to RINEX snr flag value [0-9]                COMMENT
   L1 & L2: min(max(int(snr_dBHz/6), 0), 9)                  COMMENT
   2013     4    13     0     0    0.0000000     GPS         TIME OF FIRST OBS
                                                             END OF HEADER
  13  4 13  0  0  0.0000000  0 11G13G03G01G06G23G11G07G28G19G08G10
  106376110.208 8  82890547.47548  20242676.530    20242679.8884         50.250
         49.7504
  119990736.939 7  93499420.31746  22833427.237    22833433.3434         43.250
         38.5004
  126061553.593 6  98229846.30746  23988715.401    23988730.0604         40.750
         36.7504
  131745333.539 6 102658813.98845  25070276.092    25070284.0544         36.750
         32.7504
  111910799.304 8  87203275.24847  21295903.318    21295906.4284         49.750
         45.0004
  118351056.918 7  92221696.76046  22521430.665    22521436.7104         46.500
         39.7504
  113611308.136 8  88528409.05847  21619476.667    21619480.3894         48.500
         45.7504
  120439597.594 7  93849072.56246  22918876.756    22918883.3504         45.250
         38.7504
  113586485.041 8  88509163.55747  21614715.894    21614717.8164         48.500
         46.2504
  123085365.650 7  95910749.66846  23422349.641    23422356.3564         43.750
         38.5004
  128872695.420 6 100420338.73145  24523660.836    24523672.9644         41.250
         34.5004
...

Here, the first MDB data record 120 occurs starting at byte 6870 and this record
decoded as RINEX immediately follows.  (This being the first data record, you also
get the RINEX header as well.)

If you dump this whole mess into a file, be forewarned, the result will _not_ be a
RINEX file, so don't try to use it as a RINEX file.  But the result can be incredibly
useful.  In fact, if you've ever wondered how I figure out where in a binary raw
data file things are going wonky, '+diag' was most likely used as a first step.

Also, once you understand how a binary format is structured, you can also use the
this output to see where to "cut" a binary file to break it up.  For example, we
use this option here at UNAVCO when a user accidentally records a single multi-day
(or sometimes as long as multi-month) file instead of many single day files.  For most
formats, we can slice the original raw data file into pieces that are exactly cut
to make perfect daily files.

To this end, when (and only when) using the '+diag' option, you can also dispense
with outputting the RINEX observations since the primary information of interest
are the epoch times. To do this, use '-O.obs -'.  For example:

[4717] teqc +quiet +diag . -O.obs - unip103-104.mdb
Leica MDB frame 0x9cae @ 0o00000000 = 0x00000000 = 00000000   type= 0x65 = 101
Leica MDB frame 0x9cae @ 0o00000212 = 0x0000008a = 00000138   type= 0x66 = 102
...
Leica MDB frame 0x9cae @ 0o00015123 = 0x00001a53 = 00006739   type= 0x71 = 113
Leica MDB frame 0x9cae @ 0o00015326 = 0x00001ad6 = 00006870   type= 0x78 = 120
      2.11           OBSERVATION DATA    G (GPS)             RINEX VERSION / TYPE
teqc  2017Mar7      Lou Estey           20170308 13:50:06UTCPGM / RUN BY / DATE
Solaris 5.10|UltraSparc IIIi|cc -m64 SS12.1|=+|*Sparc       COMMENT
BIT 2 OF LLI FLAGS DATA COLLECTED UNDER A/S CONDITION       COMMENT
-Unknown-                                                   MARKER NAME
-Unknown-           -Unknown-                               OBSERVER / AGENCY
462069              LEICA GRX1200       4.03/2.127          REC # / TYPE / VERS
1205                LEIAT504        NONE                    ANT # / TYPE
   -961120.9898 -5946434.9958  2096811.8942                  APPROX POSITION XYZ
         0.0000        0.0000        0.0000                  ANTENNA: DELTA H/E/N
      1     1                                                WAVELENGTH FACT L1/2
      0                                                      # / TYPES OF OBSERV
      1.0000                                                 INTERVAL
SPIDER                                                      COMMENT
Made by SPIDER v2.0                                         COMMENT
Project creator:                                            COMMENT
JOSE SANTIAGO                                               COMMENT
  SNR is mapped to RINEX snr flag value [0-9]                COMMENT
   L1 & L2: min(max(int(snr_dBHz/6), 0), 9)                  COMMENT
   2013     4    13     0     0    0.0000000     GPS         TIME OF FIRST OBS
                                                             END OF HEADER
  13  4 13  0  0  0.0000000  0 11G13G03G01G06G23G11G07G28G19G08G10
Leica MDB frame 0x9cae @ 0o00016347 = 0x00001ce7 = 00007399   type= 0x78 = 120
  13  4 13  0  0  1.0000000  0 11G13G03G01G06G23G11G07G28G19G08G10
Leica MDB frame 0x9cae @ 0o00017370 = 0x00001ef8 = 00007928   type= 0x78 = 120
  13  4 13  0  0  2.0000000  0 11G13G03G01G06G23G11G07G28G19G08G10
...

This makes for a more compact output to see where in the binary file one needs to
do the cuts.

In the next tip I'll cover a case or two of where '+diag' is showing something unexpected.

Happy teqc-ing!

cheers,
--lou

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Louis H. Estey, Ph.D.              office:  [+001] 303-381-7456
UNAVCO, 6350 Nautilus Drive           FAX:  [+001] 303-381-7451
Boulder, CO  80301-5554            e-mail:  lou  unavco.org
         WWW:http://www.unavco.org      http://jules.unavco.org

"If the universe is the answer, what is the question?"
                                                -- Leon Lederman
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Past helpful tips:

week 1894: using teqc config files - http://postal.unavco.org/pipermail/teqc/2016/002067.html
week 1895: qc of high-rate data - http://postal.unavco.org/pipermail/teqc/2016/002071.html
week 1896: UNIX/Linux shells for Windows - http://postal.unavco.org/pipermail/teqc/2016/002072.html
week 1897: '-' vs. '+' teqc options - http://postal.unavco.org/pipermail/teqc/2016/002076.html
week 1898: auto-identification of formats - http://postal.unavco.org/pipermail/teqc/2016/002092.html
week 1899: auto-identification vs. format flags - http://postal.unavco.org/pipermail/teqc/2016/002096.html
week 1900: square brackets in options - http://postal.unavco.org/pipermail/teqc/2016/002105.html
week 1901: using option '+mds' - http://postal.unavco.org/pipermail/teqc/2016/002108.html
week 1902: qc results w/ problematic nav messages - http://postal.unavco.org/pipermail/teqc/2016/002113.html
week 1903: '-no_orb[it]' and '-no_pos[ition]' options - http://postal.unavco.org/pipermail/teqc/2016/002115.html
week 1904: '-week' option - http://postal.unavco.org/pipermail/teqc/2016/002117.html
week 1905: using '+bcf' for XYZ/geodetic conversion - http://postal.unavco.org/pipermail/teqc/2016/002126.html
week 1906: the '+v[erify]' option - http://postal.unavco.org/pipermail/teqc/2016/002128.html
week 1907: '+C2', '+L5', "+L6', '+L7', '+L8', '+all' options - http://postal.unavco.org/pipermail/teqc/2016/002130.html
week 1908: getting RINEX doppler and L2 - http://postal.unavco.org/pipermail/teqc/2016/002131.html
week 1909: using paths w/ file names - http://postal.unavco.org/pipermail/teqc/2016/002132.html
week 1910: the (un)importance of file names - http://postal.unavco.org/pipermail/teqc/2016/002133.html
week 1911: notices, warnings, and errors - http://postal.unavco.org/pipermail/teqc/2016/002134.html
week 1912: the '-max_rx_SVs' option - http://postal.unavco.org/pipermail/teqc/2016/002137.html
week 1913: the end of '++igs' and '+igs' - http://postal.unavco.org/pipermail/teqc/2016/002140.html
week 1914: splicing together RINEX files - http://postal.unavco.org/pipermail/teqc/2016/002144.html
week 1915: using '-O.int' and '-O.dec' - http://postal.unavco.org/pipermail/teqc/2016/002145.html
week 1916: '+doy' option - http://postal.unavco.org/pipermail/teqc/2016/002146.html
week 1917: '-tbin' and '-ast' options - http://postal.unavco.org/pipermail/teqc/2016/002152.html
week 1918: mp12 RMS before/after Oct 2013 - http://postal.unavco.org/pipermail/teqc/2016/002158.html
week 1919: the global windowing options - http://postal.unavco.org/pipermail/teqc/2016/002159.html
week 1920: '-M.dec' and '-N.dec' options - http://postal.unavco.org/pipermail/teqc/2016/002163.html
week 1921: combining time filtering options - http://postal.unavco.org/pipermail/teqc/2016/002176.html
week 1922: helping me (or someone else on the list) help you - http://postal.unavco.org/pipermail/teqc/2016/002187.html
week 1923: the "build" line - http://postal.unavco.org/pipermail/teqc/2016/002190.html
week 1924: the qc '-w[idth]' option - http://postal.unavco.org/pipermail/teqc/2016/002193.html
week 1925: qc with explicit time windowing - http://postal.unavco.org/pipermail/teqc/2016/002194.html
week 1926: the '+rx_state' option - http://postal.unavco.org/pipermail/teqc/2016/002200.html
week 1927: the '-O.sum' option - http://postal.unavco.org/pipermail/teqc/2016/002204.html
week 1928: the '+meta' and '+mds' options - http://postal.unavco.org/pipermail/teqc/2016/002206.html
week 1930: more on '-O.sum' - http://postal.unavco.org/pipermail/teqc/2017/002207.html
week 1931: the '-O.s[ystem]' option - http://postal.unavco.org/pipermail/teqc/2017/002208.html
week 1932: leap seconds - http://postal.unavco.org/pipermail/teqc/2017/002215.html
week 1936: what you can (and shouldn't) do in a RINEX obs file - http://postal.unavco.org/pipermail/teqc/2017/002229.html
week 1938: the '+psp' option - http://postal.unavco.org/pipermail/teqc/2017/002231.html

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://postal.unavco.org/pipermail/teqc/attachments/20170308/3e24a4e5/attachment-0001.html>


More information about the teqc mailing list