[teqc] modification to next version of teqc for extracting/detecting certain metadata from raw data
lou at unavco.org
Thu Nov 1 10:13:45 MDT 2018
The next version of teqc will have modifications for extracting and detecting a
change within a single raw data file or stdin stream to any of these metadata fields:
- receiver type (*)
- receiver ID number (*)
- receiver firmware
- antenna type (possibly including radome type)
- antenna ID number
... assuming these pieces of information are in the raw data at all and in a way that teqc
has been written to read.
This seemed to be a necessary strategy given the growing frequency of receiver firmware
changes in raw data from permanent stations. In other words, instead of having a break
in file collection and file transfer at the point of a firmware change of the receiver
at a permanent station, the normal modus operandi seems to be a slight pause in data collection,
the firmware change is done, and a continuation of the data into the current data file
or stream occurs -- unless, perhaps, the operator does something special (like intentionally
starting a new file after the firmware change).
Beside receiver firmware, this approach was then extended to include the type and ID number
of the receiver and antenna. For the antenna, assumptions are made that the operator actually
entered the metadata for the antenna change correctly into the receiver and that the receiver
supports such an entry while a data file is being made or the stream is being output.
The low-level design of much of teqc's functionality in metadata collection was done
when almost all data was collected in campaign style: setting up over a monument marker
for a relatively short time (typically hours to weeks) and then moving on to another
site. It was probably almost inconceivable to change the receiver or antenna set up
in such a temporary fashion, so that these possibilities were simply not a issue. And
for a long period of time, such changes weren't really much of an issue for permanent
station data. But it's becoming quite clear this is no longer the case, and the UNAVCO
archive group (of which I'm a part) really needs a better way to detect such changes
possibly occurring in the thousands of sites from which we receive GNSS data every day --
which was the motivation behind these modifications to teqc. Obviously, we'll be
testing this in the upcoming weeks.
So, next, I'll try to summarize the upcoming behavioral modifications to teqc, and also
try to make clear what hasn't been modified:
1) If teqc detects a change in any of these fields, such as firmware, in the raw data
input file or stdin stream, then it will issue an 'Alert' message going to stderr, e.g.:
! Alert ! '<filename>': change of rx firmware from '<firmware_old>' to '<firmware_new>'
(If using a stdin stream, <filename> will be "stdin".)
This 'Alert' message is a new category of stderr message (e.g. see tip of the week
https://postal.unavco.org/pipermail/teqc/2016/002134.html). It will not be suppressed
by '-notice' or '-warn' (because it is not a 'Notice' or 'Warning') and it will not
be suppressed by '+quiet' (= '-notice -warn'), and it's not really an 'Error' either. (**)
2) If using '+meta' and there is a change in, say, firmware, the _last_ found firmware version
will be the one shown on the '+meta' output line:
receiver firmware: <last found firmware>
... unless the '-O.rv <firmware>' option is used to override and set the firmware version,
in which case the firmware version specified in this option should be shown.
(Prior to this change in teqc, teqc would report the _first_ found firmware version on the
"receiver firmware:" line, unless '-O.rv <firmware>' was used.)
The same precedence is will also be used for reporting the type and ID number of the receiver
3) The 'O.*' options:
-O.rt "<type>" for receiver type
-O.rn "<IDno>" for receiver ID number
-O.rv "<fw>" for receiver firmware
-O.at "<type>" for antenna type
-O.an "<IDno>" for antenna ID number
can still be used to override and set these fields in the '+meta' results, or, when
translating, in the RINEX obs header or in a BINEX 0x00, but doing so has no impact
on (1) above. (But, in general practice, none of the '-O.*' editing options should be
used to truly get metadata when using '+meta'.)
4) If translating and not using the '-O.*' option for a field, then what goes into
the RINEX obs header or BINEX 0x00 record depends on the last found field in the raw
data before the RINEX obs header or BINEX 0x00 record is written.
The modifications to do all of this required touching quite a few lines of code in teqc,
but most of the changes were fairly straightforward, so I'm not anticipating any (or too many)
problems because of the changes.
If you have any questions about any of this discussion, please let me know.
Louis H. Estey, Ph.D. office: [+001] 303-381-7456
UNAVCO, 6350 Nautilus Drive FAX: [+001] 303-381-7451
Boulder, CO 80301-5554 e-mail: lou unavco.org
"If the universe is the answer, what is the question?"
-- Leon Lederman
* How can there be a change of the actual receiver (implied by a change of type and/or
ID number) in a single data file? Actually, it's quite possible because many of the
files we check are made from sub-daily hours, often hourly files. So if the physical
receiver was changed, and the new receiver creates the same type of raw data format as
the old receiver, then the correct receiver fields should be in each of the sub-daily
files created by their respective receiver. And, when the sub-daily files are combined
into a daily file, one or both of these receiver fields could appear to change value.
** I may review current Notice and Warning messages and try to determine if any of
those warrant an upgrade to this new 'Alert' status.
More information about the teqc