-
How does the OPUS release 1.2 differ from the older one?
-
The first CD-ROM version of OPUS corresponded to our Build 1.0,
and our second release corresponds to Build 1.2. In addition to
a few bug fixes, there are specific changes you should be aware of.
A number of enhancements and changes to the OPUS Observation
Manager and OPUS Process Manager have been made over the last
six months. Most of these should be transparent to you, and
many of the changes will be picked up when you install the
new software. However, we wanted to point out that some changes
will require action on your part.
- OPUS is now available on the CD-ROM for the Solaris, Linux,
and Tru64 Unix platforms.
- TABS are no longer acceptable in OPUS path files. PR 34947 implemented stricter rules for the
parser. Path files should be space delimited and not contain TAB
characters.
- The STAGE_FILE keyword is now required in all OPUS path
files. PR 35572 allows the construction of
independent paths with independent pipeline stages.
- New optional keywords in your
process resource files
include CLASS, SYSTEM, DESCRIPTION, and DISPLAY_ORDER keys.
See the description under PR 34046.
- It is now required to source the
opus_login.csh in
your .cshrc. This allows task start-up to be easier and
more consistent.
-
What is a PR?
-
A PR is a Problem Report. The problem reporting system affects all
components at the Space Telescope Science Institute and includes
enhancements, change requests, as well as documentation updates. Not
all of the 35,000+ problem reports have been filed against OPUS!
-
35572: stage file name
-
The pipeline stage file name will now be obtained through a new
path file parameter for ALL OPUS SOFTWARE.
A line similar to,
STAGE_FILE OPUS_DEFINITIONS_DIR:blue_pipeline.stage
should appear in ALL path files (the NULL path is an exception). The
file specification (filespec) for the stage file should be complete.
It can contain logical names. It does NOT HAVE to reference a file in
OPUS_DEFINITIONS_DIR, though this is the recommended place, and the
file does NOT HAVE to have a name like PATH_pipeline.stage. You can
call the file anything you want.
The OMG will print warning messages in the scrolling message window
if a path is chosen for monitoring that does not have the STAGE_FILE
entry, or if the filespec listed there is not a valid stage file or
does not exist on disk.
There is no longer a "default" pipeline stage file that will be used
if the path file does not indicate one. Applications will likely fail
if the path file does not contain the STAGE_FILE parameter with a valid
filespec.
-
33198: listhead changes
-
The
listhead task can now handle wildcards in the source
argument, as well as directory/logical names in the destination
argument.
listhead \*.fits
- lists FITS
headers for all FITS files in current directory to the
same directory, with output names
*.lis
listhead \*.fits N_DADS_DIR
- lists FITS headers for all FITS files in current directory
to the directory defined by the N_DADS_DIR environment variable
listhead OCAL:\*.fits O_DADS_DIR
- lists FITS headers for all FITS files in the directory
defined by the OCAL environment variable to the
directory defined by the O_DADS_DIR environment variable
listhead o3s41010q.fits -f MYDISK:o3s41010q.hdr
- lists FITS header for a specific FITS file to a
specifically named output file
The "\" is required before the wildcard.
The "-f filename" syntax is REQUIRED to use an output filename other
than inputfilename.lis.
-
34046: PMG changes
-
The following capabilities have been added/changed for the PMG:
-
The PMG can monitor paths selectively: a new cascading menu under
"File" displays all paths available to the user (effectively, a scan of
all files named *.path in OPUS_DEFINITIONS_DIR) as toggle buttons (each
initially set on). By toggling the various paths on (filled square next
to path name) or off (no square), the user controls which active
processes are displayed in the main PMG window. Additionally, this
feature converts the "Halt OPUS" menu item (under "Manage") to a
cascading menu that displays an "All" button (always available and
triggers the same behavior as before: halting of all processes in the
PMG display) and a button for each monitored path. By choosing one of
the latter path buttons, the user can selectively halt only those
processes running in that path.
-
The user can define the nodes available for processing within the PMG:
the list of nodes (formally termed queues) is stored in the application
default resource file (i.e.,
pmg.dat). The new option
"Define nodes" under the "Utilities" menu brings up a dialog where the
user can add to or remove from this list, eliminating the need to
manually edit this file. Once the dialog appears, add a new item to
the list by typing it in the "Add/Remove" field and pressing enter. To
remove an item from the list, select it with a single left click (this
also moves it to the "Add/Remove" box for editing). Any changes made to
this list will not take effect until the PMG is exited and restarted;
however, the pmg.dat file is updated immediately and will
reflect the new list should "Define nodes" be chosen a second time
during the session.
-
The user can define print queues and the system print command within
the PMG: similar to #2 above, a new item in the "File" menu, "Print
setup", pops up a dialog where the user can define the available
printers and the command used to print items from various PMG dialogs.
The dialog functions the same as in #2 except there is an additional
text field containing the system print command; simply edit this field
to change it. Note, however, that the full print command is generated
by prepending the system print command indicated to one of the print
queue names (chosen at print time), so the last parameter in the print
command must specify the print queue. Again, the PMG must be restarted
before any changes take effect.
-
The process selection dialog box is organized logically: a new
sort algorithm attempts to logically order the list of available
processes displayed in the process selection dialog. In the past, this
list was generated once at PMG startup and was ordered according to the
file scan of the OPUS_DEFINITIONS_DIR. Now, the user controls this
ordering via two new keys in the
process resource files:
SYSTEM and DISPLAY_ORDER. Each time the process selection dialog is
brought up, the following rules are applied in creating the list of
available processes: (note that this implies that the list is dynamic
now; a new process resource file placed in OPUS_DEFINITIONS_DIR will be
found without recycling the
PMG).
- a list of all processes available is generated by
looking for *.resource in OPUS_DEFINITIONS_DIR;
- each resource file located is parsed, extracting the values
of the SYSTEM, DESCRIPTION and DISPLAY_ORDER keys;
- each process name and DESCRIPTION is inserted in the list
alphabetically by SYSTEM first, then alphabetically by
DISPLAY_ORDER and finally alphabetically by process name.
- "dividers" are added to the sorted list at points
between groups of process with the same SYSTEM value; these
dividers of are the form "---- SYSTEM: name ----" where name
is the value of the SYSTEM key for the processes that follow.
(dividers cannot be added to the list of processes to spawn)
- The pmg_proc.dat file is eliminated: the CLASS key is added to the
individual process resource files, and in conjunction with #4, obviates
the need for this file.
-
34811: .rhost, .cshrc, and opus_login.csh
-
The PMG now supports mixed Unix (Solaris) and VMS pipeline process
management. The method used by the PMG to start up processes has
changed significantly, and imposes new requirements on pipeline set-up
whether or not a mixed platform pipeline is present. These changes are
described below.
- Node names, as defined using the PMG's "Define nodes" option,
must be encoded to indicate whether the node is an Unix host or a
VMS batch queue. The syntax for the node name is as follows:
-
Unix host name:
- use network name (eg., flower or
flower.stsci.edu)
- VMS batch queue:
- use NODE::BATCHQUEUE where NODE is the
VMS host name and BATCHQUEUE is the batch
queue name (eg., for batch queue SYSTEMBATCH
on host COBRA the node name should be
COBRA::SYSTEMBATCH); the host can be any host
capable of accessing the batch queue
- If there are any Unix hosts in the pipeline, a .rhosts file must
exist in the login directory of each user that will run processes
in the pipeline.
The .rhosts
file should contain two entries per line, each entry separated by
a single space. The first entry is a host name, the second a user
name or list of user names separated by commas. The file should
contain one line per host in the pipeline with the login names of
every user that will run pipeline processes from that host. Note
that this file will allow UNAUTHENTICATED access to the host on
which it is present from each host listed provided that the user
is on the list of users for that host. As such the file should be
constructed wisely and made readable only by the owner.
- The new start up procedure for processes requires that
opus_login.com and/or opus_login.csh be 'executed' from the user's
login.com or .cshrc file. This is necessary only for
non-interactive logins and can be restricted as such.
- In order to prevent spurious messages from appearing on terminal
sessions associated with the PMG, login.com and .cshrc should not
send any output to the screen. (Certain job control messages might
still appear).
-
34947: stricter parsing rules
-
The general purpose parsing routines used by OPUS have been made more
general by removing inconsistencies and loopholes applied when parsing
files. In particular, one must call the routines specifying the
correct keyword/value separator (no longer will a
space be
used as a default separator). Furthermore, within a file being parsed
unless a line is commented out with the comment character or is empty,
it should contain a keyword plus separator (at least).
With these stricter requirements, one should ensure that all path files
use spaces (not tabs!) as keyword/value separators and
that all resource files
use "=" as the keyword/value separator.
-
34499: lower case and changes to obsfil*.dat files
-
The OSF
and PSF
files used in the
OPUS blackboard
architecture, and shown in
the OMG
and PMG
displays, will now be in lower case
characters. This was done to address
NFS filename translations that
occur for disks shared between UNIX and VMS file systems.
The error reporting for the OMG delete and copy functions was made
uniform. Since DELETE and COPY are attempted in a variety of
directories as indicated in the
[path]_obsfil_delete.dat and
[path]_obsfil_copy.dat files,
it doesn't make sense to stop if any operation fails since failure in
some of the directories is likely normal.
The new method will count the number of successful deletions or copies
and if NONE are successful an error is reported. If any of the
operations succeeds, then no error is reported to the OMG message
window (errors are still reported by the operating system in the window
in which the OMG was started).
Also, several UNIX [path]_obsfil*.dat files needed updating to meet the
stricter MPARSE rules.
-
35847: redirection in spite of noclobber setting
-
C shell scripts for the OPUS pipeline and sample processes will
properly redirect standard out and standard error in environments where
noclobber is set.
-
35923: RESUME_BLOCKS
-
A new optional keyword for
resource files,
RESUME_BLOCKS, can be used
to specified the amount of blocks needed for a process to exit the
IOWAIT state. If the RESUME_BLOCKS keyword is not in the resource
file, the process will exit the IOWAIT state when available disk space
is greater than MINBLOCKS.
Note: the MINBLOCKS keyword still determines when the IOWAIT
state is entered.
-
35948: double clicking start button in PMG
-
A bug in process selection in the PMG has been removed where the user
could accidentally start two copies of each process chosen by double
clicking the start button (either by accident or impatience).
-
35790: unique time stamping
-
The time stamp appended to a process log file, used by the PMG to
uniquely identify that process, has been made less likely to match the
time stamp of any other process by adding a random number in the range
0 to 99. Since this time stamp is the same as that used for the
Process Status File,
this change also makes the Start time in the PMG imprecise
to 99 seconds, at most.
-
36112: scrollable windows enabled
-
PMG and
OMG confirmation
dialogs containing lists of
Observation Status Files (OSFs) or
Process Status Files (PSFs) to
be deleted, halted, etc., will now include a scrollable window in which
the targeted items are displayed whenever the number of such items
exceeds a user defined threshold. This will prevent these dialogs from
becoming larger than the screen thus limiting access to the "OK",
"Cancel" and "Help" buttons at the bottom of the window.
The threshold at which a scrolled list will be used is configured in
the files pmg.dat and omg.dat in the
resource:
*noScrollLimit : 20
To change the threshold from the default value of 20, edit the
appropriate .dat file and change (only) the number.
Similarly, once the threshold is exceeded, the size of the scrolled
list visible to the user at one time is configurable through the
resource:
*ConfirmMessage*visibleItemCount : 20
in the same files.
![[OPUS]](gif/opus.gif)