[OPUS]

Frequently Asked Questions

OPUS Process Manager


 
 
 OPUS Process Manager

What is the PMG?
The PMG is the Process ManaGer, a Graphical User Interface (GUI) that allows the user to control the computer processes that make up an OPUS pipeline. The PMG communicates with OPUS processes through process status entries. The PMG display looks like this:

[Initial PMG window]
 

Do I need any software not provided on the OPUS CD to run the PMG?
The PMG is written as a Java application and consequently requires that Java 1.3 is  installed on your system. Java is, of course, available without cost from Sun Microsystems, Inc.

 
How do I start the Process Manager?
With the installation of OPUS, there are included special installation packages for the OPUS Managers.  Those instructions allow you to install the Managers on Windows platforms, Linux and many Unix platforms.  For the Windows users, after installation you should see the Opus Managers on your Start/Programs menu:

[OpusManagers Start Menu]

You can drag those icons to your Desktop (with the right mouse button) to put a Shortcut there:

[OpusManager Shortcuts]

The first thing you should see after double clicking on the PMG icon is the OPUS splash screen, and then an OPUS Server login dialog:

The OPUS Managers are able to work over the network only if they have a valid connection with the OPUS CORBA server.   That server needs to have been installed, and needs to be running for the OPUS Managers to have something to talk to.

You need to enter the proper node name where the server is running for the OPUS Managers to find that server.

If the servers are not running, a dialog will popup after you attempt to login telling you the manager could not find the servers:

[Image of No Server Dialog]

Click OK, start the servers, and start the Manager again.

Once you have entered the full node name of the server, you now are required to enter your name and password on that server so the OPUS Managers can log into that account and establish a connection.

Note: if you are using Unix, you can start up the OPUS Process Manager by entering: 

  % ~/PMG
This is a link to the installed task which executes the Java PMG.
 
What's the password dialog for?
The OPUS Managers are able to work over the network only if they have a valid connection with the OPUS CORBA server.   That server needs to have been installed, and needs to be running for the OPUS Managers to have something to talk to.

The name and password being requested are the name and password for the system on which the OPUS servers are running. Note that this information may be different from your user name and password on the system running the managers. In fact, it is possible for anyone to connect to your servers with the managers provided that you give them your account name and password.
 

A note of caution: while the password is hidden from the display, and not logged in the log file, neither is it encrypted.

What is a process status entry?
A process status file is a zero-length file (contains no data) in which all necessary control information about a computer process is contained within the filename itself. With the new OPUS CORBA servers, these files serve only as a backing store for the cached blackboard. We call the cached version a process status entry to distinguish it from its mirror: the process status file.

The Process Manager (PMG) uses these entries to control and communicate with the computer processes running within the OPUS pipeline.

The process status entry is parsed by the PMG to determine what information is to be displayed in the PMG GUI window. The PMG allows the user to modify certain fields in the process status entry to control how the process functions.

The following information is contained in a process status entry: 

   00006426-getkw___-idle___________.340d7ed9-g2f______-area51-____
   |-- 1 -| |-- 2 -| |----- 3 -----| |-- 4 -| |-- 5 --| |- 6 | | 7|
  1. PID: This portion of the process status entry contains the process ID (00006426) assigned by the system at run time (in hex format for historical reasons). The process ID is used to distinguish between multiple versions of the same process running on the same node.
  2. PROCESS: This field of the process status entry reflects the process name (getkw). This information is displayed under the "process" column of the PMG display. The default size of this field is 9 characters, so this is the default maximum size of a process name.
  3. STATUS: This portion of the process status entry contains the status of the pipeline process ("idle"). Some common values of this field are: working, suspended, idle, iowait, starting, absent (process has exited due to an error), or the name of the dataset that the process is currently working on. This information is displayed under the "proc_stat" column of the PMG.
  4. START TIME: This portion of the process status entry contains a time stamp (340d7ed9) representing when the process was started (in hexadecimal format). This information is converted to user-friendly date-time text and is displayed under the "start_time" column of the PMG.
  5. PATH: This portion of the process status entry contains the name of the path under which the process is running (g2f). This information is displayed under the "Path" column of the PMG. The default size of this field is 9 characters, so this is the maximum default size of a path name root.
  6. NODE: This portion of the process status entry contains the name of the machine on which the process is running (area51). This information is displayed under the "Node" column of the PMG. Note: the default size of this field is 20 characters, so this is the default maximum node name length.
  7. COMMAND: This portion of the process status entry is the command area. This tells the process to perform a specific task. In this example, the process status entry contains only underscores (i.e., is "blank"); therefore, the process will continue its normal operation. Possible values for this field are: halt, susp, resu, and init. The commands are issued using the "Manage" menu selection of the PMG menu bar. The command values will tell the process to terminate processing, suspend processing, resume processing, or reinitialize the process, respectively. This information is displayed under the "proc_cmd" portion of the PMG.

 
Can I modify the structure of a process status file?
Yes.

You can configure the sizes, locations, and formatting of fields in a process status entry for your pipelines by changing the default parameters in the file opus.env located in OPUS_DEFINITIONS_DIR . The portion of this file related to process status entries looks like: 

!
! PSTAT definitions
!
! The size of each PSTAT field must be specified here as must a template for
! composing a string representation of the PSTAT out of each field. The PSTAT
! fields that distinguish one PSTAT from another also must be specified.
!
! The formatting and keyword names are self-explanatory although it should
! be noted that each field must appear in the template once and only once,
! there must be at least one field labeled as "UNIQUE", and the template
! format and field sizes should be consistent with the blackboard
! implementation to be utilized (ie., appropriate for file names if BB_TYPE =
! FILE).
!
PSTAT.TEMPLATE_DELIMS = {}
PSTAT.TEMPLATE        = {PID}-{PROCESS}-{PROC_STAT}.{START_TIME}-{PATH}-{NODE}-{PROC_CMD}
PSTAT.UNIQUE1         = PID
PSTAT.UNIQUE2         = NODE
PID.SIZE              = 8
PROCESS.SIZE          = 9
PROC_STAT.SIZE        = 15
START_TIME.SIZE       = 8
PATH.SIZE             = 9
NODE.SIZE             = 20
PROC_CMD.SIZE         = 4
The comments above include the instructions for changing these parameters. The fields marked as "UNIQUE" are those used by the OPUS system to distinguish one process status entry from another. This is important because duplicate process status entries are not allowed on the blackboard. It is not recommended that you change the unique fields although you are free to do so.

Before changing any of these parameters for a pipeline, be sure that there are no process status entries in your OPUS_HOME_DIR, otherwise they will not be recognized after you make the changes. In addition you should make sure that the OPUS CORBA servers are not running.
 

How do I tell the PMG which paths to monitor?
All paths which are defined in your OPUS_DEFINITIONS_DIR will be monitored.  Each path will be present on a different tab in the main part of the PMG:

Just click on the tab for a path to see its active processes.
 

How do I select a new set of nodes?
First, a "new" machine should already be physically and operationally connected to the network of computers being used by the OPUS pipeline (the OPUS pipeline does not have to use all the machines available to the local network, but to use a given machine, it must be available to the rest of the local network). Also, make sure the relevant rhosts files are updated to permit access to the new machine.

Next, choose "Select Nodes" from the "Tools" pull down menu:

[Select Nodes]

This brings up a dialog listing all of the nodes that OPUS knows about (this information comes from the /etc/hosts file on the system running the server):

[PMG Define Nodes window]

Use the mouse (holding the Ctrl key down) to select any number of nodes for your personalized list.  That list of nodes will be used for your current session of the PMG.  If you want to save that list in your personalized environment, use "File.Save".
 

How do I start up a process?
The names of the processes are contained in the tree on the left side of the display.  Double-click on the "Processes" file to see the list of "Systems" which have been defined, and double-click on a "System" to view the "Classes" which are defined for that system:

[OPUS folder] [Processes sub-folder] [OPUS sub-folder]

Finally, double-click on a particular class to view the processes which are available in that class. If you cannot see the full name of the process, you can move the stippled slider and enlarge your view of the process tree.

[ACS CLASS] [Widen window to view all of ACS CLASS]

To select a single process, just drag it from the process tree to the path display:

When you release the mouse button, a new dialog will be displayed. This allows you to select a node, and the number of instances of this process that you wish to start on that node:

You can add a single process by clicking the "Add" option, or move the slider to indicate how many instances of the process you want.  When you click on the "Add" option, the Process Status information will appear in the path display:  Note that the process is "pending" which indicates that it is not active yet.

You can select multiple processes from the tree by using the shift or Ctrl keys when you are making your selections.  Be careful when you start your drag;  if you start the drag operation when the pointer is over a task name, then you might accidentally remove your multiple selection and have only a single process to drag.  To drag a multiple selection, put the cursor in a free area of the tree and start your drag there.

Note that you can drag a whole class of tasks to a path, which is useful if you want all members of that class.
 

What if I make a mistake, how do I remove a pending task?
Select the tasks which you want deleted, and choose the "Delete Pending" option from the "Manage" menu.  Note that only pending processes can be deleted since all other processes are active in the path. 

 

 
How then do I actually start the processes?
Select the processes you want to start on the path.  Then choose the "Start Pending" option on the "Manage" menu:

 

 
This action will send a request to the CORBA server to start the process on the selected node.  The process will show a process status ("proc_stat") of "starting".  When the server has successfully started the task, the process status will say "idle", and the process ID ("pid") will be the actual process ID on that node.

 
 
How do I terminate a running process?
You can terminate a running process (or set of processes) with the PMG by first selecting the process (or processes), then by selecting the "Terminate Selected" option in the "Manage" menu:


 

A confirmation dialog box will be displayed asking for confirmation of the termination request.  If you approve, then a request will be sent to the CORBA server, and the process command ("proc_cmd") will be set to "halt":

 


 

Can I suspend a specific process in the pipeline?
Yes. To do this, use your mouse to select one or more processes found in your PMG screen and then click on the pull down "Manage" menu. Select "Suspend Selected".

 

[Manage Menu with Suspend Selected]
 

This will update the process status entry with a "susp" in the command portion of the process status entry. The OPUS system suspends the process and changes the status of the process to "suspended" on the PMG display. The selected process will then suspend its operation, after the current dataset has completed processing.

Note: Unlike the days of the Motif managers, you should never manually rename the process status entry.  Although there is such a file located in your OPUS_HOME_DIR which reflects all the information in a PSF, that file is just a backing store for the blackboard manager; the "real" information is maintained internally by the CORBA server.  Attempts to modify that backing store might have unwanted effects on the system.


 
How do I resume a suspended process?
You can resume a suspended process by following the same procedure used to suspend that process, except that you select the "Resume Selected" option in the "Manage" pull down menu.

 
How do I bring all processes down?
You can select all processes in the path, either with the cursor (select the first, and then with the Shift key held down, select the last) or by choosing "Select All" from the "Edit" menu:

 
 
Then, as above, choose the "Terminate Selected" option on the "Manage" menu.

 
My Selections are disappearing!
When there is alot of activity going on in the pipeline, your screen is being constantly updated. When that happens, indeed your selections have a habit of disappearing. That makes it difficult to pause or terminate a process.

 
When that happens you can pause the CORBA listener. But please remember that this is a fairly drastic action, and you will want to resume the listener as soon as possible.

 
To pause the listener, select PAUSE in the TOOLS menu:

[Tools/Pause Listener]
 

To later resume the listener, select RESUME in the TOOLS menu:

[Tools/Resume Listener]
 

Note that when you pause the CORBA listener, your screen turns an ugly pink. This is to remind you that the screen is no longer being updated, and can be very out of date.

[Paused listener]
 

What is a pipeline?
A pipeline is a collection of pre-selected computer processes running on a network of computers in which the data flows through a pre-defined disk/directory structure called a "path". The processes, nodes, and paths in a pipeline are defined by the user through the Process Manager.

 
How do I define a pipeline?
A specific pipeline is defined by creating a xxx.pipeline file, where xxx is the name of the pipeline. The *.pipeline file is an ASCII file that contains three separate columns of text that list the process name, path, and node on which the process will be running. For example, this is part of a pipeline file called whopper.pipeline:
   g2f         whopper    area51
   getkw       whopper    area51
   gifin       whopper    area51
   listhd      whopper    area51
   gifin       whopper    orchid
   g2f         whopper    orchid
   getkw       whopper    orchid
The first column contains the process name (g2f for the first line). The second column contains the path in which the data will be processed (whopper). The third column contains the name of the machine on which the process will run (area51 for the first line).

So, if this pipeline file were to be used, the PMG would start the process g2f running under the whopper path on the area51 machine (and so on for the remaining lines).
 

How do I create a pipeline file?
The easiest way to make a pipeline file is to create it with a text editor. The format for each line of the file is exactly the same, as given above:
   process_name     path_name     node_name
As indicated in the example above, you should use lowercase for each of the fields. The columns are space delimited and any number of spaces can be used. Remember that the filename must be of the form *.pipeline. The file should be moved to the OPUS_DEFINITIONS_DIR directory for the PMG to find it.

It should be mentioned that the order in which the processes appear in the pipeline file is arbitrary. In other words, the processes do not have to be listed in the order in which they will be executed in the pipeline.
 

Can I save the list of processes in the PMG as a pipeline definition?
Yes.

Select the processes you want to save for a particular path.  An easy way to select all the processes is "Edit.Select All".  Then choose "Save Pipeline" in the "Tools" menu.

[Save Pipeline]

The selected tasks will be written to the Clipboard in the simple pipeline format.  Note that this is different from the "Edit.Copy Selected" option where all the PSF information is copied to the Clipboard.  When you copy the clipboard contents to a file, remember that the filename must be of the form *.pipeline. The file should be moved to the OPUS_DEFINITIONS_DIR directory for the PMG to find it.

 
How do I load a saved pipeline definition?
All pipeline files found in your OPUS_DEFINITIONS_DIR are automatically loaded when the PMG starts up.  Look at the process tree on the left, and double-click on the "Pipelines" icon.  This will show you the list of pipelines which are currently defined:

[Load Pipeline]

To inspect any one pipeline, double click on its name.  This brings up a display of the contents:

Note that this standard file display has a "Copy" button.  This will copy the file being displayed to the Clipboard.  From there you can paste it in you favorite editor, or mail it to your boss.


 
How do I start the pipeline?
Starting a pipeline is like starting a Class of tasks: just drag the pipeline to the path display.  Since the node and the number of instances are already specified in the pipeline file, the node dialog will not appear, and all processes in the pipeline will be displayed in the path as "pending".


 

From this point you can remove particular processes you don't want to start up yet (Manage.Delete Pending), or start up only selected processes (Manage.Start Pending).

 
OK. I did that, but nothing happened! Now what?
The PMG might have had a problem, but it is pretty good at leaving diagnostic messages around. Look in the Log file for the PMG.  Each user has their own log file which you can access by clicking on "Log" at the bottom right on the screen:


 
Note that the log files are stored in your preferences directory. On most platforms, each user has their own preferences directory.

 
How do I remove a path from my PMG display?
Simple.  Just select the path you want to remove by clicking on the tab for that path.  Then select "Remove Path" from the "View" menu.

The path will reappear the next time you run the PMG unless you delete the path file from your OPUS_DEFINITIONS_DIR.
 

How do I remove a column from the PMG?
In the PMG, the "path" column is clearly redundant, and it, or any other column, can easily be removed.  Just right click on the column header to bring up a short menu, and select "Hide".


 
  If you want to restore a hidden column, right click on the column where you want to insert the hidden column.  Select "UnHide Last" and the hidden column will reappear to the left of the selected column.  In the example below, the "path" column was hidden.  Then it was placed to the left of the "time" column by right clicking on "time" and selecting "UnHide Last".

Note that you can easily move any column anywhere you want it on the display.  Just drag the column header to the position you want the column to appear.  As noted above, any column can be resized as well by dragging the divider between adjacent columns.


 

Can I add a path to the PMG display?
The PMG will search your environment for all the path names and will initially create a tab for each path.  If you create a new path file, you will have to bring a new instance of the PMG up to have that path appear on the PMG display.

 
How do I get help to work?
Actually the Help.Index button will attempt to bring up a browser with this webpage in it.  The Managers are shipped with a default definition of the location of Netscape for Unix, but you can override this.  The default is:
unix.browser.bin=/opt/X11R5/lib/X11/netscape/v47/netscape
unix.browser.wrapper=/opt/X11R5/bin/netscape
You can override this definition in your own initialization file with the path to your preferred browser.

 
How do I know if a task exits abnormally?
If a process exits abnormally, the process status entry "proc_stat" field is updated with the string "absent". The PMG parses this status field and displays the word "absent" in the proc_stat portion of the PMG for that process. A running total of absent (as well as suspended processes, if any) are tracked on the statistics line of the PMG.

[PMG with Absent Process]
 

What do I do if a process exits abnormally?
You can tell what the process was doing by viewing the log file for that process.

 
How do I view the log file for a particular process?
To view a log file for a particular process, the user has two choices:
  1. select the desired process with the mouse and, under the "View" pull down menu, select "LogFile"

    [PMG Utilities Menu with Show Log chosen]

  2. Or, simply double-click on the desired process

[PMG Utilities Menu with Show Log chosen]

Either action causes another window to be created containing the log file for the selected process. The window initially displays the end of the file since pertinent error messages will have just been written there.
 

How do I save my preferences?
Each OPUS Manager has a File/Save option which will record your current settings: which path you selected, which columns you have hidden, which tabs you have defined. The next time you start the Manager, this information will be read to configure your display. This information is saved in your own initialization file with the name:
<username>PMG.ini
That file is saved in your user.home directory known to the Java Virtual Machine, and depends on which platform you are running the Managers on.

Platform user.home
Windows 2000 C:\Documents and Settings\<username>\OPUSDATA
Windows NT C:\Winnt\Profiles\<username>\OPUSDATA
Windows 98 C:\Windows\OPUSDATA
Unix ~/OPUSDATA

 

What is in the PMG.ini file?
The PMG.ini file is an initialization file which is read by the PMG each time you start the application. Both the PMG.ini file, and the corresponding OMG.ini file reside in the .jar file.

The initialization files are formatted as simple keyword = value Java properties. The following keywords are currently defined in the PMG.ini: 

#For unix your default browser is specified here for the help command
unix.browser.bin=/opt/X11R5/lib/X11/netscape/v47/netscape
unix.browser.wrapper=/opt/X11R5/bin/netscape

#this is the list of status values which will be displayed for each path
status_values=idle,processing,absent,iowait,suspended,halted,pending,starting

#omit this line unless you want to see all the debugging messages
#MessageLevel=debug

#if you want to restrict the Manage menu to only one user, put the name here
PermitManage=all
After the PMG.ini file is loaded, the managers will look for a user's file of preferences. This file is named 
<username>PMG.ini
That file is saved in your preferences directory and contains your overrides as well as definitions of how the display should appear.
 
How can I restrict the number of processes that are allowed to run in a pipeline?
It is sometimes necessary to have a single, or a small number of a particular kind of task. For example, if you have a task which scans and modifies the directory of files which are designed to trigger other processes it might be dangerous if two or more of these tasks are running at the same time.

The file pmg_restrictions.dat which is located in the OPUS_DEFINITIONS_DIR contains the control necessary for you to limit the number of tasks.

The syntax of entries is: 

PROCESS.PATH.NODE = NUM_COPIES
where Note that the "*" wildcard entry is allowed for the PATH and NODE components.

For example, 

gifin.g2f.* = 1   !Allows only 1 instance of gifin in g2f path across all cpus
mytask.*.* = 1    !Allows only 1 instance of mytask in all paths across all cpus
It is the CORBA server which uses this file to constrain the number of applications running.  The PMG can request a larger number of instances to be started, but if there is a restriction on that process, the Server will return an error.  If such a request was rejected, a message will be placed in the log file, and the PSF for the process will remain in the pending state.

Can I add functionality to the managers?
Yes. One of the major motivations for the new Java managers was to generalize the core version of the managers, stripping out the HST specific functionality. But we didn't want to lose that functionality, so we added the capability of MGRBeans (like JavaBeans) so that each site can tailor the managers to their own needs.

Without getting into too much detail, the MGRBeans are standard Java classes which implement a simple interface: 

public interface MGRBean {
    public void setFrame (MGRFrame f);  //get a reference to the MGR container
    public String getPreferredMenu();   //tell the MGRs which menu contains you
}
Each new class will appear in it's preferred menu, and will have access to the facilities provided by the Java managers. Contact opushelp@stsci.edu if you need help in writing your own MGRBeans.

 

Top of PMG FAQ

Top of OPUS FAQ