![[OPUS]](gif/opus.gif)
A useful feature of this system is the ability to monitor the status of both datasets and processes.
To take full advantage of the OPUS environment, tasks can be written as internal pollers built with the OAPI.
The default size of the "PROCESS" field in a process status entry is currently set at a maximum of 9 characters, which effectively limits your process names to this size or less. However the OAPI supports changes to the process status entry structure, including the size of this field.
The script should get its arguments from the command line, from environment variables, or from the process resource file. All keywords prepended with ENV. in the process resource file become environment variables so these variables are easily accessible.
This is a standard way to get information into your task. OPUS pipeline tasks run in the background, but command line arguments can be specified in the process resource file. This is a convenient way to use the same task for slightly different functions.
Keywords from the process resource file prepended with ENV. and their values are defined as environment variables. Note that ENV. does not become part of the keyword name. These environment variables only are available in the shell in which your task is run.
There are additional environment variables defined by the OPUS system as a task is started in response to an event. First, all tasks have access to the EVENT_TYPE variable that takes on one of the following three possible values:
EVENT_FILE_EVENT, EVENT_OSF_EVENT, or EVENT_TIME_EVENTEach of these values corresponds to the type of trigger that caused the event. The number of items in the event is placed in the EVENT_NUM variable. Unless you have configured your application to handle more than one item per event, EVENT_NUM will be 1.
Tasks that are triggered by a file event additionally have access to the EVENT_NAME variable:
EVENT_NAME The filename which triggers the event.
If EVENT_NUM is greater than 1, then there will be additional environment variables defined of the form EVENT_NAME1, EVENT_NAME2, etc..
Tasks that are triggered by an OSF have access to all the information in that OSF:
OSF_DATASET The name of the exposure that triggered the task. OSF_DATA_ID The type of the exposure (by default, a 3 character descriptor). OSF_DCF_NUM An arbitrary sequence number. OSF_START_TIME The time the exposure started in the pipeline.As in the case of file events, if EVENT_NUM is greater than 1, there will be additional environment variables defined but followed by a number (e.g., OSF_DATA_ID1) for each item in the event.
Time events have no event-related environment variables defined other than EVENT_TYPE.
You can use the values of these environment variables as command line arguments to the tasks you write, or in the bodies of the tasks themselves. See the path file section for more details on the relationship between path file variables and the environment variables from process resource files.
The OPUS system uses information in the process resource file to decide when to activate a process. In the case of external pollers, xpoll responds to an event by spawning its associated process that in turn communicates back to xpoll how successful it was in processing the event through an exit status code. The code is mapped to specific keyword values in the process resource file by xpoll, and the OPUS system is informed of the disposition of the event. External pollers are started by the OPUS system each time work is required, then they exit to be started again later by the OPUS system when more work is needed.
Internal polling processes, like g2f, are programs written with knowledge of how the OPUS blackboard works. They are typically processes with some significant start-up overhead (e.g. database reading, etc.). The process is written to perform the start-up overhead and then enter a polling loop to wait for pipeline events. The process stays active as it polls for and processes events. Internal pollers are built using the OAPI to communicate with the OPUS system, and can respond to a reinitialization command.
TASK = < xpoll -p $PATH_FILE -r listhd >You then need to define the triggers for this process. In the simplest case, the task will be triggered by a flag in the Observation Status File. You need to select a two-character mnemonic for your task; this will be the OSF stage title which the pipeline.stage file uses as the name of the column on the Observation Manager's display. We'll use "LH" in this case. You might just make the trigger:
OSF_RANK = 1 ! First Trigger OSF_TRIGGER1.LH = w ! Need a "Wait" flag in LH columnYou can then tell OPUS how to set the flags during processing, when processing has completed normally, and when the task encountered an error with the observation:
OSF_PROCESSING.LH = p ! Set the processing flag to "Processing" OSF_OK.LH = c ! Completed header listing OSF_OK.NX = w ! Waiting for the next step NX OSF_ERROR.LH = e ! Error: Set the error flagThe link between the return status of the script and these flags is defined by the XPOLL_STATE keywords in this file:
XPOLL_STATE.01 = OSF_OK XPOLL_STATE.03 = OSF_ERRORThus when your process completes with the "exit 1", OPUS will use the OSF_OK tags, and there may be more than one of these. When an error is encountered by the task and it completes with "exit 3" then the OSF_ERROR tags will be used by OPUS.
There are a number of standard resource file lines which should also be present; see the more complete discussion of process resource files.
Because your new task will likely fit in the pipeline after an existing task, you will probably want to modify the preceding task's process resource file so, on normal completion of that first task, the Observation Status File flag is set properly for your new process.
STAGE03.TITLE = LH
STAGE03.DESCRIPTION = "List Headers"
STAGE03.PROCESS01 = listhd
In addition, remember to increment the NSTAGE line in this file.
NSTAGE = 6
The name of the task is used in the construction of the process status entry, and that file has a limited number of characters to hold the task name. The default limit is nine (9) characters, however this limit can be changed.
Status messages to the standard output device will automatically be kept in a process log file. It is extraordinarily useful to write to the log file both wisely and often to document the actions taken by a pipeline task.
Also keep in mind that an external polling process has access to process resource file keywords and values through its environment only for those keywords prepended with ENV..
Note that when the report level is set to MSG_NONE, no log files are produced.