dbnerdz ....: July 2011

Wednesday, July 20, 2011

Enabling the Apache, OC4J and OPMN logging in Oracle Applications R12

1. Oracle Applications R12 logging essentials:

Oracle Applications / E-Business Suite R12 introduces new easy way in order to enable logging for any of the components that are being used within it. Also it uses new directory structures for Oracle Home(s) either for the Database Tier or the Middle Tier.
Since this note is discussing the logging of middle tier services (Apache, OC4J & OPMN), let us give a spotlight at the following essential directory structures for the above middle tier services and its logging information.
- Oracle Configuration Home for Apache, OC4J & OPMN

($ORA_CONFIG_HOME/10.1.3)

- Apache server is still being used and its configuration path is as following:

$ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf

- OC4J services , by default Oracle Applications R12 creates 3 OC4J instances:

OACore: runs OA Framework-based applications
Forms: runs Forms-base applications
OAFM (Oracle Apps Fusion Middleware): runs web services, mapviewer, ascontrol

Configuration files path for the above 3 OC4J instances is:

$ORA_CONFIG_HOME/10.1.3/j2ee/<oacore, forms, oafm>/config

Configuration file used to adjust log file path for the above 3 OC4J instances is:

$ORA_CONFIG_HOME/10.1.3/j2ee/<oacore, forms, oafm>/application-deployments/<oacore, forms, oafm>/orion-application.xml

2. Log files types:

Oracle Applications / E-Business Suite R12 can be configured to generate one of two types of logs files as following:
    a- Plain text (default):         - No log rotation available
         - Log files need to be manually removed
    b- Oracle Diagnostics Logging (ODL) XML:
         - Logs rotation based on filesize, directory size
    c- R12 logs by default are configured to generate plain text

3. Log files locations:

All log files are written in one home ( $LOG_HOME ) which is by default the (logs) directory under ( $INST_TOP ).

Please note that after enbaling the logging for any of the following services you have to bounce the services in order the changes to take effect.

4. Steps to enable Apache logging:

4.1. Apache Plain Text Logs
a- By default the Apache log level is set to (warn) in

$ORA_CONFIG_HOME/10.1.3/Apache/Apache/conf/httpd.conf
LogLevel warn

Also you can update context variable (s_apache_loglevel) in the context file ($INST_TOP/appl/admin/<SID_HOSTNAME.xml>) to set Apache log level and then run autoconfig.

b- You can select the desired LogLevel according to the following table:

Log Level	Description
emerg	Emergencies, system is not useable
alert	Action must be taken
crit	Critical conditions
error	Error conditions
warn	Warning conditions
notice	Normal but significant condition
info	Information
debug	Debug level messages

c- To enable debug set ( LogLevel debug ) in (httpd.conf) file
d- Apache log files will be generated in the following path:

$LOG_HOME/ora/10.1.3/Apache

             e- There are 2 distinct types of log files are created:
                  - Access Log (CustomLog)
                    Filename format should be like: access_log.<unique id>
                  - Error Log (ErrorLog) this includes ECID information
                   Filename format should be like: error_log.<unique id>
      4.2. Apache ODL Logs

Important Note: Please make sure that you have a directory called "oracle" under $LOG_HOME/ora/10.1.3/Apache/ before enabling the logging and if not please create it first.

a- To enable ODL logs you should set the following parameters in (httpd.conf) file

OraLogMode <value>
OraLogSeverity <message type>:<message level>

b- Set (OraLogMode) to (oracle), there are 3 options as per following table:

Value	Description
oracle	Multi-line log records in XML format, provides the most information
odl	Standard apache log format and ECID info. Default setting in R12
apache	Standard apache log format

c- Set (OraLogSeverity) according to the desired log level using following syntax:

OraLogSeverity <message type>:<message level>
Message type: INTERNAL_ERROR, ERROR, WARNING, NOTIFICATION & TRACE
Message level: 1-32 (1 most severe, 32 least)

d- Following table shows (Oracle Message Type) that can be used to set log level:

Apache Log Level	Oracle Message type
emerg	INTERNAL_ERROR:16
alert	INTERNAL_ERROR:32
crit	ERROR:16
error	ERROR:32
warn	WARNING:32
notice	NOTIFICATION:16
info	NOTIFICATION:32
debug	TRACE:32

e- Apache ODL log files will be generated in the following path:

$LOG_HOME/ora/10.1.3/Apache/oracle

5. Steps to enable OC4J logging:

a- Each OC4J instance has a separate logging configuration file:

$ORA_CONFIG_HOME/10.1.3/j2ee/<oacore, forms, oafm>/config/j2ee-logging.xml

b- By default the log level is set to something like:

<logger name='oracle' level='NOTIFICATION:1'...

c- You can set level to the desired value as using following syntax:

<message type>:<message level>
Message type: INTERNAL_ERROR, ERROR, WARNING, NOTIFICATION & TRACE
Message level: 1-32 (1 most severe, 32 least)

d- Each OC4J instance has file:

$ORA_CONFIG_HOME/10.1.3/j2ee/<oacore, forms, oafm>/application-deployments/<oacore,forms,oafm>/orion-application.xml

         Which contains tag like: <log> <file path=...> </log> to specify the log file name.
      e- Each OC4J has its own log file in the following path:
          - Plain Text Log:

$LOG_HOME/ora/10.1.3/j2ee/<oacore, forms, oafm>/<oacore,forms,oafm>_<default_group_1>/application.log

- ODL Log:

$LOG_HOME/ora/10.1.3/j2ee/<oacore, forms, oafm>/<oacore,forms,oafm>_<default_group_1>/log.xml

6. Steps to enable OPMN logging:

     a- OPMN log are generated in ( $LOG_HOME/ora/10.1.3/opmn).
     b- There are 3 log files generated:
          - (opmn.log): contains commands, exit status
          - (opmn.dbg): contains debug log messages
          - (opmn.out): contains stdout & stderr messages
               - Stdout messages are any messages written by application by System.out.println
               - Sterr messages are error messages recorded when opmn tries to start the services
     b- The above log files are plain text files.
     c- Rotation is enabled for the above log files based on file size only defined in:

$ORA_CONFIG_HOME/10.1.3/opmn/conf/opmn.xml

through parameters: ( s_opmn_log_rotation_size, s_opmn_log_rotation_time )

d- Logging is enabled per component (internal, ons or pm)
e- Levels that can be set are (component codes) as following:

none, fatal, error, warn, notify (written to .log)
debug1, debug2, debug3, debug4 (written to .dbg)

f- As an example, Set level for a running opmn as one of the following:

opmnctl set target=log comp=warn
opmnctl set target=debug comp=debug1

7. Components (Services) Status & Logs:

a- You can check the status of the services/components by passing 'status' parameter to any AD script
in $ADMIN_SCRIPTS_HOME.

b- The various logs written while starting and stopping the services are created in
($LOG_HOME/appl/admin/log/), the logs files are:

Service	Log file name
TNS Listener Start/Stop log	adalnctl.txt
Fulfillment Server Start/Stop log	jtffmctl.txt
Oracle HTTP Server start/stop log	adapcctl.txt
Concurrent Managers and ICM start/stop log	adcmctl.txt
Forms OC4J start/stop log	adformsctl.txt
OACore OC4J start/stop log	adoacorectl.txt
OAFM OC4J start/stop log	adoafmctl.txt
OPMN start/stop log	adopmnctl.txt

(adstpall.log) and (adstrtal.log) will record the messages while starting or stopping all services.

Tuesday, July 19, 2011

Error: Signal 25 Concurrent Program Completes with Error

When attempting to run a concurrent program which is accessing file "reports.log" in directory $INST_TOP/logs/appl/conc/log then following error occurs: "Error: emsg:was terminated by signal 25".

This happens when the file size of "reports.log" has reached its maximum limit at operating system which is 2GB.

If $INST_TOP/logs/appl/conc/log/reports.log will be huge -- 2GB in size, then reports can not append anything else to it and hence concurrent program accessing reports.log for its execution will error out.

To implement the solution, please execute the following steps:

1. Stop the concurrent manager.

2. Rename/Truncate existing "reports.log" in directory $INST_TOP/logs/appl/conc/log/reports.log and create a empty "reports.log".

3. Start the concurrent manager.

4. Retest the issue.

5. Migrate the solution as appropriate to other environments.

CSTRINVR - Inventory Valuation Report Errors with MSG-00999 and REP-1419

Issue :
When running the Inventory Value Report (CSTRINVR), you get the following

errors:
    MSG-00999 1: user defined exception
    MSG-00999 cstrinvr > X 30-Jun-1997 ...
    REP-1419 'beforereport : pl/sql program aborted.

Answer :

Complete the following steps:

    1. Login to System Administrator responsibility.
    2. Navigate to Profile/System.
    3. Display User profiles.
    4. Change 'CST:View Cost Privilege' profile option = YES.
If the profile option is set to No, the Inventory Value report will complete 
with error because of Insufficient Cost View Privileges.
Explaination : 
By changing 'CST:View Cost Privilege' profile option to YES, this error does
not occur.

System Hangs or Errors Seen in Event Viewer or /var/log/messages for FRMWEB When Forms Session Closed Using Browser

Applicable to :

Oracle Forms - Version: 10.1.2.0.2 and later [Release: Oracle10g and later ]
Linux x86-64

Warning or Error Message :

In /var/log/messages :
kernel: frmweb[16576]: segfault at 000000006974655e rip 00000000081d3ca9 rsp 00000000ffff9d2c error 4
(most numbers are not significant, they will likely differ on your system)
On Unix, huge core files can also be generated, possibly causing a system hang or various problems.

Reason for occurance :

The users are closing the Forms sessions using the browser and not by using the exit mechanisms in the Forms.

Answer :
To prevent these error messages, close the Forms application normally before closing the browser.

The correct procedure to

-- Close a form -- F4

-- Query a form -- F11

-- Find the query -- Ctrl + F11

If using Internet Explorer, you can pop-up a message window when users attempt to close the browser by following indications from NOTE 605327.1 - How To Trap Attempts At Closing The Parent Browser Running A Forms Application given in the oracle metalink

On Unix, setting "ulimit -c" to "0", for example before launching the "opmn" processes, will prevent writing of further "core" files. This can avoid completely the risk of a huge frmweb "core" file possibly impacting the system in case a user would close his browser without exiting Forms first:

ulimit -c 0
opmnctl startall

Forms Process (FRMWEB) Consumes 100% of CPU in Oracle Applications R12

On Oracle Applications R12 when checking the top processes on the OS level for the middle tier, you find that forms process (frmweb) almost consumes 100% of the CPU. The root cause of the issue is that returning rows from LOVs in core forms causes the forms process to grow up into memory depending on the number of rows returned.

When an end user login to forms and start working with LOV within core forms sometimes and according to the search criteria that the user will provide to filter the results in LOV, it may fetch huge numbers of records in which causes the frmweb process to grow very large, and in extreme cases this can even lock up the current process or even the whole machine.

So when executing a LOV query, every row is fetched into memory on the middle tier, the frmweb process can get extremely large, and the larger it gets the more likely it is to start paging. Eventually it starts consuming excessive CPU just paging the process in and out of memory, which is probably what you can see here in this case as the amount of memory consumed when the LOV records are fetched into memory obviously depends on the amount of data in each record.

Solution
To implement the solution, please execute the following steps:

1. Stop all services on the middle tier.

2. Set following forms environment variables:

FORMS_RECORD_GROUP_MAX to 10000 or if that proves too restrictive, increase it to 20000 or 30000.
FORMS_CATCHTERM=0

In order to set the above forms variables so next time autoconfig run does not override those values, do the following steps :

1- For Forms Variable "FORMS_CATCHTERM" the context vairable name is: "s_forms_catchterm" and you can update the context file located in ($INST_TOP/appl/admin/)

2- For other forms variable "FORMS_RECORD_GROUP_MAX" there is no variable defined in Autoconfig for that one and have to customize the autoconfig for the forms variables to set that environment as following:

  a) Go to the autoconfig Template folder:
$cd $AD_TOP/admin/template
  b) Create new directory named (custom)
$ mkdir custom
  c) Make sure that new directory has same file permissions as ($AD_TOP/admin/template)
  d) Copy the following autoconfig template to the new custom directory:
$cp $AD_TOP/admin/template/APPLSYS_ux.env $AD_TOP/admin/template/custom/APPLSYS_ux.env
  e) Edit the file copied file under custom directory and add the following 2 lines at the end of section:

####################################
# Oracle Forms environment variables
####################################

FORMS_RECORD_GROUP_MAX=10000
export FORMS_RECORD_GROUP_MAX

  f) Save and exit from the file.
  g) Next time autoconfig run, it will read the custom directory and check for any customizations there.

3. Run Autoconfig on the middle tier and make sure it is completed successfully.

4. Startup all services.

5. Monitor the forms process to see its CPU usage, and you will see that form process usage is reduced and not causing any more CPU consumption up to 100% as before.

6. Migrate the solution as appropriate to other environments.

dbnerdz ....