Debugging Guide for Reports Issues in Oracle E-Business Suite Release 12 (Doc ID 1669163.1)

Section 1: Overview

Oracle Forms and Reports 10.1.2.x as used by Oracle E-Business Suite Release 12 are components of Oracle Application Server 10.1.2. The latest version in Oracle E-Business Suite Release 12 is Oracle Reports version 10.1.2.3.

In Oracle E-Business Suite Release 12, all reports are submitted online through the concurrent manager, as there is no standalone reports server.

When a customer submits a report through the concurrent request screen, the $INST_TOP/ora/10.1.2/bin/appsrwrun.sh script is called. This script sets Oracle Application specific settings required for running the report, executes $ORACLE_HOME/bin/reports.sh to set the reports environment, and then calls 10.1.2 $ORACLE_HOME/bin/rwrun to execute the report.

Reports (rdf files) are placed under $PROD_TOP/reports/US in Release 12. The Oracle Reports user exits available in Oracle E-Business Suite are found in $FND_TOP/bin/fndrwxit.so (or fndrwxit.sl).

In UNIX, we have an environment variable REPORTS_DEFAULT_DISPLAY (defined under 10.1.2 $ORACLE_HOME/bin/reports.sh). This tells reports to use DISPLAY of a third party like XVFB,VNC or to use headless operations available in JDK on 10.1.2 ORACLE_HOME. 

REPORTS_DEFAULT_DISPLAY=YES means DISPLAY is not needed and the report is able to use headless operations from JDK for operations that need graphical or image generation work.
REPORTS_DEFAULT_DISPLAY=NO means DISPLAY is needed and the report depends on the DISPLAY parameter set correctly to the third party XVNC/XVBF displays. REPORTS_DEFAULT_DISPLAY=NO is set for AIX only.

REPORTS_SERVER_CONFDIR is used to specify where the rwbuilder.conf file will reside. This can be set to a directory that the user running the reports can write. The rwbuilder.conf file will be copied into this directory and used by the reports executables. This variable is set to $INST_TOP/ora/10.1.2/reports/conf in APPS instance.

Section 2: How to Debug Report Issues

2.1 Steps to Run a Report from Command Line
2.2 Further Debugging
When you receive a report issue, you must first assess whether the issue stems from the reports layer or is a concurrent program issue. To figure this out, you need to run a report from the command line.

2.1 Steps to Run a Report from Command Line

Take the following steps to run a report from the command line.

1. Enable trace in the report so the command used to submit it will appear in the log file.
   
   
   1. Log in to Oracle Applications as a user who has system administrator responsibilities.
      
      
   2. As system administrator, navigate to the Define Concurrent Program Screen and query the report.
      
      
   3. Check the box next to Enable Trace.
      
      
   4. Save the record.
2. Switch responsibilities and submit the concurrent request to run the report.
   
   
3. You will get full reports command in concurrent program log file.
   
   
4. Log into the middle tier and source the environment.
   
   
5. Now run the command from the command line after adding userid=userid/password with this command.

If you see that the issue is reproduced after the completion of step 5, then it is confirmed that the issue is within the reports layer. If this is the case, you must check whether all patches listed in My Oracle Support Knowledge Document 437878.1, Upgrading OracleAS 10g Forms and Reports in Oracle E-Business Suite Release 12, have been properly applied in the customer's instance. You can verify this using the opatch lsinventory command.

2.2 Further Debugging

If all patches have been properly applied in the customer's instance and the customer is still receiving this error, it may be a new issue. Review the following points for further debugging:

2.2.1:df -kh output of $REPORTS_TMP


• As the reports temp file is created here, check to make sure enough space is available for reports temp file creation in this directory.
2.2.2:df -kh output of $INST_TOP/logs/ora/10.1.2/reports/cache


• The reports cache file is created in this directory. Check $INST_TOP/ora/10.1.2/reports/conf/rwbuilder.conf file for REPORTS_CACHE directory path. Enough space should be available for reports cache file creation in this directory.
2.2.3:If the core file has been generated by the failed reports process, you need to review the stacktrace generated from the core file.


• Please follow My Oracle Support Knowledge Document 1812.1, TECH: Getting a Stack Trace from a CORE file on Unix, for generating stacktrace from reports core file.
2.2.4:Check output of ls -ltr 10.1.2 $ORACLE_HOME/bin/reports.sh. You must verify whether customer has the latest reports.sh under $ORACLE_HOME/bin. We have seen some report issues where the customer has an old reports.sh file in 10.1.2 $ORACLE_HOME/bin which will set the wrong LD_LIBRARY_PATH.

2.2.5:Review the strace output of reports command. You can take strace output of reports command with the following command (you need to run the report from command line here):

strace -o reports .out -f <entire reports command with userid/password>
2.2.6:Excessive amounts of the srw.message included in the rdf file can cause signal 6/11 or out of memory error or performance issues in reports. These messages must be removed from the rdf file.

2.2.7:Customers often cite a crash when they are running reports containing a large volume of data. Currently, there is no limitation on the reports temp file size and output file size, provided that the customer has applied all reports one-off patches from My Oracle Support Knowledge Document 437878.1, Upgrading OracleAS 10g Forms and Reports in Oracle E-Business Suite Release 12, and CA_FILEIO_64 is defined to YES in the customer's instance (already included in template fix 9413844).


• Verify whether all reports one-off patches have been applied in the customer's instance and all post-installation steps are executed properly if they are still receiving errors when dealing with a large volume of data.
  
  You can also check the size of reports temp and cache file with the following commands in Linux (during reports execution).
  lsof|grep <reports process id>|grep dat
lsof|grep <reports process id>|grep cache)
  You may also need to review the strace output of report command.
   
2.2.8:Reports may consume 100% CPU or may run for 2-3 days and never finish or crashes in function 'rwfpngposition' when number of pages in report output exceeds 65535 and the report uses page numbering format "m of N". There is no fix for this issue yet. Respective product teams need to change their page numbering format to "m" to avoid such issues. The customer may be hit by this also while running a report with a large volume of data. We need to contact owner of the rdf file to modify the page numbering format.

2.2.9:You can increase the heap size in jvmOptions in $INST_TOP/ora/10.1.2/reports/conf/rwbuilder.conf to check the reports execution with more jvm memory.

Change the following lines:
<engine id="rwEng" class="oracle.reports.engine.EngineImpl"
initEngine="1" maxEngine="1" minEngine="0" engLife="50" maxIdle="30"
callbackTimeOut="60000">
to:

<engine id="rwEng" class="oracle.reports.engine.EngineImpl" initEngine="1"
maxEngine="1" minEngine="0" engLife="50" maxIdle="30"
callbackTimeOut="60000"
jvmOptions="-Xms512m -Xmx512m">
You should suggest this to your customer when report fails with the error -java.lang.OutOfMemoryError.

2.2.10:You may need to review the reports trace for debugging a report issue. Enabling reports tracing generates a text file that describes the series of steps completed during the execution of the reports. Tracing can be set to capture all events or just specific types of events.

Under the system administrator responsibility, navigate to the Define Concurrent Programs form.

From here, click on Concurrent -> Program -> Define.
Query the report by entering the report name in the Program field.

Check 'Enable Trace' in the 'Request' section. Enter your trace options in the 'Options' field in the 'Executable' section using the format:

TRACEOPTS=< >TRACEFILE=< > TRACEMODE=< >

The following table is a list of possible trace options:



Trace Options	Description
TRACE_ALL	Log all possible trace information in the trace file (default option).
TRACE_APP	Log trace information on all the reports objects in the trace file.
TRACE_BRK	List all breakpoints in the trace file.
TRACE_DBG	Log debug information.
TRACE_DST	List distribution lists in the trace file.
TRACE_ERR	List error messages and warnings in the trace file.
TRACE_EXC	List Reports Server exceptions.
TRACE_INF	Dumps any information not covered by the other options.
TRACE_LOG	Duplicate log information in your trace file.
TRACE_PLS	Log trace information on all the PL/SQL objects in the trace file.
TRACE_PRF	Log performance statistics in the trace file.
TRACE_SQL	Log trace information on all the SQL in the trace file (includes binds).
TRACE_STA	Provide server and engine state information.
TRACE_TMS	Enter a timestamp for each entry in the trace file.
TRACE_WRN	List server warning messages.
TRACEOPTS=(TRACE_APP, TRACE_PRF)	To use multiple options, list options in parentheses.
TRACEFILE=Any valid file name including the full path to the file	Specify the report trace file name.
TRACEMODE=TRACE_APPEND	Add the new information to the end of the file.
TRACEMODE=TRACE_REPLACE	Overwrite the file.
TRACEOPTS=< >TRACEFILE=< > TRACEMODE=<>	Collect reports trace file by adding tracing options to the end of the reports command line.


 
Note: You will receive the following exception in the reports trace file:

Exception 50125 (org.omg.CORBA.OBJ_ADAPTER: vmcid: SUN minor code: 202 completed: Maybe
at com.sun.corba.se.internal.corba.ORB.disconnect(Unknown Source)
at oracle.reports.server.ConnectionManager.release(ConnectionManager.java:183)
at oracle.reports.server.ConnectionImpl.disconnect(ConnectionImpl.java:660)
at oracle.reports.client.ReportRunner.releaseConnection(ReportRunner.java:333)
at oracle.reports.definition.RWServerInProcess.release(RWServerInProcess.java:368)
at oracle.reports.definition.RWServerInProcess.showProgress(RWServerInProcess.java:360)
at oracle.reports.definition.RWServerInProcess.run(RWServerInProcess.java:198)
): Internal error org.omg.CORBA.OBJ_ADAPTER: vmcid: SUN minor code: 202 completed: Maybe
This is not an error message. This exception is found in all reports trace files. This can be ignored while analyzing the report trace files.You can also apply Patch 5659594 in 10.1.2 ORACLE_HOME.With this patch you will not see these warning messages in reports trace file.

2.2.11:If you want to check the values of environment variables during reports execution, you need to run modify appsrwrun.sh. To do so, add env> /tmp/envvalues.out just before -exec $ORACLE_HOME/bin/rwrun "$@" command in $INST_TOP/ora/10.1.2/bin/appsrwrun.sh.

You can also modify appsrwrun.sh to take strace output of the report command if your customer does not feel comfortable with running the report from command line or if the report is executed from some product executable file. For that you need to modify appsrwrun.sh with the following:
exec strace -o /tmp/strace.out -f $ORACLE_HOME/bin/rwrun "$@"
2.2.12:If you want to generate DB trace for reports process, complete the following steps:


1. Under the system administrator responsibility, navigate to the Define Concurrent Programs form (Concurrent -> Program -> Define).
   
   
2. Query the reports by entering the reports name in the program field.
   
   
3. Check 'Enable Trace' in the request section and save the record.
   
   
4. Submit the concurrent request to run the reports. DB trace file will be generated under USER_DUMP_DEST. Check the trace file with the string *CR<concurrent_request_id>* in this directory (for example, -txkr12d2_ora_17752_SYSADMIN_CR40362585.trc). You can run tkprof TRACE_FILE_NAME to generate the tkprof output from this trace file.

2.2.13:JVMCI161: FATAL ERROR in native method error in AIX


• Customer may get -JVMCI161: FATAL ERROR in native method :Wrong method ID used to invoke a Java method error in AIX while executing report.Customer needs to upgrade jdk to 1.6/1.7 in 10.1.2 ORACLE_HOME for this issue.
Note: Concerning the rwconverter issue, note that if you need to try manual recompilation while debugging rwconverter issue, you need to set DE_DISABLE_PLS_512=0 before executing rwconverter. During compilation through adadmin, this variable is set by ad scripts. During reports execution through the concurrent program, appsrwrun.sh takes care of this environment variable.

Section 3: Known Issues

• Reports-idleTimeout is set to 90000
  Long running reports-idleTimeout is set to 90000 in $INST_TOP/ora/10.1.2/reports/conf/rwbuilder.conf, allowing you to run reports that takes up to 90000 minutes.
   
• Cache Directory Cleaning
  In Oracle E-Business Suite, report cache files are cleared automatically after report execution is completed if <property name="cacheSize" value="0"/> and <property name="noVoidedOutputError" value="yes"/> are in rwbuilder.conf and if Patch 14374587 is applied in 10.1.2 ORACLE_HOME.