Tuesday, November 17, 2015

ORA-01017: invalid username/password; logon denied on oracle.apps.fnd.odf2.FndXdfCmp

Issue:
Invoking Utility FndXdfCmp ...
Class: oracle.apps.fnd.odf2.FndXdfCmp
Method: applyXDF
Arguments: system &systempwd &un_apps &pw_apps &jdbc_protocol &jdbc_db_addr type &fullpath_cz_patch/115/xdf_CZ_MODEL_NODE_OBJ_TYPE.xdf &fullpath_fnd_patch/115/xdf_xsl



Solution:

Patch 17839156 Gives Error ORA-01017: invalid username/password; logon denied on oracle.apps.fnd.odf2.FndXdfCmp (Doc ID 1623924.1)

SQL> alter system set sec_case_sensitive_logon=false;

Tuesday, October 20, 2015

Backup of Weblogic Domain: pack command

 /u01/app01/Oracle/Middleware/user_projects/domains/demo_domain[oraapp@weblogic bin]$ pwd
/u01/app01/Oracle/Middleware/wlserver_10.3/common/bin

[oraapp@weblogic bin]$ ./pack.sh
-domain="/u01/app01/Oracle/Middleware/user_projects/domains/demo_domain"  -template="/u01/bkp/apps_bkp/17OCT15/demo_domain.jar"
-template_name="WebCenter Content Domain"
-log=/u01/bkp/apps_bkp/17OCT15/demo_domain.log
-log_priority=INFO
<< read domain from "/u01/app01/Oracle/Middleware/user_projects/domains/demo_domain"
>>  succeed: read domain from "/u01/app01/Oracle/Middleware/user_projects/domains/demo_domain"
<< write template to "/u01/bkp/apps_bkp/17OCT15/demo_domain.jar"
....................................................................................................
>>  succeed: write template to "/u01/bkp/apps_bkp/17OCT15/demo_domain.jar"
<< close template
>>  succeed: close template
[oraapp@weblogic bin]$
 

RMAN: Cold Backup Script

#!/bin/sh
# Set Oracle Environment for DB
#. ~/.bashrc
. ~/.bash_profile

echo
echo "`date` – Started `basename $0`"
echo

export NLS_DATE_FORMAT='DD-MON-YY HH24:MI:SS'
rman target / << EOF
spool log to rman_coldbackup.log;
set echo on
run
{
shutdown immediate;
startup mount;
allocate channel ch1 type disk format '/export/dbclone/DEV_RMAN_COLDBKP/full_backup_%d_%s_%p';
allocate channel ch2 type disk format '/export/dbclone/DEV_RMAN_COLDBKP/full_backup_%d_%s_%p';
allocate channel ch3 type disk format '/export/dbclone/DEV_RMAN_COLDBKP/full_backup_%d_%s_%p';
allocate channel ch4 type disk format '/export/dbclone/DEV_RMAN_COLDBKP/full_backup_%d_%s_%p';
allocate channel ch5 type disk format '/export/dbclone/DEV_RMAN_COLDBKP/full_backup_%d_%s_%p';
allocate channel ch6 type disk format '/export/dbclone/DEV_RMAN_COLDBKP/full_backup_%d_%s_%p';

BACKUP CURRENT CONTROLFILE format '/export/dbclone/DEV_RMAN_COLDBKP/ctrl_file_%d_%s_%p';
BACKUP AS COMPRESSED BACKUPSET DATABASE;

release channel ch1;
release channel ch2;
release channel ch3;
release channel ch4;
release channel ch5;
release channel ch6;

alter database open;
}
EOF
if [ $? -eq 0 ]
then
echo "====================="
echo "RMAN Backup Completed"
echo "====================="
else
echo "=================="
echo "RMAN Backup Failed"
echo "=================="
exit 1
fi

echo
echo "`date` - Finished `basename $0`"
echo

Monday, October 19, 2015

Enable Tracing in EBS: "Initialization SQL Statement – Custom"

A user event trace is very handy for tracing sql operations to debug various issues. The benefit of a user event trace is that it is linked to a specific user so that only code run by this user is traced. This makes diagnosis easier when compared to similar tracing methods at the database level where all user calls are traced.

Step 1
Login to Oracle Applications and select the System Administrator responsibility.
Choose Profile - System


Step 2
In the find profile field select the user which you wish to trace. In the profile field enter 'Initialization SQL Statement - Custom"
Select find


Step 3
In the find profile results form copy and paste the following into the 'user' field. Do not update the site level field.

BEGIN FND_CTL.FND_SESS_CTL('','', '', 'TRUE','','ALTER SESSION SET TRACEFILE_IDENTIFIER='||''''||'USERID' ||''''||' EVENTS ='||''''||' 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 '||''''); END;

Note: copy and paste the above as one line. Failure to paste the values properly or pasting the values with incorrect syntax will result in this user not being able to login.


Step 4 (Optional)
Save the profile option change. Stop and start the java virtual machine for the change to take effect
11i: use adapcctl.sh
r12: use adoacorectl.sh


Step 5
Login to the application and reproduce the issue. Then quickly log off. Try and avoid any un-necessary keystrokes as this simply makes the log files larger and the issue hard to pinpoint in the logs.


Step 6
login to unix/windows as the oracle user. Navigate to the user dump destination which us usually $ORACLE_HOME/admin/<context>/udump
You will see a series of trace files generated during the time of your issue reproduction. Tkprof all of the generated files and upload both the raw and tkprof files to your service request.

You can also check the user dump destination via the following:
sql> show parameter user_dump_dest;



Notes:

When using this profile, DO NOT turn trace on the menu, as doing so would actually turnoff Event 10046. Turning Event 10046 to any level, turns trace automatically.

Event 10046 level 1 is regular trace, level 4 is tracing with bind variables, level 8 is with database waits and level 12 with both,bind variables and database waits.

alter system set max_dump_file_size=UNLIMITED scope=both;

Monday, October 12, 2015

Java Keytool Essentials: Working with Java Keystores


Source: https://www.digitalocean.com/community/tutorials/java-keytool-essentials-working-with-java-keystores

Introduction

Java Keytool is a key and certificate management tool that is used to manipulate Java Keystores, and is included with Java. A Java Keystore is a container for authorization certificates or public key certificates, and is often used by Java-based applications for encryption, authentication, and serving over HTTPS. Its entries are protected by a keystore password. A keystore entry is identified by an alias, and it consists of keys and certificates that form a trust chain.
This cheat sheet-style guide provides a quick reference to keytool commands that are commonly useful when working with Java Keystores. This includes creating and modifying Java Keystores so they can be used with your Java applications.
How to Use This Guide:
  • If you are not familiar with certificate signing requests (CSRs), read the CSR section of our OpenSSL cheat sheet
  • This guide is in a simple, cheat sheet format--self-contained command line snippets
  • Jump to any section that is relevant to the task you are trying to complete (Hint: use the Contents menu on the bottom-left or your browser's Find function)
  • Most of the commands are one-liners that have been expanded to multiple lines (using the \ symbol) for clarity

Creating and Importing Keystore Entries

This section covers Java Keytool commands that are related to generating key pairs and certificates, and importing certificates.

Generate Keys in New/Existing Keystore

Use this method if you want to use HTTP (HTTP over TLS) to secure your Java application. This will create a new key pair in a new or existing Java Keystore, which can be used to create a CSR, and obtain an SSL certificate from a Certificate Authority.
This command generates a 2048-bit RSA key pair, under the specified alias (domain), in the specified keystore file (keystore.jks):
keytool -genkeypair \
        -alias domain \
        -keyalg RSA \
        -keystore keystore.jks
If the specified keystore does not already exist, it will be created after the requested information is supplied. This will prompt for the keystore password (new or existing), followed by a Distinguished Name prompt (for the private key), then the desired private key password.

Generate CSR For Existing Private Key

Use this method if you want to generate an CSR that you can send to a CA to request the issuance of a CA-signed SSL certificate. It requires that the keystore and alias already exist; you can use the previous command to ensure this.
This command creates a CSR (domain.csr) signed by the private key identified by the alias (domain) in the (keystore.jks) keystore:
keytool -certreq \
        -alias domain \
        -file domain.csr \
        -keystore keystore.jks
After entering the keystore's password, the CSR will be generated.

Import Signed/Root/Intermediate Certificate

Use this method if you want to import a signed certificate, e.g. a certificate signed by a CA, into your keystore; it must match the private key that exists in the specified alias. You may also use this same command to import root or intermediate certificates that your CA may require to complete a chain of trust. Simply specify a unique alias, such as root instead of domain, and the certificate that you want to import.
This command imports the certificate (domain.crt) into the keystore (keystore.jks), under the specified alias (domain). If you are importing a signed certificate, it must correspond to the private key in the specified alias:
keytool -importcert \
        -trustcacerts -file domain.crt \
        -alias domain \
        -keystore keystore.jks
You will be prompted for the keystore password, then for a confirmation of the import action.
Note: You may also use the command to import a CA's certificates into your Java truststore, which is typically located in $JAVA_HOME/jre/lib/security/cacerts assuming $JAVA_HOME is where your JRE or JDK is installed.

Generate Self-Signed Certificate in New/Existing Keystore

Use this command if you want to generate a self-signed certificate for your Java applications. This is actually the same command that is used to create a new key pair, but with the validity lifetime specified in days.
This command generates a 2048-bit RSA key pair, valid for 365 days, under the specified alias (domain), in the specified keystore file (keystore.jks):
keytool -genkey \
        -alias domain \
        -keyalg RSA \
        -validity 365 \
        -keystore keystore.jks
If the specified keystore does not already exist, it will be created after the requested information is supplied. This will prompt for the keystore password (new or existing), followed by a Distinguished Name prompt (for the private key), then the desired private key password.

Viewing Keystore Entries

This section covers listing the contents of a Java Keystore, such as viewing certificate information or exporting certificates.

List Keystore Certificate Fingerprints

This command lists the SHA fingerprints of all of the certificates in the keystore (keystore.jks), under their respective aliases:
keytool -list \
        -keystore keystore.jks
You will be prompted for the keystore's password. You may also restrict the output to a specific alias by using the -alias domain option, where "domain" is the alias name.

List Verbose Keystore Contents

This command lists verbose information about the entries a keystore (keystore.jks) contains, including certificate chain length, fingerprint of certificates in the chain, distinguished names, serial number, and creation/expiration date, under their respective aliases:
keytool -list -v \
        -keystore keystore.jks
You will be prompted for the keystore's password. You may also restrict the output to a specific alias by using the -alias domain option, where "domain" is the alias name.
Note: You may also use this command to view which certificates are in your Java truststore, which is typically located in $JAVA_HOME/jre/lib/security/cacerts assuming $JAVA_HOME is where your JRE or JDK is installed.

Use Keytool to View Certificate Information

This command prints verbose information about a certificate file (certificate.crt), including its fingerprints, distinguished name of owner and issuer, and the time period of its validity:
keytool -printcert \
        -file domain.crt
You will be prompted for the keystore password.

Export Certificate

This command exports a binary DER-encoded certificate (domain.der), that is associated with the alias (domain), in the keystore (keystore.jks):
keytool -exportcert
        -alias domain
        -file domain.der
        -keystore keystore.jks
You will be prompted for the keystore password. If you want to convert the DER-encoded certificate to PEM-encoding, follow our OpenSSL cheat sheet.

Modifying Keystore

This section covers the modification of Java Keystore entries, such as deleting or renaming aliases.

Change Keystore Password

This command is used to change the password of a keystore (keystore.jks):
keytool -storepasswd \
        -keystore keystore.jks
You will be prompted for the current password, then the new password. You may also specify the new password in the command by using the -new newpass option, where "newpass" is the password.

Delete Alias

This command is used to delete an alias (domain) in a keystore (keystore.jks):
keytool -delete \
        -alias domain \
        -keystore keystore.jks
You will be prompted for the keystore password.

Rename Alias

This command will rename the alias (domain) to the destination alias (newdomain) in the keystore (keystore.jks):
keytool -changealias \
        -alias domain \
        -destalias newdomain \
        -keystore keystore.jks
You will be prompted for the keystore password.

Conclusion

That should cover how most people use Java Keytool to manipulate their Java Keystores. It has many other uses that were not covered here, so feel free to ask or suggest other uses in the comments.

Sunday, October 11, 2015

Configuring SSL for Oracle Weblogic (and OFMW)

https://jvzoggel.wordpress.com/2011/12/16/configuring-ssl-for-oracle-weblogic-and-ofmw/


We have a very basic domain: 3 physical servers, 1 containing the admin server, 2 containing 2 managed servers each, running in a cluster. Nice picture to follow. :)
name of the servers: server01 (Admin) / server02 (MS1+MS2) / server03 (MS3+MS4)

Step 0 – The Weblogic Demo Certificates (the reason why)

If you ever checked your Weblogic production logs and found the Security Alert below, then your Weblogic domain has a huge gaping security hole. Even if you don’t want SSL communication for your webservices or applications. The internal (administrative) processes in your Weblogic domain still relies on the default demotrust and with this everyone can access your domain.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
</pre>
&nbsp;
 
<Alert> <Security> <SERVER02> <rbx_dev_soa_ms01> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <BEA-090152> <Demo trusted CA certificate is being used in production mode: [
[
  Version: V3
  Subject: CN=CACERT, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
  Signature Algorithm: MD5withRSA, OID = 1.2.840.113549.1.1.4
 
  Key:  Sun RSA public key, 512 bits
  modulus: 95501928778692415440753....
  public exponent: 65537
  Validity: [From: Thu Mar 21 21:12:27 CET 2002,
               To: Tue Mar 22 21:12:27 CET 2022]
  Issuer: CN=CACERT, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
  SerialNumber: [    33f10648 fcde0deb 4199921f d64537f4]
 
Certificate Extensions: 1
[1]: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  Key_CertSign
]
 
]
  Algorithm: [MD5withRSA]
  Signature:
0000: 9D 26 4C 29 C8 91 C3 A7   06 C3 24 6F AE B4 F8 82  .&L)......$o....
0010: 80 4D AA CB 7C 79 46 84   81 C4 66 95 F4 1E D8 C4  .M...yF...f.....
0020: E9 B7 D9 7C E2 23 33 A4   B7 21 E0 AA 54 2B 4A FF  .....#3..!..T+J.
0030: CB 21 20 88 81 21 DB AC   90 54 D8 7D 79 63 23 3C  .! ..!...T..yc#<
 
] The system is vulnerable to security attacks, since it trusts certificates signed by the demo trusted CA.>

Step 1 – generate the identity java keystore

We first generate a java keystores containing a private key for each physical server, since SSL hostname verifiation is based on machine names. We use “welcome1” as password here, but make sure to choose a complex password here, use it during these steps and then lock it safely away.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
keytool.exe -genkey -keysize 2048 -keyalg RSA -alias server01 -keystore server01_identity.jks
 
Enter keystore password: welcome1
 Re-enter new password: welcome1
 What is your first and last name?
   [Unknown]:  server01
 What is the name of your organizational unit?
   [Unknown]:
 What is the name of your organization?
   [Unknown]:  Rubix
 What is the name of your City or Locality?
   [Unknown]:
 What is the name of your State or Province?
   [Unknown]:
 What is the two-letter country code for this unit?
   [Unknown]:  NL
 Is CN=server01, OU=, O=Rubix, L=, ST=, C=NL correct?
   [no]:  yes
<br>

Step 2 – generate signing request

1
keytool.exe -certreq -alias server01 -file server01.csr -keystore server01_identity.jks</div>
Repeat step 1 + 2 for each physical server, replacing the -alias parameter, -keystore parameter and CN name with the correct server0* value.
 

 

Step 3 – the Certificate Authority (CA)

Now comes the tricky part that you probably need some outside forces. You will need the correct Certificate Authority to sign your certificate requests (the .CSR files you generated). You can create your own CA and self-sign them, but I would not recommend that unless you know what you are doing and understand the consequences. You could use an external Internet CA provider, but this becomes very costly and time consuming when you need such an external provider to sign every SSL enabled server in your landscape. The best situation for us would be if the current organization already has an internal CA provider, especially when the rootCA is trusted by the servers and machines in your landscape.
 
Let’s assume however that you receive the signed response, including the public keys of the rootCA and a intermediateCA. CA instances often use a intermediate CA to authorize certificates, so if they have “problems” they can withdraw this intermediate CA and don’t need to withdraw the whole RootCA making it useless.
So the situation should be that you know hold 3 PEM/DER files.
 

 

Step 4 – importing the CA response

Import the certificates in your keystore, starting with the rootCA, then the intermediateCA, then the specific server alias. Example for server01:
1
2
3
4
5
keytool -importcert -trustcacerts -alias rootca -file rootCA.cer -keystore server01_identity.jks
 
keytool -importcert -alias intermediateca -file intermediateCA.cer -keystore server01_identity.jks
 
keytool -importcert -alias server01 -file server01.cer -keystore server01_identity.jks
Repeat these steps for server02 and server03. So we know have 3 identity keystores for each server containing the signed private key and it’s certificate chain.
 

 

Step 5 – generate the trust java keystore

Now let’s generate a truststore for each server. Altough you can use the identical keystore for both identity and trust, it’s always a good practice to separate them. The identity keystone contains the private key and normally never changes during it’s lifetime, so you should secure it with a complex password. The trust store contains the certificates that you trust and as your middleware landscape grows, this file might grow and more administrators will need access to it.
 
Example generating the truststore for server01, repeat for 02 & 03.
 
1
2
keytool -importcert -trustcacerts -alias rootca -file rootCA.cer -keystore server01_trust.jks
keytool -importcert -alias intermediateca -file intermediateCA.cer -keystore server01_trust.jks
 

Step 6 – copy the java keystore files to each server

 Make sure that each server had the files stored on disk. We use the windows server example “c:\oracle\security” in this blogpost. So server01 should hold:
– c:\oracle\security\server01_identity.jks
– c:\oracle\security\server01_trust.jks
 

 

Step 7 – configure the Node Manager for SSL

The nodemanger by default uses the demoIdentity and demoTrust keystores.
So on each server hosting a managed server (server02 and server03) we will need to change this.
Open the nodemanager.properties and add the following lines:
 
1
2
3
4
5
6
7
8
9
### SSL config
KeyStores=CustomIdentityAndCustomTrust
# CustomIdentityKeyStoreType=JKS
CustomIdentityKeyStoreFileName=c://oracle//security//server01_identity.jks
CustomIdentityKeyStorePassPhrase=welcome1
CustomIdentityAlias=server01
CustomIdentityPrivateKeyPassPhrase=welcome1
# CustomTrustKeyStoreType=JKS
CustomTrustKeyStoreFileName=c://oracle//security//server01_trust.jks
Restart the NodeManager to activate the changes. The passwords in the properties file will automatically become encrypted.
 
Possible error:
1
For server rbx_dev_soa_ms03, the Node Manager associated with machine server03 is not reachable. All of the servers selected are currently in a state which is incompatible with this operation or are not associated with a running Node Manager or you are not authorized to perform the action requested. No action will be performed.
Possible solution:
Your AdminServer can not communicate with a node manager. Make sure it’s running, check Environment -> Machines in the Weblogic console if the machine has state reachable, check if the address configured there, and the address in the nodemanager.properties and the CN-name used in SSL have the same value, check if Weblogic and Nodemanager use the same SSL configuration (trust+identity).
 

 

Step 8 – configure Weblogic for SSL

Log in the Weblogic console, in our example: http://server01:7001/console
 
 
Step 8.1 – configure Weblogic for SSL, enable SSL
 
Log in the Weblogic console (in our case http://server01:7001/console) and make sure that each server has the SSL port enabled. You can find the setting “SSL Listen Port Enabled” for each server under “Configuration -> General”. Enable it and make sure you use a correct and free port.
 

Step 8.2 – configure Weblogic for SSL, changing keystores

For each server (Admin + MS) we have to change the KeyStore configuration.
So for each server go to: Configuration -> Keystores
 
  • Keystores -> “Change” = “Custom Identity and Custom Trust”
  • Custom Identity Keystore = c:/Oracle/security/serverXX_identity.jks
  • Custom Identity Keystore Type = JKS
  • Custom Identity Keystore Passphrase = welcome1
  • Custom Trust keystore = c:/Oracle/security/serverXX_trust.jks
  • Custom Trust Keystore Type = JKS
  • Custom Trust Keystore Passphrase = welcome1
 
 

Step 8.3 – configure Weblogic for SSL, changing identity:

For each server (Admin + MS) we have to change the SSL configuration.
So for each server go to: Configuration -> SSL
 
  • Private Key Alias = serverXX
  • Private Key Passphrase = welcome1
 

Step 9 – Weblogic start scripts:

By default Weblogic uses the DemoTrust.jks truststore. Since we changed this in the console configuration you would hope this would be enough. However the SetDomainEnv.sh / SetDomainEnv.cmd script still contains a reference. So for each server:
 
  • Go to %domain_home%/bin/
  • Open the SetDomainEnv script
  • Search the argument: -Djavax.net.ssl.trustStore=
  • Make sure the value is not referring to the demo trust.jks but “c:/oracle/security/serverXX_trust.jks
When you are finished restart your whole Weblogic environment (Admin + MS).
 

Step 10 – Configure Enterprise Manager:

This took me some research and the Oracle Forums to get it working. Navigate to the enterprise manager console (/em)
 

Step 10.1 – Security Provider Configuration:

I’m actually not really sure if this step is still necessary in 11g, but hey .. why not be sure:
 
  • Click Weblogic -> right-click domain
  • Select Security -> Security Provider Configuration -> KeyStore Configure
  • Keystore Path = c:/oracle/security/server01_identity.jks
 

Step 10.2 – SOA Infra properties

 
  • Click SOA -> right-click soa-infra
  • Select SOA Administration -> Common Properties
  • Click More SOA Infra Advanced Properties
  • Keystore Location = c:/oracle/security/serverXX_identity.jks

Step 10.3 – Credential Mapping:
The next step is to configure the credential mapping keys

  • Click Weblogic -> right-click domain name
  • Select Security -> Credentials
  • Create a Map named “SOA”
  • Click Create a key
  • Key = KeyPassword
  • User Name = KeyPassword
  • Password = welcome1
  • Click Create Key:
  • Key = KeyStorePassword
  • User Name = KeyStorePassword
  • Password = welcome1
 
Possible error:
1
2
3
4
5
6
7
8
9
10
11
</div>
<div>
 
<Warning> <oracle.as.jmx.framework.MessageLocalizationHelper> <J2EE JMX-46041> <The resource for bundle "oracle.soa.management.config.bpel.mbeans.MessageBundle_en" with key "oracle.soa.BPELConfigBean.auditFlushByteThreshold" cannot be found.>
INFO: SSLSocketFactoryManagerImpl.getKeystoreLocation SOA Keystore location: c:/oracle/security/server04_identity.jks
INFO: SSLSocketFactoryManagerImpl.getKeystorePassword Obtained null or empty keystore password
INFO: SSLSocketFactoryManagerImpl.getKeyPassword Obtained null or empty key password
INFO: SSLSocketFactoryManagerImpl.getSSLSocketFactory Could not obtain keystore location or password
 
</div>
<div>
 
Possible solution:
You did not configure the SOA key mapping correctly as mentioned above in 9.3

 

Conclusion:

Enabling SSL takes a lot of steps, and we didn’t even mentioned it for enabling SSL client identity in OSB (Service Key Provider) and communication with an LDAP over SSL. But there’s more to come I guess.
If you have any remarks please let me know in the comments. Hope it helps. :)
 
 

Links:

WebLogic Server - identity vs trust keystores

Source:  http://one-size-doesnt-fit-all.blogspot.com/2009/09/weblogic-server-identity-vs-trust.html

Summary:
Identity keystore:
This will be used to store the server certificate(private key/digital certificate pairs). When the client contacts server the digital certificate presented in this keystore will be sent.

Trust Keystore:
This will contain all the certificates for the trusted partners(ie partners). When server connects with the partner it will use this key store.


In computing most technologies have lots of terms and acronyms to learn, it's par for the course, you get used to it. However in computer security the frustration is multiplied as there are often many different terms that mean the same thing. It makes implementing security hard, because understanding it is hard, and I'm not surprised why security is considered badly implemented because the average Joe will struggle.

I've been trying recently to get straight in my head what is stored in the WLS identity and trust keystores, and what the difference between identity and trust is anyhow. Thanks to kind assistance from Gerard Davison, I think I can now post my understandings, and as usual, hopefully the post is helpful to other readers. As noted however security to me is a difficult area, and so be sure to check the facts here, your mileage with this post may vary.

The following WLS documentation attempts to explain the concepts of identity and trust:

http://download.oracle.com/docs/cd/E12839_01/web.1111/e13707/identity_trust.htm#i1170342


...in ripping out one of the core paragraphs, with a slight rewrite of my own we can see the concept of identity, and how it relates to the public and private keys:

"The public key is embedded in a digital certificate with additional information describing the owner of the public key, such as name, street address, and e-mail address *as well as the hostname*. *Along with this the digital certificate containing the public key, and the separate related private key, provide identity for the server*."

...ultimately to identify the server, to assert the server is who the server says it is.

The digital certificate containing the public key is also referred to as the "server certificate", as for example in 1-way-SSL traffic between the server and client, the server certificate containing the public key is what is initially passed to the client.

There is a missing piece in the puzzle. Regardless that the digital certificate states the owner of the public key, their name and so on, how does a client know that the "identity" asserted by the digital certificate is true? That's where Certificate Authorities (CAs) come in.

Ignoring self signed digital certificates, a typical digital certificate used on the internet containing the public key and owner details is signed by a trusted CA who has verified the identity of the owner. Presumably when purchasing digital certificates from CAs, this is what some of the cost covers, the CAs research into ensuring that the identity details embedded in the digital cert are actually true.

At runtime on receiving the digital certificate, the client checks the CA and if the CA is one that the client trusts (or a CA in a chain of trusted CAs), then the identity of the server is established/verified.

Thus the "identity" of the server is established by what's stored in the "identity" keystore, and its contents are what are farmed out to clients establishing secure connections with the server, who then verify the supplied digital certificate's CA against the clients own list of trusted CAs. The "identity keystore" is also referred to as the "server keystore", because it establishes the server's identity (ie. I am who I say I am).

WLS side note: As mentioned the digital certificate also includes the host name of the server, or in other words the digital certificate is pegged to that server and that server alone. This implies on that server with its relating digital certificate, *all* of the applications will share that single digital certificate for secure communications. Occasionally a requirement will arise where each application must have its own digital certificate. In WLS because keystores are configured under an individual WLS "managed server", if you have two separate applications, it is not possible to use separate digital certificates for each in one managed server. The solution is to create another managed server with its own keystores.

WLS web service side note: Following on from the previous side note, for web services that use in-message encryption and digital signatures, there is often the requirement for multiple different digital certificates. Under WLS to provision the WS-Security model, WLS has a separate Web Service Security Configuration (WSSC) to provision this setup.

Finally regarding the trust keystore, what is its job in all of this? The trust keystore is typically used for storing CA digital certificates, essentially the CAs who will be used to check any digital certificates that are given to the server at runtime (just the same as the client did above). In the standard 1-way-SSL between a client and the WLS server, the trust keystore doesn't come into the equation as the client has its own trust keystore (containing the CAs) and the server has nothing to verify. Yet in the case of mutual SSL (aka. 2 way SSL) between the client and server, the client and server actually swap each other digital certificates to establish identity of both parties, and in this case the server must be able to test the identity of the client through the CA of the client's digital certificate.

Mutual SSL side note: the setup of mutual SSL is more complicated than this. Readers are advised to refer to the following Oracle article.

Wednesday, July 8, 2015

Advanced Format: 4k Sector Size


For understanding of Advance Formating and other various terminologies like Emulation mode and Native mode, check this site: http://flashdba.com/4k-sector-size/




Tuesday, June 2, 2015

Common Tracing Techniques in Oracle E-Business Applications

Source:
FAQ: Common Tracing Techniques in Oracle E-Business Applications 11i and R12 (Doc ID 296559.1)
Why Can't Users Enable Forms Trace in 12.1.3 ? (Doc ID 1223753.1)
FAQ - How to Use Debug Tools and Scripts for the VCP (aka APS) and EBS Applications (Doc ID 245974.1)



Enable trace in the Oracle Application forms.

One can enable trace through the forms by using the Help menu, choosing the diagnostics menu, trace and then selecting the appropriate trace for your needs. Most commonly if debugging an error, you should at least provide trace with binds.  When debugging a performance issue, you may consider using trace with binds and waits.

For example, the following is the navigation to enable trace in a form:
1. Goto the Oracle Applications
2. Login
3. Open the form where the error occurs but do not yet cause the error.
4. Enable SQL*Trace by choosing Help > Diagnostics > Trace > Trace with binds
5. A message appears indicating that the trace will be recorded
6. Note the file name and location of the file
7. Now reproduce the error.
8. Once the error occurs, disable trace as soon as possible.
9. Disable SQL*Trace by choosing Help > Diagnostics > Trace > Trace off
10. The same file name and location will be noted in case you need it again.
11. Retrieve the trace file from the DB(user_dump_dest).



Enable trace for a Concurrent Program.

Checkbox

With system administration privileges, a simple way to enable trace in a concurrent program is to review the concurrent program definition and select trace enabled. This will enable trace without binds for all users that run the program.

For example, the following steps could be used.
1. Goto Sysadmin > Concurrent > Program > Define
2. Query the concurrent program
3. Check the trace box to enable trace

If you require bind variables for your trace file from a concurrent program, a more sophisticated method is required as noted below.

Debug Options Button! (Binds and Waits)
When submitting a concurrent program in R12.1 and higher, the concurrent program submission screen has a "Debug Options" button where you can enable trace. The button opens a window with various SQL Tracing options including Binds and Waits. The selection applies just to the current run of the program.



Enable trace for ALL user actions

How does one enable trace for all actions that occur for a user?
One can enable trace for all actions that a user takes by setting a profile option for the user. This will function when the user logs into the Oracle Applications forms or when logging into the Self Service Web Applications. This method uses an Event Trace.

The instructions for enabling the event trace follow:
1) First enable trace for a specific user:
- Bring up the Core application - System Administrator responsibility
- Move to Profile/System
- Check off the USER box - and enter your username that you are using when getting the error
- Search on the following profile option - 'Initialization SQL Statement - Custom'

Please set this at the user level with the following string:

BEGIN FND_CTL.FND_SESS_CTL('','', '', 'TRUE','','ALTER SESSION SET TRACEFILE_IDENTIFIER='||''''||'4269824.999' ||''''||' EVENTS ='||''''||' 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 '||''''); END; 


This must be one complete line of text - so may be best to copy the string into notepad prior to putting it into the profile.


2) Once the profile is set, please change responsibilities and directly go and reproduce the problem.
To locate the trace file, use the following sql in sqlplus:

SQL> select name, value from v$parameter where name like 'user_dump_dest'; 

- The value is the location of the trace file on the database server.

- This is the trace file created - please tkprof and upload BOTH the raw and tkprof trace file to My Oracle Support.

3) Ensure that the profile option is unset before exiting the Application.
This is a very important step. If this is not unset, unnecessary trace files will be generated.

4) Note that a similar technique can be used for enabling trace with binds for a user in SQL*Plus. The commands would look like the following:

SQL> ALTER SESSION SET TRACEFILE_IDENTIFIER='577' EVENTS ='10046 TRACE NAME CONTEXT FOREVER, LEVEL 12';
SQL> select 'x' from dual; -- or any SQL that you want to trace
SQL> alter session set sql_trace = false;


How To Generate and Retrieve Trace Files for SSA (Self Service Applications)

1.Have DBA or someone with APPS access to SQL run
    alter system set max_dump_file_size = 'unlimited';
    This will prevent trace being cutoff and not usable.


2.The profile FND: Diagnostics = Yes is required to have the Diagnostics button appear in Self Service menus.
    1.IF setup for the first time, THEN exit and sign on again to the SSA


3.Click on Diagnostics button (In 12.2 - click the GEAR icon and choose Diagnostics)
    1.Choose Set Trace Level from the drop down, the press Go
    2.Choose the trace level - Trace with BINDS (or for performance issue, then Trace with Binds and Waits)
    and press Save
    3.A single trace number will be shown… This is NOT usually the trace that is needed.
    4.As more steps are performed, more trace numbers will be written to this screen and they will be the ones that be for the functionality that is being tested


4.Perform the actions necessary in Self Service Applications - Note: Use the least number of steps possible!


5.Then click on Diagnostics button again
    1.Choose Set Trace Level from the drop down, the press Go
    2.Now there will be multiple trace file numbers showing on the screen.
    3.Write down the trace numbers and then click Disable Trace and press Save (the number will disappear when you press Save)


6.Have the DBA retrieve the trace files from the user_dump_dest of the database.





Using TKPROF

A trace file can be reviewed using TKPROF. TKPROF reformats the raw data so that it is easier to review. The TKPROF commands are normally run at the operating system command prompt.  This will be signified with a $ as this is a common prompt in UNIX.
1. Retrieve the trace file.
2. Issue a command like the following to create a TKPROF version of the trace file.  The explain option will look to see how each query is handled by the database in terms of what tables and indexes are referenced.

$ tkprof sys=no explain=apps/

However, TKPROF does not take into account bind variables. Therefore, these are ignored in the output. When you need to analyze bind variables, consider using TraceAnalyzer.
A handy technique when trouble shooting performance issues is to use TKPROF to look at the longest running queries.  Since trace files related to performance can be huge, one might spend hours looking through the results to find the queries causing the issue.  If you sort the file by the longest running queries first, it makes it much easier to investigate.  The following example sorts by longest running queries first with the sort options selected as "sort='(prsela,exeela,fchela)'" and limits the results to the "Top 10" long running queries with the "print=10" command:

$ tkprof sys=no explain=apps/ sort='(prsela,exeela,fchela)' print=10 
 

What is a trace file?

A trace file is a log of SQL run in a particular session or sessions focused on selects, inserts, updates, and deletes. A trace file can be used in many circumstances including reviewing performance, finding tables and views referenced, or finding the root of an error. More details about trace files can be read in the database user's guide related to your version.




What are bind variables?

A bind variable is a substitution value used within a trace file. To improve performance, the Oracle database will reuse SQL substituting values for variables as necessary. For example, the same SQL statement could be run for each
of your employees. The employee id might be a variable that is changed before you run each SQL. This could be translated to the database as a bind variable so that the same SQL can be used each time but the correct variable can be substituted just before the SQL is run.

If someone requests that you enable bind variables for your trace file, this causes the database to document the substitution variables within the trace file as well as the SQL run. For example, we may see that a select statement ran against the employee table ten times, but without the binds, we would not know what employee was selected each time.




Tuesday, May 5, 2015

Patching & Maintenance Advisor: E-Business Suite




Friday, April 17, 2015

WebLogic Server fails to start with SecurityServiceRuntimeException Error

WebLogic Server does not start, showing the following stack trace, soon after Domain creation:


weblogic=>./startWebLogic.sh
.
.
JAVA Memory arguments: -Xms256m -Xmx512m  -XX:MaxPermSize=256m
.
WLS Start Mode=Production
.
CLASSPATH=/app01/oracle/Middleware/patch_wls1036/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/app01/oracle/Middleware/patch_ocp371/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/app01/java/jdk1.6.0_43/lib/tools.jar:/app01/oracle/Middleware/wlserver_10.3/server/lib/weblogic_sp.jar:/app01/oracle/Middleware/wlserver_10.3/server/lib/weblogic.jar:/app01/oracle/Middleware/modules/features/weblogic.server.modules_10.3.6.0.jar:/app01/oracle/Middleware/wlserver_10.3/server/lib/webservices.jar:/app01/oracle/Middleware/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/app01/oracle/Middleware/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/app01/oracle/Middleware/wlserver_10.3/common/derby/lib/derbyclient.jar:/app01/oracle/Middleware/wlserver_10.3/server/lib/xqrl.jar
.
PATH=/app01/oracle/Middleware/wlserver_10.3/server/bin:/app01/oracle/Middleware/modules/org.apache.ant_1.7.1/bin:/app01/java/jdk1.6.0_43/jre/bin:/app01/java/jdk1.6.0_43/bin:/app01/oracle/Middleware/wlserver_10.3/server/bin:/app01/oracle/Middleware/modules/org.apache.ant_1.7.1/bin:/app01/java/jdk1.6.0_43/jre/bin:/app01/java/jdk1.6.0_43/bin:/app01/oracle/Middleware/wlserver_10.3/server/bin:/app01/oracle/Middleware/modules/org.apache.ant_1.7.1/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/weblogic/bin:/home/weblogic/bin:/home/weblogic/bin:/app01/java/jdk1.6.0_43
.
***************************************************
*  To start WebLogic Server, use a username and   *
*  password assigned to an admin-level user.  For *
*  server administration, use the WebLogic Server *
*  console at http://hostname:port/console        *
***************************************************
starting weblogic with Java version:
java version "1.6.0_43"
Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
Starting WLS with line:
/app01/java/jdk1.6.0_43/bin/java -server   -Xms256m -Xmx512m  -XX:MaxPermSize=256m -Dweblogic.Name=MedRecAdmServer -Djava.security.policy=/app01/oracle/Middleware/wlserver_10.3/server/lib/weblogic.policy  -Dweblogic.ProductionModeEnabled=true  -Xverify:none  -da -Dplatform.home=/app01/oracle/Middleware/wlserver_10.3 -Dwls.home=/app01/oracle/Middleware/wlserver_10.3/server -Dweblogic.home=/app01/oracle/Middleware/wlserver_10.3/server   -Dweblogic.management.discover=true  -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app01/oracle/Middleware/patch_wls1036/profiles/default/sysext_manifest_classpath:/app01/oracle/Middleware/patch_ocp371/profiles/default/sysext_manifest_classpath  weblogic.Server
<Apr 17, 2015 8:44:26 AM EDT> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
<Apr 17, 2015 8:44:26 AM EDT> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
<Apr 17, 2015 8:44:26 AM EDT> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 20.14-b01 from Sun Microsystems Inc.>
<Apr 17, 2015 8:44:27 AM EDT> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.6.0  Tue Nov 15 08:52:36 PST 2011 1441050 >
<Apr 17, 2015 8:44:28 AM EDT> <Info> <Security> <BEA-090065> <Getting boot identity from user.>
Enter username to boot WebLogic server:weblogic
Enter password to boot WebLogic server:
<Apr 17, 2015 8:44:35 AM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Apr 17, 2015 8:44:35 AM EDT> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<Apr 17, 2015 8:44:35 AM EDT> <Notice> <Log Management> <BEA-170019> <The server log file /app01/oracle/Middleware/user_projects/domains/MedRecDomain/servers/MedRecAdmServer/logs/MedRecAdmServer.log is opened. All server side log events will be written to this file.>
<Apr 17, 2015 8:44:38 AM EDT> <Warning> <Security> <BEA-090076> <A failure occurred attempting to load LDIF for provider Authorizer from file /app01/oracle/Middleware/wlserver_10.3/server/lib/XACMLAuthorizerInit.ldift.>
<Apr 17, 2015 8:44:38 AM EDT> <Error> <Security> <BEA-090870> <The realm "myrealm" failed to be loaded: weblogic.security.service.SecurityServiceException: com.bea.common.engine.ServiceInitializationException: weblogic.security.spi.ProviderInitializationException: A failure occurred attempting to load LDIF for provider Authorizer from file /app01/oracle/Middleware/wlserver_10.3/server/lib/XACMLAuthorizerInit.ldift..
weblogic.security.service.SecurityServiceException: com.bea.common.engine.ServiceInitializationException: weblogic.security.spi.ProviderInitializationException: A failure occurred attempting to load LDIF for provider Authorizer from file /app01/oracle/Middleware/wlserver_10.3/server/lib/XACMLAuthorizerInit.ldift.
    at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealm(CommonSecurityServiceManagerDelegateImpl.java:466)
    at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadRealm(CommonSecurityServiceManagerDelegateImpl.java:841)
    at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealms(CommonSecurityServiceManagerDelegateImpl.java:870)
    at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1032)
    at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
    Truncated. see log file for complete stacktrace
Caused By: com.bea.common.engine.ServiceInitializationException: weblogic.security.spi.ProviderInitializationException: A failure occurred attempting to load LDIF for provider Authorizer from file /app01/oracle/Middleware/wlserver_10.3/server/lib/XACMLAuthorizerInit.ldift.
    at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:365)
    at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:315)
    at com.bea.common.engine.internal.ServiceEngineImpl.lookupService(ServiceEngineImpl.java:257)
    at com.bea.common.engine.internal.ServicesImpl.getService(ServicesImpl.java:72)
    at weblogic.security.service.CSSWLSDelegateImpl.getService(CSSWLSDelegateImpl.java:155)
    Truncated. see log file for complete stacktrace
Caused By: weblogic.security.spi.ProviderInitializationException: A failure occurred attempting to load LDIF for provider Authorizer from file /app01/oracle/Middleware/wlserver_10.3/server/lib/XACMLAuthorizerInit.ldift.
    at com.bea.common.store.bootstrap.internal.BootStrapServiceImpl.loadFullLDIFTemplate(BootStrapServiceImpl.java:910)
    at com.bea.common.store.bootstrap.internal.BootStrapServiceImpl.loadLDIFTemplate(BootStrapServiceImpl.java:688)
    at com.bea.common.store.bootstrap.internal.BootStrapServiceImpl.loadLDIFXACMLAuthorizerTemplate(BootStrapServiceImpl.java:176)
    at com.bea.common.store.bootstrap.internal.BootStrapServiceImpl.loadLDIFXACMLAuthorizerTemplate(BootStrapServiceImpl.java:160)
    at com.bea.common.security.internal.service.BootStrapServiceImpl.loadLDIFXACMLAuthorizerTemplate(BootStrapServiceImpl.java:106)
    Truncated. see log file for complete stacktrace
Caused By: <openjpa-1.1.1-SNAPSHOT-r422266:1172209 fatal store error> kodo.jdo.FatalDataStoreException: The transaction has been rolled back.  See the nested exceptions for details on the errors that occurred.
    at org.apache.openjpa.kernel.BrokerImpl.newFlushException(BrokerImpl.java:2170)
    at org.apache.openjpa.kernel.BrokerImpl.flush(BrokerImpl.java:2017)
    at org.apache.openjpa.kernel.BrokerImpl.flushSafe(BrokerImpl.java:1915)
    at org.apache.openjpa.kernel.BrokerImpl.beforeCompletion(BrokerImpl.java:1833)
    at org.apache.openjpa.kernel.LocalManagedRuntime.commit(LocalManagedRuntime.java:81)
    Truncated. see log file for complete stacktrace
Caused By: <openjpa-1.1.1-SNAPSHOT-r422266:1172209 nonfatal store error> kodo.jdo.DataStoreException: ORA-00942: table or view does not exist
 {prepstmnt 1198520800 INSERT INTO BEAXACMLAP (DOMN, REALMN, TYPEN, CN, XVER, CTS, WCN, WCI, WXF, XD, XS, MTS) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) [params=(String) MedRecDomain, (String) myrealm, (String) Policies, (String) urn:bea:xacml:2.0:entitlement:resource:type@E@Fmbean@G, (String) 1.0, (null) null, (null) null, (null) null, (Blob) oracle.sql.BLOB@315cb235, (InputStream) java.io.ByteArrayInputStream@3cb7cee4, (String) 3, (Timestamp) 2015-04-17 08:44:38.792] [reused=0]} [code=942, state=42000]
FailedObject: com.bea.common.security.store.data.XACMLAuthorizationPolicy@55ab3cda
    at org.apache.openjpa.jdbc.sql.DBDictionary.narrow(DBDictionary.java:4207)
    at org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException(DBDictionary.java:4171)
    at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:102)
    at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:72)
    at kodo.jdbc.kernel.BatchingPreparedStatementManager.flushInternal(BatchingPreparedStatementManager.java:214)
    Truncated. see log file for complete stacktrace
Caused By: org.apache.openjpa.lib.jdbc.ReportingSQLException: ORA-00942: table or view does not exist
 {prepstmnt 1198520800 INSERT INTO BEAXACMLAP (DOMN, REALMN, TYPEN, CN, XVER, CTS, WCN, WCI, WXF, XD, XS, MTS) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) [params=(String) MedRecDomain, (String) myrealm, (String) Policies, (String) urn:bea:xacml:2.0:entitlement:resource:type@E@Fmbean@G, (String) 1.0, (null) null, (null) null, (null) null, (Blob) oracle.sql.BLOB@315cb235, (InputStream) java.io.ByteArrayInputStream@3cb7cee4, (String) 3, (Timestamp) 2015-04-17 08:44:38.792] [reused=0]} [code=942, state=42000]
    at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:192)
    at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:57)
    at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:866)
    at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:269)
    at org.apache.openjpa.jdbc.kernel.JDBCStoreManager$CancelPreparedStatement.executeUpdate(JDBCStoreManager.java:1453)
    Truncated. see log file for complete stacktrace
>
<Apr 17, 2015 8:44:38 AM EDT> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Apr 17, 2015 8:44:38 AM EDT> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:

There are 1 nested errors:

weblogic.security.service.SecurityServiceRuntimeException: [Security:090399]Security Services Unavailable
    at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:917)
    at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054)
    at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
    at weblogic.security.SecurityService.start(SecurityService.java:141)
    at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

>
<Apr 17, 2015 8:44:38 AM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<Apr 17, 2015 8:44:38 AM EDT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<Apr 17, 2015 8:44:38 AM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
weblogic.oraclesys.com /app01/oracle/Middleware/user_projects/domains/MedRecDomain/bin
weblogic=>cd $WL_HOME/server/lib
weblogic.oraclesys.com /app01/oracle/Middleware/wlserver_10.3/server/lib
weblogic=>ls



Probable Solution:

The database administrator needs to run the SQL script that creates the RDBMS tables in the datastore which are used by the RDBMS security store. A set of SQL scripts for creating and removing these tables for each supported RDBMS system is available in WL_HOME/server/lib
During the process of configuring the RDBMS security store via the Configuration Wizard, you are presented with the option of testing the database connection. Oracle strongly recommends using this option because it can verify that the connection is set up properly. If there were a problem with the database connection, you might not be able subsequently to boot the domain if the security providers that control access to that domain are unable to obtain the necessary security policies.

Create RDBMS Tables in the Security Datastore

Prior to booting the domain, the database administrator needs to run the SQL script that creates the RDBMS tables in the datastore used by the RDBMS security store. A set of SQL scripts for creating these tables for, and also removing them from, each supported RDBMS system is available in the following WebLogic Server installation directory:
WL_HOME/server/lib

When running the appropriate SQL script for the database serving as the RDBMS security store, be sure to specify the same connection properties, including the credentials of the user who has access, the database URL, etc., as specified for that RDBMS during domain creation.
Oracle 9i, 10g, 11g
———————————————
CREATE- rdbms_security_store_oracle.sql

REMOVE- rdbms_security_store_oracle_remove.sql
MS-SQL 2000, 2005
———————————————
rdbms_security_store_sqlserver.sql

rdbms_security_store_sqlserver_remove.sql
DB2 9.2, 9.5
———————————————

rdbms_security_store_db2.sql
rdbms_security_store_db2_remove.sql
Derby
———————————————

rdbms_security_store_derby.sql