Increasing/Decreasing Heap Size of the Weblogic JVM

BEFORE:

$ ps -ef|grep Xms
oracle 32751 32700 36 21:50 ?        00:03:54 /u01/app/jdk1.8.0_91/bin/java -server -Xms512m -Xmx1024m -Dweblogic.Name=AdminServer


To change the Xms and Xmx parameter update the env file at "$DOMAIN_HOME/bin/setDomainEnv.sh" with values below and stop and start the weblogic instance for the changes to take effect.

USER_MEM_ARGS="-Xms256M -Xmx256M"
export USER_MEM_ARGS

i.e.

$ grep USER_MEM_ARGS setDomainEnv.sh
# USER_MEM_ARGS   - The variable to override the standard memory arguments
# IF USER_MEM_ARGS the environment variable is set, use it to override ALL MEM_ARGS values

USER_MEM_ARGS="-Xms256M -Xmx256M"
export USER_MEM_ARGS
 
if [ "${USER_MEM_ARGS}" != "" ] ; then
        MEM_ARGS="${USER_MEM_ARGS}"





AFTER:

$ ps -ef|grep Xms
oracle 3506  3455 76 22:57 ?        00:01:38 /u01/app/jdk1.8.0_91/bin/java -server -Xms256M -Xmx256M -Dweblogic.Name=AdminServer

WebLogic R12c: nodemanager failed to start

The configuration that is set in the Admin Console"Environment, Machines, Configuration, Node Manager" conflicts with the configuration in $DOMAIN_HOME/nodemanager/nodemanager.properties.

Error: "For Server $ServerName, the Node Manager Associated With Machine $MachineName is Not Reachable" Attempting To Start A Managed Server Within The Administration Console (Doc ID 2054244.1)


=>/app01/Oracle_Home/user_projects/domains/ORDS/bin
=>./startNodeManager.sh
NODEMGR_HOME is already set to /app01/Oracle_Home/user_projects/domains/ORDS/nodemanager
/app01/Oracle_Home/wlserver/server/bin/startNodeManager.sh: !: not found
CLASSPATH=/app01/Java/jdk1.7.0_79/lib/tools.jar:/app01/Oracle_Home/wlserver/server/lib/weblogic_sp.jar:/app01/Oracle_Home/wlserver/server/lib/weblogic.jar:/app01/Oracle_Home/oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b3/lib/ant-contrib.jar:/app01/Oracle_Home/wlserver/modules/features/oracle.wls.common.nodemanager_2.0.0.0.jar::/app01/Oracle_Home
+ /app01/Java/jdk1.7.0_79/bin/java -server -d64 -Xms32m -Xmx200m -XX:MaxPermSize=128m -Dcoherence.home=/app01/Oracle_Home/coherence -Dbea.home=/app01/Oracle_Home -Dweblogic.RootDirectory=/app01/Oracle_Home/user_projects/domains/ORDS -Xverify:none -Djava.endorsed.dirs=/app01/Java/jdk1.7.0_79/jre/lib/endorsed:/app01/Oracle_Home/oracle_common/modules/endorsed -Djava.security.policy=/app01/Oracle_Home/wlserver/server/lib/weblogic.policy -Dweblogic.nodemanager.JavaHome=/app01/Java/jdk1.7.0_79 weblogic.NodeManager -v
<Jan 2, 2017 3:38:46 AM EST> <INFO> <Loading domains file: /app01/Oracle_Home/user_projects/domains/ORDS/nodemanager/nodemanager.domains>
<Jan 2, 2017 3:38:49 AM EST> <INFO> <Loaded NodeManager configuration properties from '/app01/Oracle_Home/user_projects/domains/ORDS/nodemanager/nodemanager.properties'>
Node manager v12.1.3

Configuration settings:

DomainsFile=/app01/Oracle_Home/user_projects/domains/ORDS/nodemanager/nodemanager.domains
LogLimit=0
DomainsDirRemoteSharingEnabled=false
AuthenticationEnabled=true
LogLevel=INFO
DomainsFileEnabled=true
ListenAddress=localhost
NativeVersionEnabled=true
ProcessDestroyTimeout=20000
ListenPort=8156
LogToStderr=true
weblogic.StartScriptName=startWebLogic.sh
SecureListener=true
LogCount=1
LogAppend=true
weblogic.StopScriptEnabled=false
StateCheckInterval=500
CrashRecoveryEnabled=false
weblogic.StartScriptEnabled=true
LogFile=/app01/Oracle_Home/user_projects/domains/ORDS/nodemanager/nodemanager.log
LogFormatter=weblogic.nodemanager.server.LogFormatter
coherence.StartScriptEnabled=false
ListenBacklog=50
NodeManagerHome=/app01/Oracle_Home/user_projects/domains/ORDS/nodemanager
weblogic.startup.JavaHome=/app01/Java/jdk1.7.0_79
weblogic.startup.MW_Home=
coherence.startup.JavaHome=/app01/Java/jdk1.7.0_79
coherence.startup.MW_Home=

Domain name mappings:

ORDS -> /app01/Oracle_Home/user_projects/domains/ORDS

<Jan 2, 2017 3:38:49 AM EST> <INFO> <WebLogic Server 12.1.3.0.0  Wed May 21 18:53:34 PDT 2014 1604337 >
<Jan 2, 2017 3:38:50 AM EST> <INFO> <Secure socket listener started on port 8156, host localhost>
^C



Solution:
Changed ListenAddress in "$DOMAIN_HOME/nodemanager/nodemanager.properties"

before:
ListenAddress=localhost
SecureListener=false


after:
ListenAddress=dev.oracle.com 
SecureListener=true



=>./startNodeManager.sh
NODEMGR_HOME is already set to /app01/Oracle_Home/user_projects/domains/ORDS/nodemanager
/app01/Oracle_Home/wlserver/server/bin/startNodeManager.sh: !: not found
CLASSPATH=/app01/Java/jdk1.7.0_79/lib/tools.jar:/app01/Oracle_Home/wlserver/server/lib/weblogic_sp.jar:/app01/Oracle_Home/wlserver/server/lib/weblogic.jar:/app01/Oracle_Home/oracle_common/modules/net.sf.antcontrib_1.1.0.0_1-0b3/lib/ant-contrib.jar:/app01/Oracle_Home/wlserver/modules/features/oracle.wls.common.nodemanager_2.0.0.0.jar::/app01/Oracle_Home
+ /app01/Java/jdk1.7.0_79/bin/java -server -d64 -Xms32m -Xmx200m -XX:MaxPermSize=128m -Dcoherence.home=/app01/Oracle_Home/coherence -Dbea.home=/app01/Oracle_Home -Dweblogic.RootDirectory=/app01/Oracle_Home/user_projects/domains/ORDS -Xverify:none -Djava.endorsed.dirs=/app01/Java/jdk1.7.0_79/jre/lib/endorsed:/app01/Oracle_Home/oracle_common/modules/endorsed -Djava.security.policy=/app01/Oracle_Home/wlserver/server/lib/weblogic.policy -Dweblogic.nodemanager.JavaHome=/app01/Java/jdk1.7.0_79 weblogic.NodeManager -v
<Jan 2, 2017 3:48:36 AM EST> <INFO> <Loading domains file: /app01/Oracle_Home/user_projects/domains/ORDS/nodemanager/nodemanager.domains>
<Jan 2, 2017 3:48:39 AM EST> <INFO> <Loaded NodeManager configuration properties from '/app01/Oracle_Home/user_projects/domains/ORDS/nodemanager/nodemanager.properties'>
Node manager v12.1.3

Configuration settings:

DomainsFile=/app01/Oracle_Home/user_projects/domains/ORDS/nodemanager/nodemanager.domains
LogLimit=0
DomainsDirRemoteSharingEnabled=false
AuthenticationEnabled=true
LogLevel=INFO
DomainsFileEnabled=true
ListenAddress=dev.oracle.com
NativeVersionEnabled=true
ProcessDestroyTimeout=20000
ListenPort=8156
LogToStderr=true
weblogic.StartScriptName=startWebLogic.sh
SecureListener=false
LogCount=1
LogAppend=true
weblogic.StopScriptEnabled=false
StateCheckInterval=500
CrashRecoveryEnabled=false
weblogic.StartScriptEnabled=true
LogFile=/app01/Oracle_Home/user_projects/domains/ORDS/nodemanager/nodemanager.log
LogFormatter=weblogic.nodemanager.server.LogFormatter
coherence.StartScriptEnabled=false
ListenBacklog=50
NodeManagerHome=/app01/Oracle_Home/user_projects/domains/ORDS/nodemanager
weblogic.startup.JavaHome=/app01/Java/jdk1.7.0_79
weblogic.startup.MW_Home=
coherence.startup.JavaHome=/app01/Java/jdk1.7.0_79
coherence.startup.MW_Home=

Domain name mappings:

ORDS -> /app01/Oracle_Home/user_projects/domains/ORDS

<Jan 2, 2017 3:48:39 AM EST> <INFO> <WebLogic Server 12.1.3.0.0  Wed May 21 18:53:34 PDT 2014 1604337 >
<Jan 2, 2017 3:48:39 AM EST> <INFO> <Plain socket listener started on port 8156, host dev.oracle.com>


 

WegLogic 12c silent installation on SUN Solaris

WebLogic 12c(12.1.3) on Solaris SPARC 64 bit fails with the error below:


java -jar fmw_12.1.3.0.0_wls.jar -silent -responseFile wls.rsp
Launcher log file is /var/tmp/OraInstall2016-12-29_10-33-15AM/launcher2016-12-29_10-33-15AM.log.
Extracting files...............................
Starting Oracle Universal Installer

Checking if CPU speed is above 300 MHz.   Actual 1415 MHz    Passed
Checking swap space: must be greater than 512 MB.   Actual 17934 MB    Passed
Checking if this platform requires a 64-bit JVM.   Actual 64    Passed
Checking temp space: must be greater than 300 MB.   Actual 56261 MB    Passed

Preparing to launch the Oracle Universal Installer from /var/tmp/OraInstall2016-12-29_10-33-15AM
Log: /var/tmp/OraInstall2016-12-29_10-33-15AM/install2016-12-29_10-33-15AM.log
Copyright (c) 1996, 2014, Oracle and/or its affiliates. All rights reserved.
Reading response file..
Starting check : CertifiedVersions
Expected result: One of 5.10,5.11
Actual Result: 5.10
Check complete. The overall result of this check is: Passed
CertifiedVersions Check: Success.
Starting check : CheckJDKVersion
Expected result: 1.7.0_15
Actual Result: 1.7.0_79
Check complete. The overall result of this check is: Passed
CheckJDKVersion Check: Success.
Validations are enabled for this session.
Verifying data......
[VALIDATION] [ERROR]:INST-07546: Unable to find install type Fusion Middleware Infrastructure
[VALIDATION] [SUGGESTION]:Provide a valid install type
installation Failed. Exiting installation due to data validation failure.


Solution:

Changed  "INSTALL_TYPE" parameter in response file.
< INSTALL_TYPE=Fusion Middleware Infrastructure
---
> INSTALL_TYPE=WebLogic Server
 



Launcher log file is /var/tmp/OraInstall2016-12-29_11-14-20AM/launcher2016-12-29_11-14-20AM.log.
Extracting files...............................
Starting Oracle Universal Installer

Checking if CPU speed is above 300 MHz.   Actual 1415 MHz    Passed
Checking swap space: must be greater than 512 MB.   Actual 17923 MB    Passed
Checking if this platform requires a 64-bit JVM.   Actual 64    Passed
Checking temp space: must be greater than 300 MB.   Actual 55245 MB    Passed


Preparing to launch the Oracle Universal Installer from /var/tmp/OraInstall2016-12-29_11-14-20AM
Log: /var/tmp/OraInstall2016-12-29_11-14-20AM/install2016-12-29_11-14-20AM.log
Copyright (c) 1996, 2014, Oracle and/or its affiliates. All rights reserved.
Reading response file..
Starting check : CertifiedVersions
Expected result: One of 5.10,5.11
Actual Result: 5.10
Check complete. The overall result of this check is: Passed
CertifiedVersions Check: Success.
Starting check : CheckJDKVersion
Expected result: 1.7.0_15
Actual Result: 1.7.0_79
Check complete. The overall result of this check is: Passed
CheckJDKVersion Check: Success.
Validations are enabled for this session.
Verifying data......
Copying Files...
You can find the log of this install session at:
 /var/tmp/OraInstall2016-12-29_11-14-20AM/install2016-12-29_11-14-20AM.log
-----------20%----------40%----------60%----------80%--------100%

The installation of Oracle Fusion Middleware 12c WebLogic Server and Coherence 12.1.3.0.0 completed successfully.
Logs successfully copied to /var/opt/oracle/oraInventory/logs.
 


source:
http://www.kernelbytes.com/silent-mode-installation-weblogic12c/
https://docs.oracle.com/middleware/1213/core/OUIRF/toc.htm

Backup of Weblogic Domain: pack command

 /u01/app01/Oracle/Middleware/user_projects/domains/demo_domain[oraapp@weblogic bin]$ pwd
/u01/app01/Oracle/Middleware/wlserver_10.3/common/bin

[oraapp@weblogic bin]$ ./pack.sh
-domain="/u01/app01/Oracle/Middleware/user_projects/domains/demo_domain"  -template="/u01/bkp/apps_bkp/17OCT15/demo_domain.jar"
-template_name="WebCenter Content Domain"
-log=/u01/bkp/apps_bkp/17OCT15/demo_domain.log
-log_priority=INFO
<< read domain from "/u01/app01/Oracle/Middleware/user_projects/domains/demo_domain"
>>  succeed: read domain from "/u01/app01/Oracle/Middleware/user_projects/domains/demo_domain"
<< write template to "/u01/bkp/apps_bkp/17OCT15/demo_domain.jar"
....................................................................................................
>>  succeed: write template to "/u01/bkp/apps_bkp/17OCT15/demo_domain.jar"
<< close template
>>  succeed: close template
[oraapp@weblogic bin]$
 

Configuring SSL for Oracle Weblogic (and OFMW)

https://jvzoggel.wordpress.com/2011/12/16/configuring-ssl-for-oracle-weblogic-and-ofmw/


We have a very basic domain: 3 physical servers, 1 containing the admin server, 2 containing 2 managed servers each, running in a cluster. Nice picture to follow. :)
name of the servers: server01 (Admin) / server02 (MS1+MS2) / server03 (MS3+MS4)

Step 0 – The Weblogic Demo Certificates (the reason why)

If you ever checked your Weblogic production logs and found the Security Alert below, then your Weblogic domain has a huge gaping security hole. Even if you don’t want SSL communication for your webservices or applications. The internal (administrative) processes in your Weblogic domain still relies on the default demotrust and with this everyone can access your domain.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

</pre> &nbsp;   <Alert> <Security> <SERVER02> <rbx_dev_soa_ms01> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <BEA-090152> <Demo trusted CA certificate is being used in production mode: [ [   Version: V3   Subject: CN=CACERT, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US   Signature Algorithm: MD5withRSA, OID = 1.2.840.113549.1.1.4     Key:  Sun RSA public key, 512 bits   modulus: 95501928778692415440753....   public exponent: 65537   Validity: [From: Thu Mar 21 21:12:27 CET 2002,                To: Tue Mar 22 21:12:27 CET 2022]   Issuer: CN=CACERT, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US   SerialNumber: [    33f10648 fcde0deb 4199921f d64537f4]   Certificate Extensions: 1 [1]: ObjectId: 2.5.29.15 Criticality=true KeyUsage [   Key_CertSign ]   ]   Algorithm: [MD5withRSA]   Signature: 0000: 9D 26 4C 29 C8 91 C3 A7   06 C3 24 6F AE B4 F8 82  .&L)......$o.... 0010: 80 4D AA CB 7C 79 46 84   81 C4 66 95 F4 1E D8 C4  .M...yF...f..... 0020: E9 B7 D9 7C E2 23 33 A4   B7 21 E0 AA 54 2B 4A FF  .....#3..!..T+J. 0030: CB 21 20 88 81 21 DB AC   90 54 D8 7D 79 63 23 3C  .! ..!...T..yc#<   ] The system is vulnerable to security attacks, since it trusts certificates signed by the demo trusted CA.>

Step 1 – generate the identity java keystore

We first generate a java keystores containing a private key for each physical server, since SSL hostname verifiation is based on machine names. We use “welcome1” as password here, but make sure to choose a complex password here, use it during these steps and then lock it safely away.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

keytool.exe -genkey -keysize 2048 -keyalg RSA -alias server01 -keystore server01_identity.jks   Enter keystore password: welcome1  Re-enter new password: welcome1  What is your first and last name?    [Unknown]:  server01  What is the name of your organizational unit?    [Unknown]:  What is the name of your organization?    [Unknown]:  Rubix  What is the name of your City or Locality?    [Unknown]:  What is the name of your State or Province?    [Unknown]:  What is the two-letter country code for this unit?    [Unknown]:  NL  Is CN=server01, OU=, O=Rubix, L=, ST=, C=NL correct?    [no]:  yes <br>

Step 2 – generate signing request

1	keytool.exe -certreq -alias server01 -file server01.csr -keystore server01_identity.jks</div>

Repeat step 1 + 2 for each physical server, replacing the -alias parameter, -keystore parameter and CN name with the correct server0* value.

 

 

Step 3 – the Certificate Authority (CA)

Now comes the tricky part that you probably need some outside forces. You will need the correct Certificate Authority to sign your certificate requests (the .CSR files you generated). You can create your own CA and self-sign them, but I would not recommend that unless you know what you are doing and understand the consequences. You could use an external Internet CA provider, but this becomes very costly and time consuming when you need such an external provider to sign every SSL enabled server in your landscape. The best situation for us would be if the current organization already has an internal CA provider, especially when the rootCA is trusted by the servers and machines in your landscape.

 

Let’s assume however that you receive the signed response, including the public keys of the rootCA and a intermediateCA. CA instances often use a intermediate CA to authorize certificates, so if they have “problems” they can withdraw this intermediate CA and don’t need to withdraw the whole RootCA making it useless.
So the situation should be that you know hold 3 PEM/DER files.

 

 

Step 4 – importing the CA response

Import the certificates in your keystore, starting with the rootCA, then the intermediateCA, then the specific server alias. Example for server01:

1 2 3 4 5

keytool -importcert -trustcacerts -alias rootca -file rootCA.cer -keystore server01_identity.jks   keytool -importcert -alias intermediateca -file intermediateCA.cer -keystore server01_identity.jks   keytool -importcert -alias server01 -file server01.cer -keystore server01_identity.jks

Repeat these steps for server02 and server03. So we know have 3 identity keystores for each server containing the signed private key and it’s certificate chain.

 

 

Step 5 – generate the trust java keystore

Now let’s generate a truststore for each server. Altough you can use the identical keystore for both identity and trust, it’s always a good practice to separate them. The identity keystone contains the private key and normally never changes during it’s lifetime, so you should secure it with a complex password. The trust store contains the certificates that you trust and as your middleware landscape grows, this file might grow and more administrators will need access to it.

 

Example generating the truststore for server01, repeat for 02 & 03.

 

1 2	keytool -importcert -trustcacerts -alias rootca -file rootCA.cer -keystore server01_trust.jks keytool -importcert -alias intermediateca -file intermediateCA.cer -keystore server01_trust.jks

 

Step 6 – copy the java keystore files to each server

 Make sure that each server had the files stored on disk. We use the windows server example “c:\oracle\security” in this blogpost. So server01 should hold:

– c:\oracle\security\server01_identity.jks

– c:\oracle\security\server01_trust.jks

 

 

Step 7 – configure the Node Manager for SSL

The nodemanger by default uses the demoIdentity and demoTrust keystores.
So on each server hosting a managed server (server02 and server03) we will need to change this.

Open the nodemanager.properties and add the following lines:

 

1 2 3 4 5 6 7 8 9

### SSL config KeyStores=CustomIdentityAndCustomTrust # CustomIdentityKeyStoreType=JKS CustomIdentityKeyStoreFileName=c://oracle//security//server01_identity.jks CustomIdentityKeyStorePassPhrase=welcome1 CustomIdentityAlias=server01 CustomIdentityPrivateKeyPassPhrase=welcome1 # CustomTrustKeyStoreType=JKS CustomTrustKeyStoreFileName=c://oracle//security//server01_trust.jks

Restart the NodeManager to activate the changes. The passwords in the properties file will automatically become encrypted.

 

Possible error:

1

For server rbx_dev_soa_ms03, the Node Manager associated with machine server03 is not reachable. All of the servers selected are currently in a state which is incompatible with this operation or are not associated with a running Node Manager or you are not authorized to perform the action requested. No action will be performed.

Possible solution:

Your AdminServer can not communicate with a node manager. Make sure it’s running, check Environment -> Machines in the Weblogic console if the machine has state reachable, check if the address configured there, and the address in the nodemanager.properties and the CN-name used in SSL have the same value, check if Weblogic and Nodemanager use the same SSL configuration (trust+identity).

 

 

Step 8 – configure Weblogic for SSL

Log in the Weblogic console, in our example: http://server01:7001/console

 

 

Step 8.1 – configure Weblogic for SSL, enable SSL

 

Log in the Weblogic console (in our case http://server01:7001/console) and make sure that each server has the SSL port enabled. You can find the setting “SSL Listen Port Enabled” for each server under “Configuration -> General”. Enable it and make sure you use a correct and free port.

 

Step 8.2 – configure Weblogic for SSL, changing keystores

For each server (Admin + MS) we have to change the KeyStore configuration.

So for each server go to: Configuration -> Keystores

 

• Keystores -> “Change” = “Custom Identity and Custom Trust”
• Custom Identity Keystore = c:/Oracle/security/serverXX_identity.jks
• Custom Identity Keystore Type = JKS
• Custom Identity Keystore Passphrase = welcome1
• Custom Trust keystore = c:/Oracle/security/serverXX_trust.jks
• Custom Trust Keystore Type = JKS
• Custom Trust Keystore Passphrase = welcome1

 

 

Step 8.3 – configure Weblogic for SSL, changing identity:

For each server (Admin + MS) we have to change the SSL configuration.

So for each server go to: Configuration -> SSL

 

• Private Key Alias = serverXX
• Private Key Passphrase = welcome1

 

Step 9 – Weblogic start scripts:

By default Weblogic uses the DemoTrust.jks truststore. Since we changed this in the console configuration you would hope this would be enough. However the SetDomainEnv.sh / SetDomainEnv.cmd script still contains a reference. So for each server:

 

• Go to %domain_home%/bin/
• Open the SetDomainEnv script
• Search the argument: -Djavax.net.ssl.trustStore=
• Make sure the value is not referring to the demo trust.jks but “c:/oracle/security/serverXX_trust.jks

When you are finished restart your whole Weblogic environment (Admin + MS).

 

Step 10 – Configure Enterprise Manager:

This took me some research and the Oracle Forums to get it working. Navigate to the enterprise manager console (/em)

 

Step 10.1 – Security Provider Configuration:

I’m actually not really sure if this step is still necessary in 11g, but hey .. why not be sure:

 

• Click Weblogic -> right-click domain
• Select Security -> Security Provider Configuration -> KeyStore Configure
• Keystore Path = c:/oracle/security/server01_identity.jks

 

Step 10.2 – SOA Infra properties

 

• Click SOA -> right-click soa-infra
• Select SOA Administration -> Common Properties
• Click More SOA Infra Advanced Properties
• Keystore Location = c:/oracle/security/serverXX_identity.jks

Step 10.3 – Credential Mapping:
The next step is to configure the credential mapping keys

• Click Weblogic -> right-click domain name
• Select Security -> Credentials
• Create a Map named “SOA”
• Click Create a key
• Key = KeyPassword
• User Name = KeyPassword
• Password = welcome1
• Click Create Key:
• Key = KeyStorePassword
• User Name = KeyStorePassword
• Password = welcome1

 

Possible error:

1 2 3 4 5 6 7 8 9 10 11

</div> <div>   <Warning> <oracle.as.jmx.framework.MessageLocalizationHelper> <J2EE JMX-46041> <The resource for bundle "oracle.soa.management.config.bpel.mbeans.MessageBundle_en" with key "oracle.soa.BPELConfigBean.auditFlushByteThreshold" cannot be found.> INFO: SSLSocketFactoryManagerImpl.getKeystoreLocation SOA Keystore location: c:/oracle/security/server04_identity.jks INFO: SSLSocketFactoryManagerImpl.getKeystorePassword Obtained null or empty keystore password INFO: SSLSocketFactoryManagerImpl.getKeyPassword Obtained null or empty key password INFO: SSLSocketFactoryManagerImpl.getSSLSocketFactory Could not obtain keystore location or password   </div> <div>

 

Possible solution:

You did not configure the SOA key mapping correctly as mentioned above in 9.3

 

Conclusion:

Enabling SSL takes a lot of steps, and we didn’t even mentioned it for enabling SSL client identity in OSB (Service Key Provider) and communication with an LDAP over SSL. But there’s more to come I guess.

If you have any remarks please let me know in the comments. Hope it helps. :)

 

 

Links:

• Oracle Forums – 11g keystore & password config <- +1 for Rod Tunnels

WebLogic Server - identity vs trust keystores

Source:  http://one-size-doesnt-fit-all.blogspot.com/2009/09/weblogic-server-identity-vs-trust.html

Summary:
Identity keystore:
This will be used to store the server certificate(private key/digital certificate pairs). When the client contacts server the digital certificate presented in this keystore will be sent.

Trust Keystore:
This will contain all the certificates for the trusted partners(ie partners). When server connects with the partner it will use this key store.

In computing most technologies have lots of terms and acronyms to learn, it's par for the course, you get used to it. However in computer security the frustration is multiplied as there are often many different terms that mean the same thing. It makes implementing security hard, because understanding it is hard, and I'm not surprised why security is considered badly implemented because the average Joe will struggle.

I've been trying recently to get straight in my head what is stored in the WLS identity and trust keystores, and what the difference between identity and trust is anyhow. Thanks to kind assistance from Gerard Davison, I think I can now post my understandings, and as usual, hopefully the post is helpful to other readers. As noted however security to me is a difficult area, and so be sure to check the facts here, your mileage with this post may vary.

The following WLS documentation attempts to explain the concepts of identity and trust:

http://download.oracle.com/docs/cd/E12839_01/web.1111/e13707/identity_trust.htm#i1170342

...in ripping out one of the core paragraphs, with a slight rewrite of my own we can see the concept of identity, and how it relates to the public and private keys:

"The public key is embedded in a digital certificate with additional information describing the owner of the public key, such as name, street address, and e-mail address *as well as the hostname*. *Along with this the digital certificate containing the public key, and the separate related private key, provide identity for the server*."

...ultimately to identify the server, to assert the server is who the server says it is.

The digital certificate containing the public key is also referred to as the "server certificate", as for example in 1-way-SSL traffic between the server and client, the server certificate containing the public key is what is initially passed to the client.

There is a missing piece in the puzzle. Regardless that the digital certificate states the owner of the public key, their name and so on, how does a client know that the "identity" asserted by the digital certificate is true? That's where Certificate Authorities (CAs) come in.

Ignoring self signed digital certificates, a typical digital certificate used on the internet containing the public key and owner details is signed by a trusted CA who has verified the identity of the owner. Presumably when purchasing digital certificates from CAs, this is what some of the cost covers, the CAs research into ensuring that the identity details embedded in the digital cert are actually true.

At runtime on receiving the digital certificate, the client checks the CA and if the CA is one that the client trusts (or a CA in a chain of trusted CAs), then the identity of the server is established/verified.

Thus the "identity" of the server is established by what's stored in the "identity" keystore, and its contents are what are farmed out to clients establishing secure connections with the server, who then verify the supplied digital certificate's CA against the clients own list of trusted CAs. The "identity keystore" is also referred to as the "server keystore", because it establishes the server's identity (ie. I am who I say I am).

WLS side note: As mentioned the digital certificate also includes the host name of the server, or in other words the digital certificate is pegged to that server and that server alone. This implies on that server with its relating digital certificate, *all* of the applications will share that single digital certificate for secure communications. Occasionally a requirement will arise where each application must have its own digital certificate. In WLS because keystores are configured under an individual WLS "managed server", if you have two separate applications, it is not possible to use separate digital certificates for each in one managed server. The solution is to create another managed server with its own keystores.

WLS web service side note: Following on from the previous side note, for web services that use in-message encryption and digital signatures, there is often the requirement for multiple different digital certificates. Under WLS to provision the WS-Security model, WLS has a separate Web Service Security Configuration (WSSC) to provision this setup.

Finally regarding the trust keystore, what is its job in all of this? The trust keystore is typically used for storing CA digital certificates, essentially the CAs who will be used to check any digital certificates that are given to the server at runtime (just the same as the client did above). In the standard 1-way-SSL between a client and the WLS server, the trust keystore doesn't come into the equation as the client has its own trust keystore (containing the CAs) and the server has nothing to verify. Yet in the case of mutual SSL (aka. 2 way SSL) between the client and server, the client and server actually swap each other digital certificates to establish identity of both parties, and in this case the server must be able to test the identity of the client through the CA of the client's digital certificate.

Mutual SSL side note: the setup of mutual SSL is more complicated than this. Readers are advised to refer to the following Oracle article.

WebLogic Server fails to start with SecurityServiceRuntimeException Error

WebLogic Server does not start, showing the following stack trace, soon after Domain creation:


weblogic=>./startWebLogic.sh
.
.
JAVA Memory arguments: -Xms256m -Xmx512m  -XX:MaxPermSize=256m
.
WLS Start Mode=Production
.
CLASSPATH=/app01/oracle/Middleware/patch_wls1036/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/app01/oracle/Middleware/patch_ocp371/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/app01/java/jdk1.6.0_43/lib/tools.jar:/app01/oracle/Middleware/wlserver_10.3/server/lib/weblogic_sp.jar:/app01/oracle/Middleware/wlserver_10.3/server/lib/weblogic.jar:/app01/oracle/Middleware/modules/features/weblogic.server.modules_10.3.6.0.jar:/app01/oracle/Middleware/wlserver_10.3/server/lib/webservices.jar:/app01/oracle/Middleware/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/app01/oracle/Middleware/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/app01/oracle/Middleware/wlserver_10.3/common/derby/lib/derbyclient.jar:/app01/oracle/Middleware/wlserver_10.3/server/lib/xqrl.jar
.
PATH=/app01/oracle/Middleware/wlserver_10.3/server/bin:/app01/oracle/Middleware/modules/org.apache.ant_1.7.1/bin:/app01/java/jdk1.6.0_43/jre/bin:/app01/java/jdk1.6.0_43/bin:/app01/oracle/Middleware/wlserver_10.3/server/bin:/app01/oracle/Middleware/modules/org.apache.ant_1.7.1/bin:/app01/java/jdk1.6.0_43/jre/bin:/app01/java/jdk1.6.0_43/bin:/app01/oracle/Middleware/wlserver_10.3/server/bin:/app01/oracle/Middleware/modules/org.apache.ant_1.7.1/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/weblogic/bin:/home/weblogic/bin:/home/weblogic/bin:/app01/java/jdk1.6.0_43
.
***************************************************
*  To start WebLogic Server, use a username and   *
*  password assigned to an admin-level user.  For *
*  server administration, use the WebLogic Server *
*  console at http://hostname:port/console        *
***************************************************
starting weblogic with Java version:
java version "1.6.0_43"
Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01, mixed mode)
Starting WLS with line:
/app01/java/jdk1.6.0_43/bin/java -server   -Xms256m -Xmx512m  -XX:MaxPermSize=256m -Dweblogic.Name=MedRecAdmServer -Djava.security.policy=/app01/oracle/Middleware/wlserver_10.3/server/lib/weblogic.policy  -Dweblogic.ProductionModeEnabled=true  -Xverify:none  -da -Dplatform.home=/app01/oracle/Middleware/wlserver_10.3 -Dwls.home=/app01/oracle/Middleware/wlserver_10.3/server -Dweblogic.home=/app01/oracle/Middleware/wlserver_10.3/server   -Dweblogic.management.discover=true  -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app01/oracle/Middleware/patch_wls1036/profiles/default/sysext_manifest_classpath:/app01/oracle/Middleware/patch_ocp371/profiles/default/sysext_manifest_classpath  weblogic.Server
<Apr 17, 2015 8:44:26 AM EDT> <Info> <Security> <BEA-090905> <Disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
<Apr 17, 2015 8:44:26 AM EDT> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
<Apr 17, 2015 8:44:26 AM EDT> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 20.14-b01 from Sun Microsystems Inc.>
<Apr 17, 2015 8:44:27 AM EDT> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.6.0  Tue Nov 15 08:52:36 PST 2011 1441050 >
<Apr 17, 2015 8:44:28 AM EDT> <Info> <Security> <BEA-090065> <Getting boot identity from user.>
Enter username to boot WebLogic server:weblogic
Enter password to boot WebLogic server:
<Apr 17, 2015 8:44:35 AM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Apr 17, 2015 8:44:35 AM EDT> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<Apr 17, 2015 8:44:35 AM EDT> <Notice> <Log Management> <BEA-170019> <The server log file /app01/oracle/Middleware/user_projects/domains/MedRecDomain/servers/MedRecAdmServer/logs/MedRecAdmServer.log is opened. All server side log events will be written to this file.>
<Apr 17, 2015 8:44:38 AM EDT> <Warning> <Security> <BEA-090076> <A failure occurred attempting to load LDIF for provider Authorizer from file /app01/oracle/Middleware/wlserver_10.3/server/lib/XACMLAuthorizerInit.ldift.>
<Apr 17, 2015 8:44:38 AM EDT> <Error> <Security> <BEA-090870> <The realm "myrealm" failed to be loaded: weblogic.security.service.SecurityServiceException: com.bea.common.engine.ServiceInitializationException: weblogic.security.spi.ProviderInitializationException: A failure occurred attempting to load LDIF for provider Authorizer from file /app01/oracle/Middleware/wlserver_10.3/server/lib/XACMLAuthorizerInit.ldift..
weblogic.security.service.SecurityServiceException: com.bea.common.engine.ServiceInitializationException: weblogic.security.spi.ProviderInitializationException: A failure occurred attempting to load LDIF for provider Authorizer from file /app01/oracle/Middleware/wlserver_10.3/server/lib/XACMLAuthorizerInit.ldift.
    at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealm(CommonSecurityServiceManagerDelegateImpl.java:466)
    at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadRealm(CommonSecurityServiceManagerDelegateImpl.java:841)
    at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealms(CommonSecurityServiceManagerDelegateImpl.java:870)
    at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1032)
    at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
    Truncated. see log file for complete stacktrace
Caused By: com.bea.common.engine.ServiceInitializationException: weblogic.security.spi.ProviderInitializationException: A failure occurred attempting to load LDIF for provider Authorizer from file /app01/oracle/Middleware/wlserver_10.3/server/lib/XACMLAuthorizerInit.ldift.
    at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:365)
    at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:315)
    at com.bea.common.engine.internal.ServiceEngineImpl.lookupService(ServiceEngineImpl.java:257)
    at com.bea.common.engine.internal.ServicesImpl.getService(ServicesImpl.java:72)
    at weblogic.security.service.CSSWLSDelegateImpl.getService(CSSWLSDelegateImpl.java:155)
    Truncated. see log file for complete stacktrace
Caused By: weblogic.security.spi.ProviderInitializationException: A failure occurred attempting to load LDIF for provider Authorizer from file /app01/oracle/Middleware/wlserver_10.3/server/lib/XACMLAuthorizerInit.ldift.
    at com.bea.common.store.bootstrap.internal.BootStrapServiceImpl.loadFullLDIFTemplate(BootStrapServiceImpl.java:910)
    at com.bea.common.store.bootstrap.internal.BootStrapServiceImpl.loadLDIFTemplate(BootStrapServiceImpl.java:688)
    at com.bea.common.store.bootstrap.internal.BootStrapServiceImpl.loadLDIFXACMLAuthorizerTemplate(BootStrapServiceImpl.java:176)
    at com.bea.common.store.bootstrap.internal.BootStrapServiceImpl.loadLDIFXACMLAuthorizerTemplate(BootStrapServiceImpl.java:160)
    at com.bea.common.security.internal.service.BootStrapServiceImpl.loadLDIFXACMLAuthorizerTemplate(BootStrapServiceImpl.java:106)
    Truncated. see log file for complete stacktrace
Caused By: <openjpa-1.1.1-SNAPSHOT-r422266:1172209 fatal store error> kodo.jdo.FatalDataStoreException: The transaction has been rolled back.  See the nested exceptions for details on the errors that occurred.
    at org.apache.openjpa.kernel.BrokerImpl.newFlushException(BrokerImpl.java:2170)
    at org.apache.openjpa.kernel.BrokerImpl.flush(BrokerImpl.java:2017)
    at org.apache.openjpa.kernel.BrokerImpl.flushSafe(BrokerImpl.java:1915)
    at org.apache.openjpa.kernel.BrokerImpl.beforeCompletion(BrokerImpl.java:1833)
    at org.apache.openjpa.kernel.LocalManagedRuntime.commit(LocalManagedRuntime.java:81)
    Truncated. see log file for complete stacktrace
Caused By: <openjpa-1.1.1-SNAPSHOT-r422266:1172209 nonfatal store error> kodo.jdo.DataStoreException: ORA-00942: table or view does not exist
 {prepstmnt 1198520800 INSERT INTO BEAXACMLAP (DOMN, REALMN, TYPEN, CN, XVER, CTS, WCN, WCI, WXF, XD, XS, MTS) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) [params=(String) MedRecDomain, (String) myrealm, (String) Policies, (String) urn:bea:xacml:2.0:entitlement:resource:type@E@Fmbean@G, (String) 1.0, (null) null, (null) null, (null) null, (Blob) oracle.sql.BLOB@315cb235, (InputStream) java.io.ByteArrayInputStream@3cb7cee4, (String) 3, (Timestamp) 2015-04-17 08:44:38.792] [reused=0]} [code=942, state=42000]
FailedObject: com.bea.common.security.store.data.XACMLAuthorizationPolicy@55ab3cda
    at org.apache.openjpa.jdbc.sql.DBDictionary.narrow(DBDictionary.java:4207)
    at org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException(DBDictionary.java:4171)
    at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:102)
    at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:72)
    at kodo.jdbc.kernel.BatchingPreparedStatementManager.flushInternal(BatchingPreparedStatementManager.java:214)
    Truncated. see log file for complete stacktrace
Caused By: org.apache.openjpa.lib.jdbc.ReportingSQLException: ORA-00942: table or view does not exist
 {prepstmnt 1198520800 INSERT INTO BEAXACMLAP (DOMN, REALMN, TYPEN, CN, XVER, CTS, WCN, WCI, WXF, XD, XS, MTS) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) [params=(String) MedRecDomain, (String) myrealm, (String) Policies, (String) urn:bea:xacml:2.0:entitlement:resource:type@E@Fmbean@G, (String) 1.0, (null) null, (null) null, (null) null, (Blob) oracle.sql.BLOB@315cb235, (InputStream) java.io.ByteArrayInputStream@3cb7cee4, (String) 3, (Timestamp) 2015-04-17 08:44:38.792] [reused=0]} [code=942, state=42000]
    at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:192)
    at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:57)
    at org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:866)
    at org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:269)
    at org.apache.openjpa.jdbc.kernel.JDBCStoreManager$CancelPreparedStatement.executeUpdate(JDBCStoreManager.java:1453)
    Truncated. see log file for complete stacktrace
>
<Apr 17, 2015 8:44:38 AM EDT> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Apr 17, 2015 8:44:38 AM EDT> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:

There are 1 nested errors:

weblogic.security.service.SecurityServiceRuntimeException: [Security:090399]Security Services Unavailable
    at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:917)
    at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054)
    at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
    at weblogic.security.SecurityService.start(SecurityService.java:141)
    at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
    at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
    at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

>
<Apr 17, 2015 8:44:38 AM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<Apr 17, 2015 8:44:38 AM EDT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<Apr 17, 2015 8:44:38 AM EDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>
weblogic.oraclesys.com /app01/oracle/Middleware/user_projects/domains/MedRecDomain/bin
weblogic=>cd $WL_HOME/server/lib
weblogic.oraclesys.com /app01/oracle/Middleware/wlserver_10.3/server/lib
weblogic=>ls

Probable Solution:

The database administrator needs to run the SQL script that creates the RDBMS tables in the datastore which are used by the RDBMS security store. A set of SQL scripts for creating and removing these tables for each supported RDBMS system is available in WL_HOME/server/lib
During the process of configuring the RDBMS security store via the Configuration Wizard, you are presented with the option of testing the database connection. Oracle strongly recommends using this option because it can verify that the connection is set up properly. If there were a problem with the database connection, you might not be able subsequently to boot the domain if the security providers that control access to that domain are unable to obtain the necessary security policies.

Create RDBMS Tables in the Security Datastore

Prior to booting the domain, the database administrator needs to run the SQL script that creates the RDBMS tables in the datastore used by the RDBMS security store. A set of SQL scripts for creating these tables for, and also removing them from, each supported RDBMS system is available in the following WebLogic Server installation directory:
WL_HOME/server/lib
When running the appropriate SQL script for the database serving as the RDBMS security store, be sure to specify the same connection properties, including the credentials of the user who has access, the database URL, etc., as specified for that RDBMS during domain creation.
Oracle 9i, 10g, 11g
———————————————
CREATE- rdbms_security_store_oracle.sql
REMOVE- rdbms_security_store_oracle_remove.sql
MS-SQL 2000, 2005
———————————————
rdbms_security_store_sqlserver.sql
rdbms_security_store_sqlserver_remove.sql
DB2 9.2, 9.5
———————————————
rdbms_security_store_db2.sql
rdbms_security_store_db2_remove.sql
Derby
———————————————
rdbms_security_store_derby.sql

Oracle Weblogic Blogs

https://blogs.oracle.com/WebLogicServer/

https://blogs.oracle.com/archbeat/