Tuesday, October 20, 2015

Backup of Weblogic Domain: pack command

 /u01/app01/Oracle/Middleware/user_projects/domains/demo_domain[oraapp@weblogic bin]$ pwd
/u01/app01/Oracle/Middleware/wlserver_10.3/common/bin

[oraapp@weblogic bin]$ ./pack.sh
-domain="/u01/app01/Oracle/Middleware/user_projects/domains/demo_domain"  -template="/u01/bkp/apps_bkp/17OCT15/demo_domain.jar"
-template_name="WebCenter Content Domain"
-log=/u01/bkp/apps_bkp/17OCT15/demo_domain.log
-log_priority=INFO
<< read domain from "/u01/app01/Oracle/Middleware/user_projects/domains/demo_domain"
>>  succeed: read domain from "/u01/app01/Oracle/Middleware/user_projects/domains/demo_domain"
<< write template to "/u01/bkp/apps_bkp/17OCT15/demo_domain.jar"
....................................................................................................
>>  succeed: write template to "/u01/bkp/apps_bkp/17OCT15/demo_domain.jar"
<< close template
>>  succeed: close template
[oraapp@weblogic bin]$
 

RMAN: Cold Backup Script

#!/bin/sh
# Set Oracle Environment for DB
#. ~/.bashrc
. ~/.bash_profile

echo
echo "`date` – Started `basename $0`"
echo

export NLS_DATE_FORMAT='DD-MON-YY HH24:MI:SS'
rman target / << EOF
spool log to rman_coldbackup.log;
set echo on
run
{
shutdown immediate;
startup mount;
allocate channel ch1 type disk format '/export/dbclone/DEV_RMAN_COLDBKP/full_backup_%d_%s_%p';
allocate channel ch2 type disk format '/export/dbclone/DEV_RMAN_COLDBKP/full_backup_%d_%s_%p';
allocate channel ch3 type disk format '/export/dbclone/DEV_RMAN_COLDBKP/full_backup_%d_%s_%p';
allocate channel ch4 type disk format '/export/dbclone/DEV_RMAN_COLDBKP/full_backup_%d_%s_%p';
allocate channel ch5 type disk format '/export/dbclone/DEV_RMAN_COLDBKP/full_backup_%d_%s_%p';
allocate channel ch6 type disk format '/export/dbclone/DEV_RMAN_COLDBKP/full_backup_%d_%s_%p';

BACKUP CURRENT CONTROLFILE format '/export/dbclone/DEV_RMAN_COLDBKP/ctrl_file_%d_%s_%p';
BACKUP AS COMPRESSED BACKUPSET DATABASE;

release channel ch1;
release channel ch2;
release channel ch3;
release channel ch4;
release channel ch5;
release channel ch6;

alter database open;
}
EOF
if [ $? -eq 0 ]
then
echo "====================="
echo "RMAN Backup Completed"
echo "====================="
else
echo "=================="
echo "RMAN Backup Failed"
echo "=================="
exit 1
fi

echo
echo "`date` - Finished `basename $0`"
echo

Monday, October 19, 2015

Enable Tracing in EBS: "Initialization SQL Statement – Custom"

A user event trace is very handy for tracing sql operations to debug various issues. The benefit of a user event trace is that it is linked to a specific user so that only code run by this user is traced. This makes diagnosis easier when compared to similar tracing methods at the database level where all user calls are traced.

Step 1
Login to Oracle Applications and select the System Administrator responsibility.
Choose Profile - System


Step 2
In the find profile field select the user which you wish to trace. In the profile field enter 'Initialization SQL Statement - Custom"
Select find


Step 3
In the find profile results form copy and paste the following into the 'user' field. Do not update the site level field.

BEGIN FND_CTL.FND_SESS_CTL('','', '', 'TRUE','','ALTER SESSION SET TRACEFILE_IDENTIFIER='||''''||'USERID' ||''''||' EVENTS ='||''''||' 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 '||''''); END;

Note: copy and paste the above as one line. Failure to paste the values properly or pasting the values with incorrect syntax will result in this user not being able to login.


Step 4 (Optional)
Save the profile option change. Stop and start the java virtual machine for the change to take effect
11i: use adapcctl.sh
r12: use adoacorectl.sh


Step 5
Login to the application and reproduce the issue. Then quickly log off. Try and avoid any un-necessary keystrokes as this simply makes the log files larger and the issue hard to pinpoint in the logs.


Step 6
login to unix/windows as the oracle user. Navigate to the user dump destination which us usually $ORACLE_HOME/admin/<context>/udump
You will see a series of trace files generated during the time of your issue reproduction. Tkprof all of the generated files and upload both the raw and tkprof files to your service request.

You can also check the user dump destination via the following:
sql> show parameter user_dump_dest;



Notes:

When using this profile, DO NOT turn trace on the menu, as doing so would actually turnoff Event 10046. Turning Event 10046 to any level, turns trace automatically.

Event 10046 level 1 is regular trace, level 4 is tracing with bind variables, level 8 is with database waits and level 12 with both,bind variables and database waits.

alter system set max_dump_file_size=UNLIMITED scope=both;

Monday, October 12, 2015

Java Keytool Essentials: Working with Java Keystores


Source: https://www.digitalocean.com/community/tutorials/java-keytool-essentials-working-with-java-keystores

Introduction

Java Keytool is a key and certificate management tool that is used to manipulate Java Keystores, and is included with Java. A Java Keystore is a container for authorization certificates or public key certificates, and is often used by Java-based applications for encryption, authentication, and serving over HTTPS. Its entries are protected by a keystore password. A keystore entry is identified by an alias, and it consists of keys and certificates that form a trust chain.
This cheat sheet-style guide provides a quick reference to keytool commands that are commonly useful when working with Java Keystores. This includes creating and modifying Java Keystores so they can be used with your Java applications.
How to Use This Guide:
  • If you are not familiar with certificate signing requests (CSRs), read the CSR section of our OpenSSL cheat sheet
  • This guide is in a simple, cheat sheet format--self-contained command line snippets
  • Jump to any section that is relevant to the task you are trying to complete (Hint: use the Contents menu on the bottom-left or your browser's Find function)
  • Most of the commands are one-liners that have been expanded to multiple lines (using the \ symbol) for clarity

Creating and Importing Keystore Entries

This section covers Java Keytool commands that are related to generating key pairs and certificates, and importing certificates.

Generate Keys in New/Existing Keystore

Use this method if you want to use HTTP (HTTP over TLS) to secure your Java application. This will create a new key pair in a new or existing Java Keystore, which can be used to create a CSR, and obtain an SSL certificate from a Certificate Authority.
This command generates a 2048-bit RSA key pair, under the specified alias (domain), in the specified keystore file (keystore.jks):
keytool -genkeypair \
        -alias domain \
        -keyalg RSA \
        -keystore keystore.jks
If the specified keystore does not already exist, it will be created after the requested information is supplied. This will prompt for the keystore password (new or existing), followed by a Distinguished Name prompt (for the private key), then the desired private key password.

Generate CSR For Existing Private Key

Use this method if you want to generate an CSR that you can send to a CA to request the issuance of a CA-signed SSL certificate. It requires that the keystore and alias already exist; you can use the previous command to ensure this.
This command creates a CSR (domain.csr) signed by the private key identified by the alias (domain) in the (keystore.jks) keystore:
keytool -certreq \
        -alias domain \
        -file domain.csr \
        -keystore keystore.jks
After entering the keystore's password, the CSR will be generated.

Import Signed/Root/Intermediate Certificate

Use this method if you want to import a signed certificate, e.g. a certificate signed by a CA, into your keystore; it must match the private key that exists in the specified alias. You may also use this same command to import root or intermediate certificates that your CA may require to complete a chain of trust. Simply specify a unique alias, such as root instead of domain, and the certificate that you want to import.
This command imports the certificate (domain.crt) into the keystore (keystore.jks), under the specified alias (domain). If you are importing a signed certificate, it must correspond to the private key in the specified alias:
keytool -importcert \
        -trustcacerts -file domain.crt \
        -alias domain \
        -keystore keystore.jks
You will be prompted for the keystore password, then for a confirmation of the import action.
Note: You may also use the command to import a CA's certificates into your Java truststore, which is typically located in $JAVA_HOME/jre/lib/security/cacerts assuming $JAVA_HOME is where your JRE or JDK is installed.

Generate Self-Signed Certificate in New/Existing Keystore

Use this command if you want to generate a self-signed certificate for your Java applications. This is actually the same command that is used to create a new key pair, but with the validity lifetime specified in days.
This command generates a 2048-bit RSA key pair, valid for 365 days, under the specified alias (domain), in the specified keystore file (keystore.jks):
keytool -genkey \
        -alias domain \
        -keyalg RSA \
        -validity 365 \
        -keystore keystore.jks
If the specified keystore does not already exist, it will be created after the requested information is supplied. This will prompt for the keystore password (new or existing), followed by a Distinguished Name prompt (for the private key), then the desired private key password.

Viewing Keystore Entries

This section covers listing the contents of a Java Keystore, such as viewing certificate information or exporting certificates.

List Keystore Certificate Fingerprints

This command lists the SHA fingerprints of all of the certificates in the keystore (keystore.jks), under their respective aliases:
keytool -list \
        -keystore keystore.jks
You will be prompted for the keystore's password. You may also restrict the output to a specific alias by using the -alias domain option, where "domain" is the alias name.

List Verbose Keystore Contents

This command lists verbose information about the entries a keystore (keystore.jks) contains, including certificate chain length, fingerprint of certificates in the chain, distinguished names, serial number, and creation/expiration date, under their respective aliases:
keytool -list -v \
        -keystore keystore.jks
You will be prompted for the keystore's password. You may also restrict the output to a specific alias by using the -alias domain option, where "domain" is the alias name.
Note: You may also use this command to view which certificates are in your Java truststore, which is typically located in $JAVA_HOME/jre/lib/security/cacerts assuming $JAVA_HOME is where your JRE or JDK is installed.

Use Keytool to View Certificate Information

This command prints verbose information about a certificate file (certificate.crt), including its fingerprints, distinguished name of owner and issuer, and the time period of its validity:
keytool -printcert \
        -file domain.crt
You will be prompted for the keystore password.

Export Certificate

This command exports a binary DER-encoded certificate (domain.der), that is associated with the alias (domain), in the keystore (keystore.jks):
keytool -exportcert
        -alias domain
        -file domain.der
        -keystore keystore.jks
You will be prompted for the keystore password. If you want to convert the DER-encoded certificate to PEM-encoding, follow our OpenSSL cheat sheet.

Modifying Keystore

This section covers the modification of Java Keystore entries, such as deleting or renaming aliases.

Change Keystore Password

This command is used to change the password of a keystore (keystore.jks):
keytool -storepasswd \
        -keystore keystore.jks
You will be prompted for the current password, then the new password. You may also specify the new password in the command by using the -new newpass option, where "newpass" is the password.

Delete Alias

This command is used to delete an alias (domain) in a keystore (keystore.jks):
keytool -delete \
        -alias domain \
        -keystore keystore.jks
You will be prompted for the keystore password.

Rename Alias

This command will rename the alias (domain) to the destination alias (newdomain) in the keystore (keystore.jks):
keytool -changealias \
        -alias domain \
        -destalias newdomain \
        -keystore keystore.jks
You will be prompted for the keystore password.

Conclusion

That should cover how most people use Java Keytool to manipulate their Java Keystores. It has many other uses that were not covered here, so feel free to ask or suggest other uses in the comments.

Sunday, October 11, 2015

Configuring SSL for Oracle Weblogic (and OFMW)

https://jvzoggel.wordpress.com/2011/12/16/configuring-ssl-for-oracle-weblogic-and-ofmw/


We have a very basic domain: 3 physical servers, 1 containing the admin server, 2 containing 2 managed servers each, running in a cluster. Nice picture to follow. :)
name of the servers: server01 (Admin) / server02 (MS1+MS2) / server03 (MS3+MS4)

Step 0 – The Weblogic Demo Certificates (the reason why)

If you ever checked your Weblogic production logs and found the Security Alert below, then your Weblogic domain has a huge gaping security hole. Even if you don’t want SSL communication for your webservices or applications. The internal (administrative) processes in your Weblogic domain still relies on the default demotrust and with this everyone can access your domain.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
</pre>
&nbsp;
 
<Alert> <Security> <SERVER02> <rbx_dev_soa_ms01> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <BEA-090152> <Demo trusted CA certificate is being used in production mode: [
[
  Version: V3
  Subject: CN=CACERT, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
  Signature Algorithm: MD5withRSA, OID = 1.2.840.113549.1.1.4
 
  Key:  Sun RSA public key, 512 bits
  modulus: 95501928778692415440753....
  public exponent: 65537
  Validity: [From: Thu Mar 21 21:12:27 CET 2002,
               To: Tue Mar 22 21:12:27 CET 2022]
  Issuer: CN=CACERT, OU=FOR TESTING ONLY, O=MyOrganization, L=MyTown, ST=MyState, C=US
  SerialNumber: [    33f10648 fcde0deb 4199921f d64537f4]
 
Certificate Extensions: 1
[1]: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  Key_CertSign
]
 
]
  Algorithm: [MD5withRSA]
  Signature:
0000: 9D 26 4C 29 C8 91 C3 A7   06 C3 24 6F AE B4 F8 82  .&L)......$o....
0010: 80 4D AA CB 7C 79 46 84   81 C4 66 95 F4 1E D8 C4  .M...yF...f.....
0020: E9 B7 D9 7C E2 23 33 A4   B7 21 E0 AA 54 2B 4A FF  .....#3..!..T+J.
0030: CB 21 20 88 81 21 DB AC   90 54 D8 7D 79 63 23 3C  .! ..!...T..yc#<
 
] The system is vulnerable to security attacks, since it trusts certificates signed by the demo trusted CA.>

Step 1 – generate the identity java keystore

We first generate a java keystores containing a private key for each physical server, since SSL hostname verifiation is based on machine names. We use “welcome1” as password here, but make sure to choose a complex password here, use it during these steps and then lock it safely away.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
keytool.exe -genkey -keysize 2048 -keyalg RSA -alias server01 -keystore server01_identity.jks
 
Enter keystore password: welcome1
 Re-enter new password: welcome1
 What is your first and last name?
   [Unknown]:  server01
 What is the name of your organizational unit?
   [Unknown]:
 What is the name of your organization?
   [Unknown]:  Rubix
 What is the name of your City or Locality?
   [Unknown]:
 What is the name of your State or Province?
   [Unknown]:
 What is the two-letter country code for this unit?
   [Unknown]:  NL
 Is CN=server01, OU=, O=Rubix, L=, ST=, C=NL correct?
   [no]:  yes
<br>

Step 2 – generate signing request

1
keytool.exe -certreq -alias server01 -file server01.csr -keystore server01_identity.jks</div>
Repeat step 1 + 2 for each physical server, replacing the -alias parameter, -keystore parameter and CN name with the correct server0* value.
 

 

Step 3 – the Certificate Authority (CA)

Now comes the tricky part that you probably need some outside forces. You will need the correct Certificate Authority to sign your certificate requests (the .CSR files you generated). You can create your own CA and self-sign them, but I would not recommend that unless you know what you are doing and understand the consequences. You could use an external Internet CA provider, but this becomes very costly and time consuming when you need such an external provider to sign every SSL enabled server in your landscape. The best situation for us would be if the current organization already has an internal CA provider, especially when the rootCA is trusted by the servers and machines in your landscape.
 
Let’s assume however that you receive the signed response, including the public keys of the rootCA and a intermediateCA. CA instances often use a intermediate CA to authorize certificates, so if they have “problems” they can withdraw this intermediate CA and don’t need to withdraw the whole RootCA making it useless.
So the situation should be that you know hold 3 PEM/DER files.
 

 

Step 4 – importing the CA response

Import the certificates in your keystore, starting with the rootCA, then the intermediateCA, then the specific server alias. Example for server01:
1
2
3
4
5
keytool -importcert -trustcacerts -alias rootca -file rootCA.cer -keystore server01_identity.jks
 
keytool -importcert -alias intermediateca -file intermediateCA.cer -keystore server01_identity.jks
 
keytool -importcert -alias server01 -file server01.cer -keystore server01_identity.jks
Repeat these steps for server02 and server03. So we know have 3 identity keystores for each server containing the signed private key and it’s certificate chain.
 

 

Step 5 – generate the trust java keystore

Now let’s generate a truststore for each server. Altough you can use the identical keystore for both identity and trust, it’s always a good practice to separate them. The identity keystone contains the private key and normally never changes during it’s lifetime, so you should secure it with a complex password. The trust store contains the certificates that you trust and as your middleware landscape grows, this file might grow and more administrators will need access to it.
 
Example generating the truststore for server01, repeat for 02 & 03.
 
1
2
keytool -importcert -trustcacerts -alias rootca -file rootCA.cer -keystore server01_trust.jks
keytool -importcert -alias intermediateca -file intermediateCA.cer -keystore server01_trust.jks
 

Step 6 – copy the java keystore files to each server

 Make sure that each server had the files stored on disk. We use the windows server example “c:\oracle\security” in this blogpost. So server01 should hold:
– c:\oracle\security\server01_identity.jks
– c:\oracle\security\server01_trust.jks
 

 

Step 7 – configure the Node Manager for SSL

The nodemanger by default uses the demoIdentity and demoTrust keystores.
So on each server hosting a managed server (server02 and server03) we will need to change this.
Open the nodemanager.properties and add the following lines:
 
1
2
3
4
5
6
7
8
9
### SSL config
KeyStores=CustomIdentityAndCustomTrust
# CustomIdentityKeyStoreType=JKS
CustomIdentityKeyStoreFileName=c://oracle//security//server01_identity.jks
CustomIdentityKeyStorePassPhrase=welcome1
CustomIdentityAlias=server01
CustomIdentityPrivateKeyPassPhrase=welcome1
# CustomTrustKeyStoreType=JKS
CustomTrustKeyStoreFileName=c://oracle//security//server01_trust.jks
Restart the NodeManager to activate the changes. The passwords in the properties file will automatically become encrypted.
 
Possible error:
1
For server rbx_dev_soa_ms03, the Node Manager associated with machine server03 is not reachable. All of the servers selected are currently in a state which is incompatible with this operation or are not associated with a running Node Manager or you are not authorized to perform the action requested. No action will be performed.
Possible solution:
Your AdminServer can not communicate with a node manager. Make sure it’s running, check Environment -> Machines in the Weblogic console if the machine has state reachable, check if the address configured there, and the address in the nodemanager.properties and the CN-name used in SSL have the same value, check if Weblogic and Nodemanager use the same SSL configuration (trust+identity).
 

 

Step 8 – configure Weblogic for SSL

Log in the Weblogic console, in our example: http://server01:7001/console
 
 
Step 8.1 – configure Weblogic for SSL, enable SSL
 
Log in the Weblogic console (in our case http://server01:7001/console) and make sure that each server has the SSL port enabled. You can find the setting “SSL Listen Port Enabled” for each server under “Configuration -> General”. Enable it and make sure you use a correct and free port.
 

Step 8.2 – configure Weblogic for SSL, changing keystores

For each server (Admin + MS) we have to change the KeyStore configuration.
So for each server go to: Configuration -> Keystores
 
  • Keystores -> “Change” = “Custom Identity and Custom Trust”
  • Custom Identity Keystore = c:/Oracle/security/serverXX_identity.jks
  • Custom Identity Keystore Type = JKS
  • Custom Identity Keystore Passphrase = welcome1
  • Custom Trust keystore = c:/Oracle/security/serverXX_trust.jks
  • Custom Trust Keystore Type = JKS
  • Custom Trust Keystore Passphrase = welcome1
 
 

Step 8.3 – configure Weblogic for SSL, changing identity:

For each server (Admin + MS) we have to change the SSL configuration.
So for each server go to: Configuration -> SSL
 
  • Private Key Alias = serverXX
  • Private Key Passphrase = welcome1
 

Step 9 – Weblogic start scripts:

By default Weblogic uses the DemoTrust.jks truststore. Since we changed this in the console configuration you would hope this would be enough. However the SetDomainEnv.sh / SetDomainEnv.cmd script still contains a reference. So for each server:
 
  • Go to %domain_home%/bin/
  • Open the SetDomainEnv script
  • Search the argument: -Djavax.net.ssl.trustStore=
  • Make sure the value is not referring to the demo trust.jks but “c:/oracle/security/serverXX_trust.jks
When you are finished restart your whole Weblogic environment (Admin + MS).
 

Step 10 – Configure Enterprise Manager:

This took me some research and the Oracle Forums to get it working. Navigate to the enterprise manager console (/em)
 

Step 10.1 – Security Provider Configuration:

I’m actually not really sure if this step is still necessary in 11g, but hey .. why not be sure:
 
  • Click Weblogic -> right-click domain
  • Select Security -> Security Provider Configuration -> KeyStore Configure
  • Keystore Path = c:/oracle/security/server01_identity.jks
 

Step 10.2 – SOA Infra properties

 
  • Click SOA -> right-click soa-infra
  • Select SOA Administration -> Common Properties
  • Click More SOA Infra Advanced Properties
  • Keystore Location = c:/oracle/security/serverXX_identity.jks

Step 10.3 – Credential Mapping:
The next step is to configure the credential mapping keys

  • Click Weblogic -> right-click domain name
  • Select Security -> Credentials
  • Create a Map named “SOA”
  • Click Create a key
  • Key = KeyPassword
  • User Name = KeyPassword
  • Password = welcome1
  • Click Create Key:
  • Key = KeyStorePassword
  • User Name = KeyStorePassword
  • Password = welcome1
 
Possible error:
1
2
3
4
5
6
7
8
9
10
11
</div>
<div>
 
<Warning> <oracle.as.jmx.framework.MessageLocalizationHelper> <J2EE JMX-46041> <The resource for bundle "oracle.soa.management.config.bpel.mbeans.MessageBundle_en" with key "oracle.soa.BPELConfigBean.auditFlushByteThreshold" cannot be found.>
INFO: SSLSocketFactoryManagerImpl.getKeystoreLocation SOA Keystore location: c:/oracle/security/server04_identity.jks
INFO: SSLSocketFactoryManagerImpl.getKeystorePassword Obtained null or empty keystore password
INFO: SSLSocketFactoryManagerImpl.getKeyPassword Obtained null or empty key password
INFO: SSLSocketFactoryManagerImpl.getSSLSocketFactory Could not obtain keystore location or password
 
</div>
<div>
 
Possible solution:
You did not configure the SOA key mapping correctly as mentioned above in 9.3

 

Conclusion:

Enabling SSL takes a lot of steps, and we didn’t even mentioned it for enabling SSL client identity in OSB (Service Key Provider) and communication with an LDAP over SSL. But there’s more to come I guess.
If you have any remarks please let me know in the comments. Hope it helps. :)
 
 

Links:

WebLogic Server - identity vs trust keystores

Source:  http://one-size-doesnt-fit-all.blogspot.com/2009/09/weblogic-server-identity-vs-trust.html

Summary:
Identity keystore:
This will be used to store the server certificate(private key/digital certificate pairs). When the client contacts server the digital certificate presented in this keystore will be sent.

Trust Keystore:
This will contain all the certificates for the trusted partners(ie partners). When server connects with the partner it will use this key store.


In computing most technologies have lots of terms and acronyms to learn, it's par for the course, you get used to it. However in computer security the frustration is multiplied as there are often many different terms that mean the same thing. It makes implementing security hard, because understanding it is hard, and I'm not surprised why security is considered badly implemented because the average Joe will struggle.

I've been trying recently to get straight in my head what is stored in the WLS identity and trust keystores, and what the difference between identity and trust is anyhow. Thanks to kind assistance from Gerard Davison, I think I can now post my understandings, and as usual, hopefully the post is helpful to other readers. As noted however security to me is a difficult area, and so be sure to check the facts here, your mileage with this post may vary.

The following WLS documentation attempts to explain the concepts of identity and trust:

http://download.oracle.com/docs/cd/E12839_01/web.1111/e13707/identity_trust.htm#i1170342


...in ripping out one of the core paragraphs, with a slight rewrite of my own we can see the concept of identity, and how it relates to the public and private keys:

"The public key is embedded in a digital certificate with additional information describing the owner of the public key, such as name, street address, and e-mail address *as well as the hostname*. *Along with this the digital certificate containing the public key, and the separate related private key, provide identity for the server*."

...ultimately to identify the server, to assert the server is who the server says it is.

The digital certificate containing the public key is also referred to as the "server certificate", as for example in 1-way-SSL traffic between the server and client, the server certificate containing the public key is what is initially passed to the client.

There is a missing piece in the puzzle. Regardless that the digital certificate states the owner of the public key, their name and so on, how does a client know that the "identity" asserted by the digital certificate is true? That's where Certificate Authorities (CAs) come in.

Ignoring self signed digital certificates, a typical digital certificate used on the internet containing the public key and owner details is signed by a trusted CA who has verified the identity of the owner. Presumably when purchasing digital certificates from CAs, this is what some of the cost covers, the CAs research into ensuring that the identity details embedded in the digital cert are actually true.

At runtime on receiving the digital certificate, the client checks the CA and if the CA is one that the client trusts (or a CA in a chain of trusted CAs), then the identity of the server is established/verified.

Thus the "identity" of the server is established by what's stored in the "identity" keystore, and its contents are what are farmed out to clients establishing secure connections with the server, who then verify the supplied digital certificate's CA against the clients own list of trusted CAs. The "identity keystore" is also referred to as the "server keystore", because it establishes the server's identity (ie. I am who I say I am).

WLS side note: As mentioned the digital certificate also includes the host name of the server, or in other words the digital certificate is pegged to that server and that server alone. This implies on that server with its relating digital certificate, *all* of the applications will share that single digital certificate for secure communications. Occasionally a requirement will arise where each application must have its own digital certificate. In WLS because keystores are configured under an individual WLS "managed server", if you have two separate applications, it is not possible to use separate digital certificates for each in one managed server. The solution is to create another managed server with its own keystores.

WLS web service side note: Following on from the previous side note, for web services that use in-message encryption and digital signatures, there is often the requirement for multiple different digital certificates. Under WLS to provision the WS-Security model, WLS has a separate Web Service Security Configuration (WSSC) to provision this setup.

Finally regarding the trust keystore, what is its job in all of this? The trust keystore is typically used for storing CA digital certificates, essentially the CAs who will be used to check any digital certificates that are given to the server at runtime (just the same as the client did above). In the standard 1-way-SSL between a client and the WLS server, the trust keystore doesn't come into the equation as the client has its own trust keystore (containing the CAs) and the server has nothing to verify. Yet in the case of mutual SSL (aka. 2 way SSL) between the client and server, the client and server actually swap each other digital certificates to establish identity of both parties, and in this case the server must be able to test the identity of the client through the CA of the client's digital certificate.

Mutual SSL side note: the setup of mutual SSL is more complicated than this. Readers are advised to refer to the following Oracle article.