I have been recently working on a project to customize OAM, sounds easy right? it is if you have the correct information, you have to pay attention to the details so you know what to look for and where you to find it:

Considerations:

1. Custom pages does not mean Login page only
2. Change password and dependants are also included
3. Version of OAM 11.1.2.0.0
4. Custom Pages technology JSP
5. Cluster environment.

OAM Setup:

1. Configure WebGate (Not part of this post)
2. Create a New Authentication Scheme
3. Update Password Service URL

New Authentication Scheme

• Login to  oamconsole http:server:port/oamconsole
• Create a New Authentication Scheme

The important bit:

• Challenge Method: Form
• Challenge URL:  Your login page
• Context Type: external

Custom Pages

OAM provides a base project so you don’t have to start everything from scratch or at least it will give you an idea of the proper structure your project should have. You can use any web echnology you prefer, I’ll show the solution with JSP.

Login Page

 Get values from Header and from URL:

<%
 //Request OAM_REQ parameter from Header OAM Version 11.1.2
 String oamReq = request.getHeader("OAM_REQ");
 //Error code sent by OAM in case of failure
 String errCode = request.getHeader("p_error_code"); 

 String requestId = request.getParameter("request_id");
 String resourceUrl = request.getParameter("resource_url");
%>

The Form:

 <form name="ssoFrm" id="login" autocomplete="off"
 action="/oam/server/auth_cred_submit"
 method="POST">

 <label for="username">Username</label>
 <input name="userid" type="text" id="username"
 class="required"></input>

 <label for="password">Password</label>
 <input name="password" type="password" id="password"
 class="required" size="36"></input>

 <input name="request_id" value="<%=requestId%>"
 type="hidden"></input>

 <% if(oamReq != null && !"".equals(oamReq) ) { %>

 <input id="serverctx" name="OAM_REQ" type="hidden"
 value='<%=oamReq%>'/>
 <% }%>

<%
if(!errCode.equals("")){
//You can map this value in a Resource Bundle
 errorMessage=errCode;
 }
 %>
 <%=errorMessage%>

 <button id="sSubmit" name="sSubmit" type="submit">
Login</button>
 </form>

Change Password

Make sure you change the Password Service URL to point to your custom page:   Change Password  Page must contain:

 
<%
 //Set the Expires and Cache Control Headers
 response.setHeader("Cache-Control", "no-cache, no-store");
 response.setHeader("Pragma", "no-cache");
 response.setHeader("Expires", "0");
 response.setContentType("text/html; charset=UTF-8");
%>

There are 2 considerations we need to take into account for this page:

1. Get policy rules from OAM
2. In case of errors being present get password error messages

 
 <%
String reqId = request.getParameter(GenericConstants.REQUEST_ID);
//Password Policies
String rules=request.getParameter("ruleDesc");
//In case of error, gets the error messages
String errorCode=request.getHeader("p_error_code");
String errorMsg=request.getParameter("p_sec_error_msg");
//Type of the request.
String type= request.getParameter("type");
//In case of not following Password Policies, errors thrown back.
String rejectedRules=request.getParameter("rejectedRuleDesc");
%>

 
  <%
 if(type.equals("CHANGE_ACCEPT")){
 String redirectURL = "LandingPage.jsp";
 response.sendRedirect("/"+redirectURL);
 }
%>

 
<%
 String rejectedMsg = "";
 if(rejectedRules != ""){
 if(rejectedRules.indexOf('~') != -1) {
 String[] rejected = rejectedRules.split("~");
 for(String eachResult : rejected) {
 if(eachResult.indexOf(":") != -1) {
 String messageKey = eachResult.substring(0, eachResult.indexOf(":"));
 String placeHolderValue = eachResult.substring(eachResult.indexOf("::") + 2, eachResult.length());
 String msgString = props.getProperties().getProperty(messageKey);
 String message = MessageFormat.format(msgString, placeHolderValue);
 String displayValue = placeHolderValue;
 rejectedMsg +=message + "";
 }
 else {
 String resourceBundleKey = eachResult;
 String msgString = props.getProperties().getProperty(resourceBundleKey);
 if(null != msgString){
 rejectedMsg += msgString +"";
 }
 }
}
}
}
%>

Log the user in when the change is accepted:

<%
if(type.equals("CHANGE_ACCEPT")){
 %>
<script type="text/javascript">
 window.document.forms[0].submit();
 </script>
 <%
 }
 %>

References:
http://docs.oracle.com/cd/E40329_01/dev.1112/e27134/custpages.htm