To start off, do not implement ISA Server in a workgroup, unless you really, really want to. I would prefer to leave it part of the domain in most scenarios. This post is not to debate “To Join or Not to Join” the domain. ISA Server MVP Tom Shinder has written a beautiful article listing out good reasons why ISA Server SHOULD be a domain member.
Now, what if you went ahead and deployed your ISA Server in a workgroup due to technical/security (aww come on)/nagging manager reasons and had to set up LDAPS for authentication between your ISA Server and domain controller? Of course the normal Active Directory authentication will not work from the ISA Server computer because it is not a member of the domain, hence we need to use LDAP or LDAPS (Secure LDAP).
For certain features like the ISA Server 2006 FBA Change Password option to work, configuring LDAPS properly is a must.
To use LDAPS, you need to install a server certificate on the LDAP server (domain controller in this case), and the root certificate from the CA that issued the server certificate should be installed on the computer running ISA Server. Here’s an article on how to do that (yes, it applies to Microsoft CA as well).
Once you’ve done all this stuff, you can check if all is well by trying to run the LDP.exe tool from ISA Server as mentioned in the article. Pay special attention to the issues section at the end of the article.
If you still face problems, and suspect the certificates may be the problem, open Internet Explorer on the ISA Server computer, and browse to https://myldapserver.domain.local:636. If you have configured everything correctly, you should not receive any certificate errors. Of course, you will eventually get ‘Page cannot be displayed’. In addition, The following warning can be ignored:
Once this works without errors, and you still face problems, run the ISA Server Best Practices Analyzer and look for any certificate related issues and resolve them.