"Failure [0xC3EC7814] Pool is not ready" while adding OCS Server to Pool

I was trying to get Office Communications Server 2007 (OCS) to work on our environment for the past few days and ran into some challenging issues.

OCS 2007 is being installed on Windows Server 2003 x64 R2 with SP2, and the SQL 2005 back end is on a separate machine. Co-existing the two on the same box is not supported.

Our environment used to have an LCS 2005 Pool, which had been decommissioned while setting out on the OCS adventure. The schema preparation, creating an enterprise pool, and configuring the pool – all went through successfully. The show-stopper was the ‘adding server to pool’ part.

During the ‘Add Server to Pool‘ wizard operation, the installation of the program files and some additional configuration succeeded. However, when it was time to activate the server, all hell broke loose and the operation failed.

When I opened the logs, The ‘add server to pool’ log file showed:

Failure
[0xC3EC796C] One or more errors occurred during execution of the wizard; the wizard was unable to complete successfully. Please check the log file for more information.

Drilling further down into the log, I found that the problem was while activating the server. I clicked on the link to open the Activation logs, and found the following generic error:

Failure
[0xC3EC7814] Pool is not ready.

After ironing out every logical possibility, days of searching the net, and after asking in the MVP Private newsgroups with no luck, I decided to contact Product Support.

A few hours later, the problem turned out to be remnants of the past LCS pools that were still lurking in the Active Directory. We opened up ADSIEDIT, navigated to the Domain partition –> System container –> Policies container. Over here, we found the GUID of the LCS 2005 pools and deleted them manually. Since we didn’t have any more LCS 2005 servers, we didn’t need them anyway.

Due to this, OCS 2007 Pools were not actually being created in AD, even though the wizard ran successfully. After a successful pool creation, OCS 2007 Pools should be seen in Domain partition –> System container –>  Microsoft –> Rtc Service –> Pools. However, the Pools container was apparently empty even after I created OCS 2007 pools.

Once we got rid of the LCS 2005 pool GUIDs mentioned earlier:

1. We removed the pool we created using the command lcscmd.exe /forest /action:RemovePool /poolname:<poolname> /force

2. Deleted the SQL 2005 database and log files for this pool

3. Re-created the pool from scratch starting with the ‘Create Enterprise Pool‘ wizard.

Server activation was successful and we are back in business!

How to understand an Exchange Server 2007 NDR

Here’s one for users and IT support personnel who sometimes have problems understanding what exactly an email non-delivery report (NDR) is trying to convey. Messaging experts, please excuse.

In Exchange Server, all NDRs returned to the sender appear to come from the local Exchange Server of your organization and not from the remote recipient’s mail server – even if the problem is at the receiving end. If the mail is received at the remote server and an error occurs during further re-routing/relaying, then the NDR might not appear to come from your organization’s Exchange Server. The NDR is formatted in an easy-to-read email message by the Exchange server in your organization and is sent back to the sender.

So, how easy is it to understand an NDR?

At first sight of the new, well-designed NDR of Exchange Server 2007, most users and non-email administrators tend to think that the problem is always on the local Exchange Server. To add to the confusion the NDR contains the words “Sent by Microsoft Exchange Server 2007″ and “Generating server: “.

Here are some tips:

• The text marked in blue is not what’s important. It will always show YOUR organizations edge transport server – unless the error occurred at a subsequent mail re-routing operation at the destination.
• Pay attention to the part that I’ve marked in red. The part labeled (1) is more important. It gives you an overview of what’s wrong – but need not always give you the full picture.
• The part labeled (2) is the server on which the error occurred. If this doesn’t look like one of the servers inside your organization, the problem is most likely not at your end.
• The part labeled (3) is the error reported by the server mentioned in (2).
• Part (4) shows the flow of the message between various servers both within and outside your organization. All it takes is a little effort to understand what’s going on.
• Trust your email servers . Don’t always think the problem is at your end, even if it looks like your server is reporting the error. Make an earnest attempt and apply some educated logic to figure out where the problem lies.

The more users you train on how to read NDRs the lesser helpdesk calls you will get. I’ve seen that sometimes very simple NDRs like the following get escalated all the way to the email administrator as an “email problem”:

Your message did not reach some or all of the intended recipients.

Subject: RE: Acquisition of Yahoo Sent: 4/15/2008 11:09 PM
The following recipient(s) could not be reached:
shijaz@2hotmail.com on 4/15/2008 11:09 PM The e-mail account does not exist at the organization this message was sent to. Check the e-mail address, or contact the recipient directly to find out the correct address.

The solution? Teach the user how to type an email address correctly .

Certificate name mismatch in Outlook while running Exchange Server 2007

If you have installed Exchange 2007 Client Access Servers in your organization, and if you have installed your SSL certificates (even commercial ones) on IIS, Outlook MAPI users may receive ‘Security Alert’ messages similar to the above in Outlook.

The name on the security certificate is invalid or does not match the name of the site.

This is because of the certificate that you have installed on IIS. Outlook 2007 MAPI clients use Client Access Servers for the Autodiscover service. The Autodiscovery web service (a virtual directory on the Client Access Server) is used for automatically finding the mailbox server for a given user. When the Autodiscover service is accessed by Outlook, and the name on the security certificate installed in IIS doesn’t match the internal FQDN of the Client Access server (CAS), this error results.

Suppose your company’s public domain name is mycompany.com. You may have obtained a certificate for webmail.mycompany.com and installed on the IIS of your Client Access Server. This is correct because users on the internet will type the public name.

However, the same IIS on the CAS is hosts the Autodiscover virtual directory as well and this certificate applies. Your internal domain name might be mycmpny.local and the client access server FQDN might be CAS1.mycmpny.local. Outlooks 2007 uses this internal name to connect to Autodiscovery, and hence the mismatch error.

To fix this problem, open Exchange Management Shell and type the following commands:

Set-ClientAccessServer -Identity CAS1 -AutodiscoverServiceInternalUri https://webmail.mycompany.com/autodiscover/autodiscover.xml

Set-WebServicesVirtualDirectory -Identity “CAS1\EWS (Default Web Site)” -InternalUrl https://webmail.mycompany.com/ews/exchange.asmx

Set-OABVirtualDirectory -Identity “CAS1\oab (Default Web Site)” -InternalUrl https://webmail.mycompany.com/oab

Set-UMVirtualDirectory -Identity “CAS1\unifiedmessaging (Default Web Site)” -InternalUrl https://webmail.mycompany.com/unifiedmessaging/service.asmx

Pay attention to the text in red, you will need to change it to reflect your server’s running parameters. Recycle the MSExchangeAutodiscoverAppPool. Your users should no longer receive the security alert.

Error opening Address Book in Outlook Web Access

While trying to open Outlook Web Access hosted on an Exchange Server 2007 Client Access Server, I get an error stating that Outlook Web Access could not connect to Active Directory, followed by a detailed stack trace:

Request Url: https://owaURL/owa/forms/premium/DirectoryView.aspx?ae=AddressList&t=Recipients&a=
User host address:
User: someone
EX Address: /o=MYORG/ou=MYOU/cn=RECIPIENTS/cn=SOMEONE
SMTP Address: someone@mydomain.com
OWA version: 8.0.685.24
Mailbox server: mail.mydomain.com


My initial search fetched a Microsoft KB article 919166, which deals with exactly the same problem. However, unlike the conditions mentioned in the article, the locale on my domain controller and Exchange servers are the same and my domain controller has Windows Server 2003 Service Pack 2 which supersedes the mentioned hotfix.

So I called Microsoft, and it turned out to be related more to KB886683 while OWA is querying the Global Catalog. To fix the problem:

1. Open ADSIEDIT.
2. Navigate to CN=Configuration, CN=Services, CN=Windows NT, CN=Directory Service
3. Right click on CN=Directory Service and choose Properties.
4. Edit the multi-valued attribute msDS-Other-Settings
5. If you see a string value DisableVLVSupport=1, remove it and change it to DisableVLVSupport=0 and add it back. Click OK all the way out.

Replicate the changes across all your domain controllers. You should now be able to open your address book.

Failure in its many forms

I was fiddling around with ISA Server in my test lab, when I came across this rather unusual and unexpected error message:

Puzzled, I decided to click the ‘Details’ button, with the intention of getting a closer look.

No further comments.

OWA calendaring issue "This action cannot be performed"

This post discusses a common Outlook Web Access (OWA) calendaring issue reported at most forums. I could not find a satisfactory answer posted anywhere, including eventID.net, so I thought I’d share the knowledge.

A description of the problem:

When a user tries to save an appointment/calendar item or responds to a meeting request using Outlook Web Access provided by Exchange Server 2003, he/she sees the following error:

This action can’t be performed.

The user is unable to save changes to his/her calendar using OWA. Additionally, he/she may see the following error while trying to dismiss or snooze reminder popups in OWA:

One or more of your reminders cannot be snoozed or dismissed.

You will also get the errors in the event log of the Exchange server such as “Calendaring agent failed to save appointment.”

This can occur due to one or more of the following conditions:

1. As per Microsoft KB 310440, check if the required registry keys are intact.
2. Make sure your antivirus software is excluded from doing realtime scanning on Exchange database/log files.
3. Open the affected user objects attributes in ADSIEDIT. Make sure that the legacyExchangeDN attribute is in order.

To elaborate on the third condition, make sure your legacyExchangeDN attribute looks something like this:

/o=OrganizationName/ou=First Administrative Group/cn=Recipients/cn=shijaz

and not:

/o=OrganizationName/ou=First Administrative Group/cn=Recipientsshijaz

(missing “/cn=” before the user alias)

Observation:

If you move a mailbox with a malformed legacyExchangeDN attribute to Exchange Server 2007, the user will not receive meeting requests/updates. These will get stuck up on the queue with a 430 4.2.0 “STOREDRV.Deliver.Exception:MAPIExceptionCanNotComplete” error.

"Setup failed to install ADAM in replica mode"

If you have already have ISA Server 2006 Enterprise Edition installed and you are trying to installing ISA Server on another server and configuring it as a replica of the Configuration store, you may get the following error on Windows Server 2003 R2:

Setup then exits and you are unable to complete the installation. This usually happens if there was a previous failed installation from the machine that you’re trying to join to the array. You will need to cleanup the values related to the server you’re installing from the ADAM installed on your first configuration store, which stores config information for the array.

A simple solution to this is to ensure that both nodes are running Windows Server 2003 R2 and then edit the ADAM to remove the orphaned server on which installation is failing:

1. Open \Windows\ADAM\ADAM-ADSIEDIT.msc on the existing ISA Config Storage server.
2. Navigate to CN=Configuration, CN=Sites, CN=Default-First-Site-Name,CN=Servers.
3. Delete the server on which you have the installation problem.

Re-run the installation, it should succeed now.

Upgrading address lists created in Exchange Server 2003

EHLO again.

Hope you enjoyed my previous post that explains how to upgrade Exchange 2003 recipient policies for use with Exchange Server 2007.

This post deals with the “art and science” of upgrading Exchange 2003 Address Lists to its Exchange Server 2007 form. I say “art and science” because it can be a little tricky to understand for those who havent worked much on Powershell or any scripting/coding environment.

If you click on an address list created by Exchange 2003 in the Exchange Management Console, you will receive the following error:

Unable to edit the specified E-mail address policy. E-mail address policies created with legacy versions of exchange must be upgraded using the ‘Set-EmailAddressPolicy’ task, with the Exchange 2007 Recipient Filter specified. specified.

Exchange 2003 Address Lists have a recipient filter that is made up of an LDAP filter. Exchange Server 2007, on the other hand, understands only OPATH filters. The trick is to convert the LDAP filter to an OPATH filter, and this needs to be done manually.

I’m going to explain this with the help of an example. Lets open an Address List in Exchange 2003 System Manager and examine the LDAP filter:

(& (& (& (mailnickname=*) ( (& (objectCategory=person) (objectClass=user) (homeMDB=CN=Mega MailStore (EXCH01),CN=SG01,CN=InformationStore,CN=EXCH01,CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=mydomain,DC=com) ) ) ) ))

To refresh our brains, this LDAP filter basically creates an Address list out of all users that have a mailbox in the ‘Mega MailStore’ mailbox store on EXCH01 server.

Before we convert this LDAP to OPATH, lets write this in a better way:

(&
(&
(&
(mailnickname=*)
( (&
(objectCategory=person)
(objectClass=user)
(homeMDB=CN=Mega MailStore (EXCH01),CN=SG01,CN=InformationStore,CN=EXCH01,CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=mydomain,DC=com)
) )
)
)
)

Now, carefully change all ampersands (&) to an -and. The ampersands are placed in a prefix fashion in LDAP filter, but in OPATH, its much simpler – you place -and between the two parameters. Similarly, change all equal signs (=) to -eq.

(RecipientType -eq ‘UserMailbox’)
-and
(Database -eq ‘CN=Mega MailStore (EXCH01),CN=SG01,CN=InformationStore,CN=EXCH01,CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=mydomain,DC=com’)

Notice that I have also replaced the property ‘homeMDB’ with ‘Database’. This kind of change is required to convert LDAP property names to OPATH. You can get a complete list of properties here.

So, I arrive at my full command:

Set-AddressList “Mega users” -RecipientFilter { (RecipientType -eq ‘UserMailbox’) -and (Database -eq ‘CN=Mega MailStore (EXCH01),CN=SG01,CN=InformationStore,CN=EXCH01,CN=Servers,CN=First Administrative Group,CN=Administrative Groups,CN=,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=mydomain,DC=com’) }

The guys at Microsoft Exchange Team have more to say about conversion from LDAP to OPATH, and is worth a peek.

Upgrading recipient policies created in Exchange 2003

After installing Exchange Server 2007 Mailbox server into an Exchange Server 2003 organization, you open Exchange Management Console and navigate to Organization Configuration > Hub Transport > Email Address Policies.

You find all the legacy recipient policies that you created in Exchange 2003 over here, but when you try to edit a recipient policy, you get the following error:

Unable to edit the specified E-mail address policy. E-mail address policies created with legacy versions of Exchange must be upgraded using the ‘Set-EmailAddressPolicy’ task, with the Exchange 2007 Recipient Filter specified.

So just how do you fix your email address policy? Yup, you will need to use Exchange Management Shell, no matter how much you hate it.

First, lets fix the Default policy using the Set-EmailAddressPolicy cmdlet:

Hit ‘Y’ when you are asked to confirm the upgrade.
If you have additional recipient policies, you need to upgrade them as required. One important thing to remember is that, in Exchange 2007, you can specify only from the following ‘filter’ fields, as far as email address policies (recipient policies) are concerned:

• Department
• Company
• CustomAttribute1, CustomAttribute2, … , CustomAttribute15

In Exchange 2003, it was possible to define recipient policies from complex LDAP queries, but I see that kind of flexibility is unavailable in Exchange Server 2007. For instance, in Exchange Server 2003, you could create a recipient policy for all users who have mailboxes in a particular mailbox store.

Anyways, lets upgrade our policy using one of the available tactics – lets say – based on Department. If I have an Exchange 2003 recipient policy that gives all users from the sales department email addresses of the form @sales.mydomain.com, my Set-EmailAddressPolicy command would look like this:

Set-EmailAddressPolicy “Sales Dept Recipient Policy” -ConditionalDepartment ‘Sales’ -IncludedRecipients AllRecipients

Note that I do not need to specify the email address format for upgrading the recipient policy.

When Setup fails: Exchange Server 2007 Mailbox Server Role

I went ahead to install the mailbox server role on one of the brand new servers commissioned for Exchange Server 2007.