Now that Exchange 2013 SP1 (15.00.0847.032) has been released, those that work as consultants will start to see a spike in billable hours. The reason being is that our enterprise customers will now seek to begin migration projects. While the validity of that best practice has fallen by the wayside many years ago, I still experience many customers standing their ground about not executing a large-scale migration project until SP1 of the product has been released. Certain habits are difficult to break no matter how hard we challenge that line of reasoning during the pre sales process. Just to highlight the diligence around patching a messaging infrastructure, I’m sure all those that have installed 2013 SP1 have also installed the required post SP1 fixes.
Humor aside, this is a good time to review several salient reminders that were pointed out during the Microsoft Exchange Conference concerning mailbox migrations to 2013.
- Exchange 2013 has changed the methodology in which a user’s actual mailbox size is reported. Now, a more precise calculation is used which reports back all properties of a user’s mailbox and not a partial set of properties like in the legacy versions. According to Microsoft, the net effect of this change is that legacy Exchange users that are migrated into 2013 will see the “reported” size of their mailbox increase around 30%. It is important to note that the physical size on disk has not increased; rather the logical reporting size of the mailbox has increased. This means that legacy users close to their quota (~75%) will require additional attention prior to migration. Users that are close to their quota within the legacy environment and then are moved to 2013 most likely will exceed their enforced quota. An unanticipated effect may be that the user cannot send or receive new mail after migration. This is of course dependent on what the configured quota restrictions in place are. A simple fix is to increase the quotas by 30% at the database level (or mailbox) on the legacy Exchange system prior to migration. Remember to check the following limits on your legacy Exchange databases: Issue warning at (GB), Prohibit send at (GB), and Prohibit send and receive at (GB). You can verify your configured database limits prior to migration by using the following PowerShell command: Get-MailboxDatabase <database name> | fl *quota*.
- During migration projects for larger enterprise customers, any production impacting changes will need to be coordinated through a change review board. These changes are then approved through a committee vote and scheduled for a precise start and end time. During an approved change, a conference bridge is setup (with a cast of thousands) where you will need to announce when you are making a change, when the change is completed, and when the change has been tested. Don’t be the consultant that requests the DNS team to make a change to an A record for a namespace cutover only to realize the TTL is set to 86400 seconds (24 hours) while on the bridge. I guarantee that you will never forget that mistake. As part of your steps during a namespace cutover, make sure to verify and document the TTL of all relevant DNS records. If the TTL is set to a longer than desired time, then change the TTL of these DNS records 24 hours prior to the actual scheduled namespace cutover.
- It is important to review how the Offline Address Book (OAB) within your legacy Exchange environment is configured. Many mailbox databases have a setting of $null for their default OAB. This configuration simply means that the default OAB is used. However, when the first Exchange 2013 Mailbox server is introduced into the environment a new ‘default’ OAB is created. This means the Exchange 2013 will now be responsible for generating and distributing the OAB. This becomes problematic if your environment has a large OAB (some are 100 MB+) with thousands of Outlook clients distributed over many types of networks with varying degrees of available bandwidth. Yes – think about how popular you will be with the network team on Monday morning when 10,000 users open up Outlook and initiate a full OAB download at the same time. Brutal. The easy work around is to configure all legacy Exchange databases with the legacy OAB prior to install Exchange 2013. The following PowerShell command can help: Set-MailboxDatabase –Identity <database name> -OfflineAddressBook “<default OAB Name>.” For a more detailed explanation of this situation read the article written by fellow Microsoft Certified Master (and MVP) Andrew Higginbotham.
- The timing of when to swing the SMTP endpoint from the legacy Exchange infrastructure to 2013 is often discussed. The best practice for larger migrations is to make this change once 50% of users have been move to 2013.
- Some scenarios can benefit from installing newly built Exchange 2013 servers into an empty Active Directory Site. This will allow you to configure the 2013 environment fully, while providing a logical separation from internal domain joined clients AutoDiscover SCP requests. The 2013 servers can be moved to the production Active Directory site once you have completed all necessary configuration steps.
- Don’t forget all the great tools that Microsoft provides us with to help with migrations! We have the:
Hopefully, some of these reminders can help you to avoid being the consultant that has to request an emergency extension to a change control window because you forgot something!
Now where is my Exchange 2013 post SP1 project plan…
During last week’s Microsoft Exchange Conference, Jeff Mealiffe recommended that the fixes found in KB2803754 or KB2803755 should be applied when installing Exchange 2013. Specifically, KB2803755 will ensure that the .NET Framework 4.5 becomes properly tuned for the Exchange Information Store within Windows 2012. The same fix for Windows 2008 R2 is contained in KB2803754.
According to the KB article, a registry entry must be configured in order to activate this hotfix. The KB instructs the user to create the following registry key:
Create a DWORD value at [HKLM\Software\Microsoft\.NETFramework\DisableRetStructPinning] registry subkey, and set the DWORD value to 1.
After the registry key is added the server must be rebooted for all changes to be fully realized.
Now the question may be raised as to how a .NET fix can enhance the performance of an Exchange 2013 database? During the development of Exchange 2013 the store.exe process was redesigned and coded as a managed store. The managed store within Exchange 2013 now has fault isolation properties. In order to achieve this isolation, a Store.Service.exe is utilized to manage worker processes. A separate worker process (Store.Worker.exe) runs for each mounted database on a Mailbox server.
To illustrate this point, using the screenshot in the example below, there are two mounted databases (eu-01 & db01) running on a test Exchange 2013 Mailbox server.
Now if we take a look at the task manager for this server, there is the overarching Store.Service.exe running that manages Store.Worker.exe processes. There is a Store.Worker.exe process for each active database that is running.
The managed store relies heavily on the .NET Framework so we need to be especially aware of any .NET patches released.
This patch will help reduce the amount of memory that the Store.Worker.exe processes consume on each Mailbox or Multi-role server within your environment.
Unfortunately, the Microsoft Exchange Conference (MEC) in Austin came to an end last week. Much like the 2012 predecessor, the agenda was packed with fantastic content, speakers, and community involvement. Many in the community have used the term epic in describing the conference and I am more than willing to follow along. I was extremely impressed by the extent of which the Microsoft Exchange product group was represented (much like in 2012). There seemed to be members of the product group teaching at almost every session. If they were not teaching, they were in attendance showing support. The Microsoft MVP community was also well represented with a presence on the expo floor and a strong presence within the sessions. The community is what sets this conference apart from any other.
Throughout the keynote on Monday morning several consistent messages were presented. First, the strategic advantages of the Microsoft Office 365 product suite were showcased. New features such as Clutter, Groups and Oslo (OfficeGraph) were demonstrated with varying degrees of success. The message was clearly communicated that Microsoft is looking cloud-first. As such, many new features will be released to the service (Office 365) before they are incorporated into on-premise releases. OneDrive for Business was demonstrated to show just how deep the messaging integration has become. A new email was composed with a large attachment that was stored in OneDrive for Business. Very easy and useful!
After the keynote, the conference began which would deliver around 100 scheduled sessions over the three days. This year, the ‘Experts Unplugged’ session format was introduced. The ‘Experts Unplugged’ sessions consisted of a panel of 4-6 Microsoft Exchange Product Group members with a moderator (typically an Exchange MVP). As this was marketed as an open forum, the audience members were encouraged to ask questions and really drive the discussion. The moderator would typically kick off the session by asking a question to the panel members, which would inevitability draw audience participation. These sessions turned out to be quite valuable and candid! The Managed Availability, HA, and Exchange Deployment sessions were full of great real-world information. I honestly hope that Microsoft starts to incorporate more of these sessions in future conferences such as TechEd and MEC.
Below are the important notes that I took away from the conference:
- The Exchange 2013 sizing guidance has been updated to reflect feature enhancements in SP1.
- The recommended page file is now 32778 MB if your Exchange server has more than 32 GB of memory. The page file should be set as fixed and not managed by Windows.
- The CAS CPU requirements are 50% higher to accommodate the MAPI OVER HTTP (Alchemy) feature set.
- Multi-Role! Multi-Role! Multi-Role! Yes, given the relatively light requirements of the CAS role, the direction to build multi-role servers falls in line with the new sizing guidelines. Start your deployments with multi-role servers first, and only change if specific requirements dictate. And yes the question was asked – the service does take advantage of the multi-role servers for a variety of reasons. All kidding aside, the importance of building multi-role servers was repeated throughout the sessions.
- Microsoft prefers the primary client connection method to be Outlook Web App for end users. Every demonstration and talk (that I can remember) utilized Outlook Web App. The reason being is that the connection complexity is removed (i.e RPC). Apparently client connection problems based on misconfiguration (i.e. load balancers) represented a large volume of Exchange 2010 support tickets.
- Outlook Web App for Android is now released!
- The Edge Transport Role has been reintroduced in 2013 SP1. When I read the release notes of SP1, the importance of this was not initially grasped. However, several sessions pointed out that the Edge server was important for hybrid scenarios. For hybrid customers that required filtering between on-premise transport, and the service, the Edge Transport server is the only supported method.
- The Microsoft Exchange Product Group communicated clearly that NFS is not supported for Exchange. NFS cannot guarantee specific performance measurements that Exchange requires.
- MAPI over HTTP is the protocol that Microsoft is pushing. It seems that traditional Outlook Anywhere through RPC is on life support. This is a welcomed changed and will make configuration and troubleshooting much easier. The ability for clients to reconnect TCP not RCP connections will help to facilitate a better experience. The prerequisites of Exchange 2013 SP1 and Outlook 2013 SP1 may be a bit steep for larger shops.
- The .NET Framework 4.5.1 update is essential if the new MAPI over HTTP connection method will be used. This update resolves an issue that impacts the performance of MAPI over HTTP with the CAS.
- Highly recommended – The fixes found in KB2803754 or KB2803755 will help .NET Framework 4.5 become properly tuned for the Exchange Information Store.
- Certificate-based authentication will be made available in CU5 for ActiveSync.
- The value of using lagged copies was heard in several sessions. This actually does make sense with the introduction of loose log truncation in SP1. There are now three registry entries that can be added on each DAG-member server (LooseTruncation_MinCopiesToProtect, LooseTruncation_MinDiskFreeSpaceThresholdInMB, LooseTruncation_MinLogsToProtect).
- Microsoft is working around the clock to resolve the public folder scalability issues. Specifically, I heard that the limit is being targeted at 1,000,000.
- A slide deck was shared that showed the Office 365 service runs in 26 data centers. There are an astounding 125,000 databases with the environment that only take 15 seconds to failover – most impressive.
- Yammer was touted as the next great social application that the enterprise should be using. Are you? Many groans were overheard among the attendees whenever the product was mentioned. Within yammer, a MEC specific group was setup and used to distribute additional content for each session.
I cannot wait to catch up with my friends in the community and fellow MCM’s in several weeks at TechEd! For those that have a cool beverage in their hand – YAMMER!
Are you attending the Microsoft Exchange Conference and wondering what sessions can answer questions around modern public folders, encryption and managed availability? If so, listen in to the inaugural Geeks With a Blog podcast!
Service pack 1 has been released according to the Exchange product group!