Building a SharePoint 2007/2010 development environment - Part V: Guest Build

Tristan Watkins, Infrastructure Consultant
In the first four parts of this series I covered the project objectives and the system design, then turned my attention to the Hyper-V host image build and automated deployment. In this post I describe a SharePoint 2007 virtual machine build.

Where’s the SharePoint 2010 build?

In short, we're working on it. I've produced a new SharePoint 2010 beta virtual machine for this environment but we're not yet ready to publish build guidance. Stay tuned. Additionally...

Building Accessible SharePoint Systems - SharePoint User Group UK

Glyn Clough, Solutions Consultant

Last night it was good to see Martin Hatch presenting at the SharePoint User Group in London. Martin is a Solution Architect and a colleague of mine at Content and Code who has worked extensively on building accessible SharePoint solutions - most recently for the Royal National Institute of Blind People.

Key highlights for me were the use of Rendering Templates and Control Adapters to override out of the box controls and how they are displayed and function. These techniques can be used on both MOSS 2007 and SharePoint 2010, and are not solely aimed at delivering accessibility improvements.

Martin has now made available the materials from his presentation:

• PowerPoint Slides (2007 – 2010 format) (.pptx)
• PowerPoint Slides (97 – 03 format) (.ppt)
• Sample Code – SUGUK Visual Studio 2010 Solution (.zip)
• Sample Code – Control Adapters (.zip)

If you would like to read more about Martin’s work with the RNIB and the SharePoint Accessibility Solution (SAS) from Content and Code please see the accessibility section of our website.

[Also posted on Glyn’s personal blog]

Building a SharePoint 2007/2010 development environment – Part IV: Automated deployment

Tristan Watkins, Infrastructure Consultant
In the first three parts of this series I covered the project objectives and the system design, then turned my attention to the Hyper-V host image build. In this section I will look at automating deployment of that host operating system. This is lengthy, but there's a lot to cover.
Before I begin, I need to reiterate a "gotcha" that I stumbled across during the pilot. If you're thinking of testing this out by adding the Windows Deployment Services role to a machine that's using Internet Connection Sharing (ICS), think again. They don't play nicely together. Rather, you'll be able to deploy the physical machines but any ICS recipient network adapters will be left without connectivity. If you only have one physical machine for testing, consider deploying WDS to a new virtual machine or adding the role within an existing virtual machine.

Great SharePoint 2010 Learning Resource

Glyn Clough, Solutions Consultant

I was recently shown a great resource for SharePoint 2010 on Channel 9, Microsoft’s developer focussed video content site. Not being a dev, I’ve previously not spent a lot of time looking around Channel 9, but that’s all about to change…

There’s some fantastic video content and presentations on 2010, which will serve as a great introduction for both developers and non-developers alike. The content is collated in a course called ‘SharePoint 2010 Developer’ and grouped into 15 sections, so you can easily focus on the content you need.

- SharePoint 2010 Developer Home Page

It’s fantastic that Microsoft and the guys who’ve helped create/produce/test this content have put so much effort so early on into the product lifecycle on educate all the users and practitioners of SharePoint out there. We’re a million miles away from where we were with MOSS! Great work.

[Also posted on Glyn’s personal blog]

Working with SharePoint 2010: The “Developer Consultant”

Tobias Lekman, Technical Architect

From all the new features in SharePoint 2010, most seem to be focused around developers and development productivity. There are lots of talk about the new Visual Studio 2010 tools for SharePoint, the “Business Connectivity Services” and sandboxed web parts.

Does this mean that we will spend more or less time writing actual code?

At the SharePoint Conference in Las Vegas, I sat in on the “best practises” session where few new topics were covered. 90% of the material was taken from existing SharePoint guidance documentation and sandboxed solutions were touched upon. However, one single line stuck in my mind - “developers, know your platform”.

I have time after time seen developer-heavy implementations of solutions that could have been created using configuration of existing components in SharePoint. At the same time, I have seen solutions where one or two features of SharePoint have been used (and abused) to encompass all scenarios. This is simply because the implementation has been done by someone who has limited knowledge about SharePoint as a platform.
From a developer’s point of view, here are the major changes in SharePoint 2010:

• Less code and more configuration
  
• Less hand-hacking of definition files and more exports of prototypes into solution packages
  
• Less custom solutions and more reliance on out-of-the-box solutions combined with empowered end users

As an example: I recently had very similar requests from two very different clients. They both wanted “powerful” reporting across data sources and against multiple criteria. It also needed to be easily reconfigurable to fit future needs. Both clients mentioned analysis and data mining in loose terms around “dimensions”. This could easily have been steered into a long and expensive project with OLAP data warehouses, reporting models, Excel Services and Performance Point Server, and the developer in me started talking about this approach on both occasions. However, when I demonstrated what could be achieved by connecting Excel and Access directly to the SharePoint list data and performing pivot reports against the data, both clients were more than happy. In the end, I ended up proposing more time on end user training and configuration and hardly any custom development.

The same story can now be applied to requirements surrounding accessibility, standards compliance, integration of line-of-business systems and external data and workflows.

SharePoint 2010 does empower the developer, but more importantly – it makes the development cycles shorter, provides better tools for packaging prototypes into solutions and further empowers end users to interact with the platform with rich client tools.

Out: Large development teams. In: Developer consultants!

[Also posted on Tobias' personal blog]

Building a SharePoint 2007/2010 development environment – Part III: Host image build and performance benchmarks

Tristan Watkins, Infrastructure Consultant

Having agreed the project objectives and designed the system, I turned my attention to the Hyper-V host image build. This is a high-level build guide with start-up time and baseline memory consumption benchmarks at key milestones. These benchmark figures were taken from the Windows Server 2008 R2 Release Candidate build and are admittedly a bit imprecise. However, they do provide an overall indication of system performance as things were added to and removed from the installation. Although I do not have precise figures on RTM improvements, I spot-checked a few of these benchmarks when I rebuilt the system on RTM. Start-up times improved slightly at each milestone. In fact, the final benchmarks came in at 100MB less idle memory used in the RTM release.

Geographically distributed deployments of SharePoint 2010

Tobias Lekman, Technical Architect

SharePoint 2010 brings a large amount of improvements for geographically distributed SharePoint installations.

SharePoint deployments can be categorized in the following segments:

• Uni-centric deployments 
• SharePoint is installed on a single data centre with a geographically dispersed user base.
• Multi-centric deployments
• SharePoint is installed on multiple data centres across regions/continents and clustered user bases close to the data centres.
• Hybrid deployments
• SharePoint is installed on multiple data centres with a geographically dispersed user base.

Globally dispersed users can be categorized into the following segments:

• Local/LAN users
• Users with high bandwidth, 1mb connections and up and low latency at less than 50ms per roundtrip
• Continental latency
• Users with high bandwidth (ADSL, cable) but higher latency at up to 125ms per roundtrip
• Global users
• Users with high bandwidth but with even higher latency at up to 300ms per roundtrip
• Low bandwidth users
• Users in developing countries, satellite links, mobile connections. These users would have low bandwidth and very high latency – too disperse to estimate

Out of these figures we can estimate WAN performance. These figures show degradation in time for the first page to load (PLT1) over different segments assuming that LAN users are the starting point at X seconds:

Segment	PLT1	Example
LAN	X	3 seconds
Continental	2X-4X	6 to 12 seconds
Global	4X-8X	12 to 24 seconds
Low band	??	Very hard to estimate

SharePoint 2007 deployments

The vast majority of SharePoint 2007 deployments used the uni-centric deployment as network related issues were too great to overcome. This is due to the nature of a farm, where all servers communicate with each other as a “single entity”. The farm has a “shared services provider” which is a site on the farm that contains services that are shared between web applications on the farm. These include searching and user profile storage and management.

SharePoint 2007 can be installed over separate databases during a global deployment so that team/department site databases can reside closer to its intended user base, but the shared services and farm configuration database must still be shared. This causes a lot of communication over global connections and keeping these at high speed and low latency is both costly and difficult.

New capabilities in SharePoint 2010

• Cross-farm/multiple instance Shared Services
• The “shared service provider” is not obsolete. Services are provisioned per web application and can be hosted either on the same farm or on a remote or “master” farm. The services can also be configured in multiple instances (for simultaneous access over load balancing) or customised per web application. Remote shared services, i.e. published services, also involve caching between farm instances for performance optimization. The most common shared services are user profile storage, managed meta data services (farm/cross farm shared content types and taxonomy), search and BDC/BCS connectivity services.
• Uninterrupted log shipping
• Creates read-only farms by using SQL log shipping of the entire farm. This can be very useful for creating local copies of content for users on remote continents. Log shipping sends a diffs (differential data packets) of the data and is therefore very fast. As an added bonus, you can use the read-only farm as a disaster recovery platform where you can turn off the read-write farm and make the read only farm the master read-write farm.
• FSSHTTP
• This catchy acronym stands for “File Synchronization via SOAP over HTTP” and works by reading/writing diffs between the SharePoint server and Office 2010. When reading a file, a checksum is sent to the server. If there are any changes, then only the change diff is downloaded to the end user. This will drastically reduce network usage when working with large documents and/or large numbers of documents.
• ODC
• Another acronym that stands for “Office Document Cache”. This application lives on the client machine and manages diff replication of documents between the client and the server. Normally, when saving a document, the Office application will wait for the document to finish saving. With ODC, the document is saved to the “save queue” and is synchronized in the background. This also allows for offline access of already opened documents as well as "offline saving”. Multi-master merges are solved by the ODC engine.
• SharePoint Workspace
• This is the new version of “Groove”, an application that looks like any other Office application but contains an explorer-like view of the SharePoint data. You can synchronize documents and lists for offline viewing and editing.
• Office Web Applications
• Documents can be viewed/previewed and edited directly in the browser using client script and AJAX. Not only does this mean that users do not need the full Office client on their local machine but it uses much less bandwidth.
• Mobile Access View
• The mobile version of SharePoint has been vastly upgraded and now covers the entire site collection. It can be customized as in the previous version but also includes Office Web Applications. The mobile view can be used with standard browsers as well. Documents can be viewed as text only or as image snapshots. In a demonstration I saw a comparison of a document at 432kb using up only 14kb when viewed in mobile view.
• Windows 7 Branch Cache
• This works as a content proxy for any asset such as documents and images. Branch caching can work in peer-to-peer mode, where computers in the workgroup are queried for a document before going to the server. You can also install a dedicated branch cache server using Windows 2008 R2 on the network. The branch cache is again diff aware and can greatly increase performance on satellite worker hubs.

[Also posted on Tobias' personal blog]

Building a SharePoint 2007/2010 development environment – Part II: Design

Tristan Watkins, Infrastructure Consultant

In the first part of this series, I introduced the pros and cons of various SharePoint development approaches and the objectives of this system redesign. In this part I will focus on design choices and conclusions, starting with the core technology.

Why we’ve chosen Hyper-V
There are broadly five decisive factors: performance, management features (like snapshots), cost, 64-bit OS support and a full host OS (not just a virtualisation administration console):

• Hyper-V is now in its second iteration and has proved to be one of the best-performing virtualisation technologies on the market. With laptops we need to take advantage of every performance gain that we can find.
• Perhaps most importantly, and most often overlooked, Hyper-V is free if you're already running Windows Server 2008 or Windows Server 2008 R2as a client OS
• While I work for a Microsoft Gold Partner and Hyper-V is a key part of Microsoft’s future strategy (full disclosure), I believe that most SharePoint developers will have at least an MSDN subscription and a license for a Windows Server operating system, so I believe this is compelling for most SharePoint development environments
• Hyper-V is one of the few virtualisation technologies that supports 64-bit guest operating systems and within that narrow range of choices, it is one of the few that allows a user to log on to a host machine and control virtual machines within it concurrently
• For instance, VMware ESXi is also a high-performance Type-1 hypervisor that supports 64-bit operating systems, but there is no host machine other than a virtualisation administration console
• It also has associated costs (support, management tools, etc) that we would prefer to avoid
• With Windows Server 2008 R2 and Hyper-V, developers can use client tools within a familiar operating system while accessing virtual development environments at the same time
• This is also true of VMware Workstation, but it's a Type-2 hypervisor and will have relatively poor performance
• We need to have 64-bit OS support for Windows Server 2008 R2 and SharePoint 2010 - both of which do not have 32-bit flavors

Our approach
Workgroup development
By building our virtual machines in a Workgroup, we no longer need to worry about SharePoint installation/(re)configuration difficulties, as we will import the same identical virtual machine on everyone’s Hyper-V host. This would not be possible if the virtual machine was a member of a centralised domain because the domain controller would receive chatter from many identical machines and everything would quickly start to unravel

Alternately, we could run Active Directory Domain Services within a virtual machine, but this has a performance overhead that is best avoided unless it is absolutely necessary. Additionally, developing on a domain controller is not ideal

While developing in a Workgroup presents challenges for profile imports, these are far from insurmountable when LDAP directories like AD LDS or SQL users can substitute in many scenarios. The only evident need is a scenario where Active Directory Domain Services (more than just user accounts) are required, and that's certainly not going to be true of enough projects that it should form a part of the base build. To this end, as requirements for Domain Services are identified we will provide new builds, as they are likely to be very project-specific

Networking
Internet Connection Sharing network
In order to isolate identical virtual machines from each other and from network resources, I've created a Hyper-V internal network dedicated to receiving Internet Connection Sharing (ICS) from one of the host's active network connections. This is an internal network like any other Hyper-V internal network - it just so happens that the host's adapter on this network will be receiving ICS from another of the host's connected adapters. Any connection can share to the ICS network - even if Hyper-V doesn't natively support external networks of that type. Depending on the need, we will share to this ICS network from:

• Hyper-V host external connections (by default)
• Wireless
• Mobile broadband
• VPN

ICS also introduces a layer of NAT between the guests and the physical network, preventing inbound connections to guests over these networks. This is desirable as it is how we achieve physical network isolation, and is the reason why we've chosen ICS over Bridging. In the ICS Settings we enable outbound RDP, HTTP and HTTPS connections over ICS by default, although it may be useful to enable other common outbound network protocols like FTP and SMTP. Outbound connectivity from our guest virtual machines is primarily used for connecting to TFS over HTTPS.

Internal network
We have also created a Hyper-V Internal network to support "always on" communication between the host machine and the guest virtual machines. Guest virtual machines can also communicate with each other over this network.

We cannot rely solely on the ICS connection. Guest virtual machine IP addresses on the ICS network will change because they receive them via DHCP from the host's ICS connection. This is just how ICS works. The host's ICS adapter becomes a gateway on 192.168.137.1. Any Hyper-V guests on that network pick up DHCP from the host and are automatically assigned an address on the 192.168.137.xxx (255.255.255.0) IP range.

As we rely on HOSTS file entries for RDP connections from host to guest and for browsing to guest SharePoint sites from the host, we need fixed, reliable IP routing and name resolution, so we use this second network for that purpose. The hosts and guests have both been built with fixed IP addresses on this range.

Because this is a Hyper-V internal network, there is no risk of network collisions on these IP ranges (which are identical for all users). Internal traffic never leaves the machine.

Project builds
By providing self-contained environments, our technical leads and/or architects can now customise the base builds to create project-specific virtual machines that can be exported to all members of a team, reducing system/configuration inconsistencies. In practice, the build lead/architect will export the final snapshot of the project environment, which all team members will use as a starting point for project development. As the project progresses, updated builds can be released in this same manner.

Clean hosts
Host machines will be cleaned of development tools and data, allowing quick provisioning. Development tools will not need to be reconfigured, as they will reside in virtual machines that can be exported/imported to any host. Guest base build virtual machines will be provided with as much SharePoint configuration as possible in order to make them light on reconfiguration, disposable and re-deployable.

Project resumption
Project-specific exports will reduce the need for storage of multiple virtual machines - as much as possible. We will retain one virtual machine export in public storage per-project. This will reduce the amount of time involved with resurrecting environments after project completion.

Optimisation
Host machines are built on Windows Server 2008 R2, optimised as much as possible and reduced to the lightest weight achievable. The build will broadly include:

• Hyper-V R2
• Microsoft Office applications
• All browsers
• No development tools (these will all live in guest machines)
• The only exception to this is the Team Foundation Client which TFS administrators manually install. We have not been able to pick domain users from within a Workgroup environment

Content provisioning
By deploying a single project-specific virtual machine, we can bake content in to the project build, ensuring consistency and reducing the overhead associated with re-deploying content.

Improved testing and reduced volatility through the use of snapshots
By using snapshots we are able to test code and configuration changes without volatility. The benefits of this technology include:

• Capturing restore points at milestones in a server build
• Capturing a stable state before attempting volatile configuration changes
• Capturing a stable state before testing code changes
• Creating an initial restore point after importing a virtual machine and re-configuring network adapters (when required) and making other preferential settings
• This saves re-importing and reconfiguring network adapters if a machine needs to be rolled back to its initial state
• Exporting a virtual machine to capture a problem when trouble occurs
• An exact instance of a problem can be captured and shipped out for support, without having to re-create the problem in a distinct environment. This will only apply tot self-contained environments, however

Local source code storage on the host machine
Before our pilot started we identified that storing the local copies of source code within changing snapshot states could create problems. At the same time, we found it desirable to put the development tools inside our virtual machines in order to get around remote SharePoint development difficulties and to keep the host build uncluttered. To resolve this conflict, we adopted this approach:

• Created a share on the host machine and granted ownership of that directory to a new user account that is used solely for this purpose
• Created a new user account with the same name and password in the guest virtual machine
• Mapped a drive from the guest to the host's new share, using the internal network's IP address and these new credentials
• Launched Visual Studio and downloaded project source code, pointing to the newly mapped drive as the location for the local source
• Created a new Code Group for the mapped drive to enable trust for code execution

These steps are covered in more detail in the build guides that will follow in later posts. This is just an outline of the approach to extracting the local code from the snapshot state, which should be adaptable to other development systems with snapshots. For instance, we've found it necessary to download code to a unique location within this directory for each each user, as TFS tracks the network locations that code is checked out to. This is easy to get around, as each user just specifies a unique directory name. If this were insufficient for project requirements, this TFS behaviour could possibly be "fooled" by using NTFS junction points. We've not seen a need for this yet, but we're confident that with this additional option we should be able to store local source code in this manner, and this has been validated by our project experience with Hyper-V to-date.

Summary
These are the high-level design choices that emerged early in the consultancy and research, which have remained largely unchanged to-date. These choices represent one design that has been validated for our needs, which has some shortcomings like any approach. Some of these issues will be covered in the final post in this series.

The information in this post has not covered implementation in any detail, but do not fret. In the next section I will cover the step-by-step build guide for the Hyper-V host laptop.

Building a SharePoint 2007/2010 development environment – Part I: Introduction and Objectives

Tristan Watkins, Infrastructure Consultant

Over the last few months I've led the consultancy and system design for a SharePoint 2007/2010 development environment built on Hyper-V in Windows Server 2008 R2. This series of six posts will reveal the key decisions and will consolidate recommendations from a broad range of research and guidance. This first post offers a technology-agnostic introduction to the problem, pros and cons of alternative approaches and what we hoped to achieve with the new approach. The design decisions will be covered in more detail in the second post, followed by a deeper look at detailed build guidance.

Historical development approaches
We have used a variety of development approaches in the past, which have all had limitations and were often an obstacle to productivity. They included:

Shared development farms
Shared farms are appealing at face value because they put development resource hunger on beefier infrastructure. However, there are considerable drawbacks to this approach, including:

• The infrastructure burden of these environments is immense
• Due to limited physical resources, these servers housed multiple projects in a single farm, leading to hive pollution and a less reliable/predictable development environment
• When multiple developers would use the same farm, this would amplify resource contention on already strained servers and it made managing application-level change much more difficult

Local development
Developing SharePoint directly on a Windows Server 2008 laptop introduced a need for frequent laptop rebuilds, as the SharePoint customisations for one project rarely carry over to another and the hive gets polluted. This adds a hefty support burden and frequent down-time for developers.

Virtualised development using Virtual PC
Virtual PC is a friendly technology for SharePoint developers to use, as it does not over-complicate virtualisation, but the management functionality (such as snapshots and snapshot export) is inflexible and performance is poor when compared to a Type-1 hypervisor.

Third-party virtualisation technologies
Most third-party technologies have associated license costs for businesses and many do not support virtualisation of 64-bit guest operating systems. This is problematic because:

• Windows Server 2008 R2 is only available in 64-bit
• SharePoint 2010 is only available in 64-bit

Additionally, many developers have strong preferences for third-party virtualisation vendors and if this is not controlled/standardised it can introduce incompatibility issues.

Why none of these approaches were satisfactory
In short, these environments did not offer good enough performance or management flexibility and we needed a standardised approach for business continuity. We also needed a technology that would support 64-bit systems.

SharePoint development complications
SharePoint development in teams has always been tricky, as SharePoint virtual machines cannot be easily cloned and renamed. This means that SharePoint environments require:

• Scripted installation, which is great for standard builds, but not terribly easy to reconfigure for project-specific requirements, or...
• Manual installation, which is time-consuming and not always aligned with a developer’s skill set, or...
• Shared environments, which suffer from the drawbacks outlined above, or...
• Network isolation of the cloned machine so that that nothing on the network will ever be aware that there are clones

These are some of the biggest issues that need to be confronted when designing a SharePoint development environment. This list is by no means exhaustive; in the final post in this series I'll be reviewing some of the issues that we've encountered during a pilot and subsequent roll-out to all of our developers, consultants and architects.

Distilled objectives
After a thorough review of development approaches, difficulties and available technologies, a Windows Server 2008 R2 (RC) with Hyper-V pilot was announced, soliciting senior developer/architect participation, stating the following aims:

• Improve performance over Virtual PC by adoption of a new core virtualisation technology
• All available performance guidance for hardware, Windows, IIS, SQL and SharePoint would be considered for inclusion in the build
• Hyper-V R2 was adopted early for the pilot (at Release Candidate) to take advantage of further performance gains in the new version
• Project cost effectiveness would be improved by:
  
  
  • Removing development tools and SharePoint from the host machine, reducing the amount of reconfiguration on laptop rebuild and reducing rebuild requests
  • Providing a suite of pre-configured virtual environments, reducing setup time at project commencement
  • Making project-configured environments reusable within teams throughout the life of the project with Hyper-V’s snapshot technology
  • Eliminating the infrastructure burden of shared development farms and re-provisioning those resources to better purpose
  
  
• Improve the testing process by introducing snapshots

In conjunction with the pilot we upgraded physical disk speed and capacity from 160GB 5400 RPM drives to 300GB 7200 RPM, as Hyper-V snapshots chew up lots of space, and the added spindle speed is critical to improving performance.

Defining these objectives, limiting the duration of the pilot and testing the new approach were critical to gaining management support for the project. In advance of pilot deployment I trained all pilot participants on Hyper-V and introduced our approach to networking and source code storage, which would not be obvious otherwise.

In the next section I will cover the design decisions that were reached through consultancy with the business, pilot findings and research.

[Also posted on Tristan's personal blog]