HttpCode

Some of the major web 2.0 sites like facebook and twitter are providing REST based API’s to interact with their services, if your not familiar with Representational State Transfer (REST), then now might be the perfect time for you to come up to speed because it seems that Microsoft is planning on incorporating more REST features in .NET 4, we already have some REST support in the current generation of tools.

The basic idea of REST is to use the existing infrastructure that the web already provides, we have HTTP verbs such as GET, POST, DELETE and PUT, which map pretty well to the CRUD model we deal with often. The web provides us with a simple error handling model, we all understand 404, 501 and 403 error codes (file not found, server error, unauthorised access). The final building block is the URL itself, we can create descriptive URL’s such that any user can infer it’s intent just by looking at it, while some may disagree with my summation, I think all of these elements together define what REST is.

If we contrast all of this with the world of SOAP and webservices, we soon realise that the webservices/SOAP model has been over-engineered. Do we really need to re-invent error codes and the calling conventions that already sit on top of the web, do we need all this SOAP overhead, one end-point for a raft of operations? Of course there will still be a place for traditional webservices, I’m not saying they don’t work or are going away, they clearly work and will be around for a while.

The webforms model of ASP.NET hasn’t really made it easy to create REST services, it was never really designed for it, however the ASP.NET framework has made developing REST’ful services a lot easier.

The first feature that the MVC framework offers is the flexibility of offering a URL routing engine, so that it is possible to easily create nice URL’s, but more than that, the way MVC uses the convention of {controller}/{action}/{parameter} you end up with code that without much effort becomes very REST like.

The second feature that MVC offers is the way each action on a controller returns an ActionResult, the framework provides inherited result classes such as the JsonResult and ContentResult, in effect the one action can return multiple payloads of data (in this case either a JSON view or an XML view). While not strictly fitting to the REST definition, most modern REST API’s return data in a format that is most easiest to work with which is increasing becoming JSON for AJAX/Mashable operations.

With this background we can now focus our attention on SharePoint, currently there are two ways to work with the data stored:

• Web services
• Object Model

I’ve already touched on what the limitations of the web services are, the SharePoint object model really only concerns us if our code is working inside SharePoint (i.e. on the same machine/s as SharePoint).

The Benefits to SharePoint of a REST API

• Enterprise mash-ability – the web is becoming more and more useful, one reason is that its now easier to mash together applications, look at any application that makes use of virtual earth or Google earth, flickr, twitter, live services etc. Imagine having this flexibility in your organisation, if your organisation has really embraced SharePoint then most likely it contains data that could be mashed into other systems, this REST framework is about exposing that data more easily.
• It still fits with an organisations Service Oriented Architecture (SOA) in most cases. A lot of people feel strongly that SharePoint isn’t the single source of truth for data, that may be the case, but SharePoint is a tool that does aggregate data and adds value to it (a user might create a list with BDC data with additional columns describing some business function), this framework is designed to make this data more accessible to an organisation trying to construct some form of SOA.

The URL Format

A REST API that sits over the object model would provide the best of both options, we could craft a simple URL that returned the items that we are after such as:

http://mashserver/Site/            -- returns all sites
http://mashserver/Site/all/            -- same as above
http://mashserver/Site/get/{guid-id}    -- returns the selected site

http://mashserver/List/                  --returns all lists that the user has permissions for
http://mashserver/List/forsite/{site-guid-id}   -- returns all the lists for the given site id

http://mashserver/ListItems/get/{list-guid-id}  -- returns all the items in the list,

http://mashserver/BDC/   --Return all the BDC applications

http://mashserver/Permission/site/{site-guid}   --Return objects representing the security ACL’s on the given object

The REST API should expand across all areas of the SharePoint system. Each action should be decorated with an accepted verb type attribute which MVC provides.

The Return Values

We could build a HybridResult that looks at the HTTP headers to determine the accepted input and then either return JSON or XML:

In fact Omar has already provided a nice starting platform.

This HybridResult can also be smart enough to return a 404 if the object we are trying to serialize is null. If we get any security exceptions we can set the return status code to 403, again the MVC provides a nice mechanism to support this via it’s Error Handling attribute.

Problems

• Do we build our own object model, reinvent the wheel?

I think the answer is yes, we can’t serialize the SharePoint object model to our needs as a REST API, in some cases we want to return properties in a form that would be easy to use from JavaScript. We often don’t want to return the whole object graph, if you ask for all the lists, you really don’t want to have a list object returned will a collection of list items. By using the new language features in .NET 3.5 we can build extension methods that provide a neat way to convert the existing Object Model entities to those created by the MVC framework.

• Security – The double hop issue

Since the API is going to make heavy use of the object model, it’s going to have the same limitations as the object model, i.e. the API web sites will need to live on the same server as SharePoint unless the double hop authentication issue is taken care of via the use of Kerberos, which is a likely event in most larger organisation.

 Features - Thoughts?

So what do you think? Is a REST API for SharePoint something that would be useful? I’ve already started developing some proof of concept prototypes which I’ve included on the project’s Codeplex site:  http://www.codeplex.com/REST4SharePoint so please leave any feedback there.

What features do you think it should include?

What are your thoughts?

What obvious thing have I missed?

I wrote about setting Microsoft Search Server 2008 Express up and configuring it to index a web site, the search interface that I demonstrated was a standard SharePoint site based on the Search Centre Template. I thought it would be nice to show how easy it is to build your own search interface.

I think the big appeal about Microsoft Search Server 2008 Express is that it can be used without the full overhead of SharePoint. So with that in mind and the fact that I’m unhealthily in love with the MVC framework, I’ve put together a simple ASP.NET MVC sample that works with the search server:

The screen cast shows the sample application as well as an overview of the code:

The thing that is really appealing is that by using the ASP.NET MVC framework, you have complete control over the generated HTML, there is no nasty webpart hidden fields or the hideous viewstate.

I hope you come away with the idea that Search Server would be perfect for indexing your website or file shares etc, but that you ultimately have the power to present those results how you see fit.

You can download the source code to my sample application here. It makes use of the MVCContrib library for the pagination support and preview 5 of the MVC framework.

A list of all the default columns that are returned by the search web service can be found here:

• WorkId
• Rank
• Title
• Author
• Size
• Path
• Description
• Write
• SiteName
• CollapsingStatus
• HitHighlightedSummary
• HitHighlightedProperties
• ContentClass
• IsDocument
• PictureThumbnailURL

If these don’t provide what you need then you can use the other enterprise search schema’s.

By default SharePoint will create a content source called ‘Local Office SharePoint Server sites’ such as:

This will contain the starting addresses of all the sites on your SharePoint server such as:

Notice how it also includes the sps3://, this is the indexing of your user profiles.

My tip is to remove the sps3:// link from the default content source and add it as a new content source on it’s own.

The reasons why I think this is helpful:

• By default you need to crawl all your other content just to update your user profile information.
• You can schedule your profile crawls at a time that suits your active directory imports

In any case it’s worth considering breaking the profile crawl into it’s own content source.

I’ve put together a screencast that carries on from my previous post on installing and configuring Microsoft Search Server 2008 Express.

This screencast covers:

• Setting up a new external content source (my example is this indexing this site)
• Viewing the crawl log to determine the success or failure of a crawl
• Adding a search scope to give the end users the option to focus there search query to a particular area
• Enabling the search scope in the site collection
• Setting up the search centre page web parts to show the search scope dropdown

In the screen cast I mention that the navigation links in central admin or the search administration don’t link to the search scopes settings page. I manually added the url:  http://<server-name>/ssp/admin/_layouts/viewscopes.aspx?mode=ssp

The higher quality screencast can be found here or the youtube version:

You may not have heard about Microsoft Search Server 2008 Express, just as SQL Server Express is the little brother of SQL Server, Search Server 2008 Express is the little brother of MOSS. It offers most of the enterprise search capabilities of MOSS minus the BDC.

I’ve put together a screen cast of the install and initial setup of the product, you’ll find that the entire process is almost the same as WSS or MOSS, that is because the product is built from the same components. The one notable difference is the new Server Wizard page, it combines all the common settings such as app pool accounts, selection of the index server, content access accounts into one place.

The search administration is the same as MOSS with the infrastructure updates, in fact the infrastructure updates goal was to align the work done around the Enterprise Search Server space with the search capabilities of MOSS, new features like federated search etc.

I highly recommend taking a look at Search Server 2008 Express if you want to provide search to your organisation without the committing to a full MOSS install or even to increase the search capabilities of your WSS farm.

The install steps are:

• Launch the installer, make sure you select the ‘Install all components’  from the advanced option if you wish to connect to a SQL Server, otherwise the stand alone option if you wish to use the SQL Express version.
• The configuration wizard will run after the install (just like SharePoint), depending on your option, I choose to create a new farm.
• Enter your database server name and the database access account, this account should have permissions to create databases.
• Specify a port for central admin to run on, try to make this consistent across all your farms, it makes it easy to remember that way.
• Once the setup completes, central admin will be launched, a page specific to Search Server will be presented. This page will ask for the content access account, the application pool account and various other options. I really like this consolidated page, hopefully we will see more of it in SharePoint 14
• In my case the web application was created successfully, however the site collection wasn’t created, so I selected the create site collection option in central admin and used the search centre template.

The higher quality screen cast can be found here or the youtube version:

Of course you should also subscribe to the Microsoft Enterprise Search Blog to help stay up to date with this product.

So the big news is that Silverlight 2 has been released, I haven’t really paid much attention to Silverlight up until the Devsta competition which gave me the incentive to jump right in and start developing with it.

I remember on the first day of development of Console Connect I was really frustrated that I couldn’t get the exact layout of UI elements that I wanted, I hadn’t fully understood the power of the StackPanel. Once I got over that hurdle I found myself applying the same text styles to numerous textboxes which didn’t feel right, then I discovered the shared resources, by now the light was on and I was lovin it.

I was really impressed by how easy it was to pick up, I don’t recall having any problems at the code level, just the UI issues I previously mentioned. Within a week I had a couple of small games working and could do all the same things that I could in a winforms app like calling webservices and changing the UI in response to the users input. If I was a little more graphically inclined, my games could come to life with animations and sounds, but that is best left to those right side of the brain people.

One of my colleagues said to me that he didn’t think that Silverlight would get much widespread adoption, but my thoughts after having a good play with the technology are a little different.

I think the key is the shallow learning curve for .NET developers, something like this: 

Once you get over the little UI hump (which is basically the same for WPF) and learn a handful of things like data binding, resources and the way the UI can use panels, your just about at the top of the curve. I think Scott Hanselman had much of the same experience when started on his babysmash experiment.

So ultimately I think the success of Silverlight will be around the fact that developers can pick it up quickly and experience all that developer love from Microsoft. All the things that make development a breeze like a full debugger, intelli-sense and a framework that is familiar.

Or maybe I’m completely wrong and the rest of the world will have the same view as Ruby On Rail’s David Heinemeier Hansson who basically thinks that we are only scratching the surface with what can be done with Javascript and HTML, maybe rich internet applications will be a passing fad?

I previously blogged about using the BDC with MOSS user profiles and how to set that whole process up. Well I thought that I might write a little about the BDC application definition file (ADF) that is imported into MOSS and is used by the BDC to generate the meta-data and to ultimately connect to the data source.

An ADF file contains metadata describing entities and methods to populate those entities.

These are the methods that are of interest to us, all have nice relevant names:

IDEnumerator – These methods can perform filtering and can be passed parameters, the idea is that it returns an ID (and a timestamp if possible), as it’s name implies it is used to enumerate all the ID’s (or primary keys). In the context of the profile import, if your key is say, an Active Directory email address, then the IDEnumerator should return the email address field.

Specific Finder – This method accepts an ID and returns just the information related to that ID. You’ll probably create a number of these using different filter descriptors.

Now that you have an idea of what the methods are and how they operate you can design web services that are low friction for the BDC.

You’ll need a web service that returns a list of ID’s (for the IDEnumerator), you’ll need a second webservice that accepts the same ID’s that were returned by the first method, this second method will comprise your Specific Finder methods.

Also don’t create trouble for yourself by building webservices that accept a large number of parameters, you’ll regret it, just keep it simple. The guys that developed the fantastic BDC Meta-Man product also have the same advice.

I’ve provided the SQL create statement and ADF file here.

I’ve put together a screencast that demonstrates how to configure a web application so that it can host MOSS MySites.

The basic steps are:

• Create the web application
• Create a site collection to be hosted in the newly created web application, make sure this is based on the ‘My Site Host Template’
• Create a managed path for the MySites
• Change the MySite settings in the SSP central admin to reflect the URL and managed path that was setup in the previous steps
• Enable self service site creation so that new MySites are created when the user clicks on the My Site link

The high quality version can be found here or the youtube version:

One of the main features of SQL Server 2008 is database backup compression. In SQL 2008 you can now specify the backup as:

BACKUP DATABASE ‘WSSDatabase’

TO DISK = ‘C:\WSSDatabase.bak’
WITH COMPRESSION;

Or if your GUI inclined:

This will make managing your content databases a little easier, especially if your already getting heckled by your SAN Administrator or the poor guy trying to backup those 150GB backups.

The second feature is Transparent Data Encryption (TDE), this allows you to specify a key that is used for data encryption for example:

ALTER DATABASE ‘WSSDatabase’
SET ENCRYPTION ON;

The scenario where I think this feature works nicely is this; imagine your pay-role people want to store pay details, you could firstly ensure that the site collection they use is located in a nominated content database, then you could turn on the encryption and have another secured means of protecting your data.

Further information can be found here.

Those two features are pretty compelling, on all our new deployments we are recommending SQL 2008.

I’ve put together a screencast that shows how to use your data from an external source such as a SQL Server database and combine it with your user profiles.

The screencast shows how to:

• Import the application definition to setup the BDC application
• Setup a custom import connection of type business data catalog and how to set the key between active directory account and BDC data
• Setting up custom profile property section (the example is cricket statistics)
• Setting up custom profile properties that import data from the BDC data source
• Start a full import
• Perform a search which returns the user and show the new data that is displayed in the users profile page.

A good quality version of the screen cast can be found here (4 mins).

Or the low quality youtube version: