How to filter the results of Client Object Model requests sent from PowerShell on the server side, implementing the Where method

There are cases when we have to use the Managed Client Object Model from PowerShell. One of the most well-known cases is when we would like to manage our Office 365 environment (or access data stored in it) from our PowerShell scripts.

The Managed Client Object Model itself supports the server side filtering of data requested from the client using the Where method (see Applying filters to list retrieval), in contrary to the ECMAScript Client Object Model, where it is possible only via an unsupported workaround.

What it makes easy to use from advanced .NET programming languages like C# are the concepts of static methods, LINQ and lambda expression, all of them is quite far away from the standard PowerShell samples. No surprise, that the examples we can find on the web do not apply this filtering technique at all. If you have a look at this example, you can see, that it first downloads the whole list of site columns over the wire, then filters them on the client side just to get the two columns we need. If we have hundreds of columns, it can be pretty an overkill.

Note: Yes, I know that REST makes it possible (via the $filter query option) to filter the items returned, however it has its own limitation, like the lack of the batch requests – e.g. aggregating requests on the client side and sending them in batches via executeQuery – that is (at least, IMHO) one of the best features of the Client Object Model. Furthermore, I feel REST does not fit so good to the concept of PowerShell as the usage of the Managed Client Object Model.

Since we are able to use the same .NET libraries from PowerShell as from C#, it should be possible to achieve the same filtering result as well. But how to start?

First, let’s see a simple C# sample of a Managed Client Object Model call. In this sample we get the list of those fields of the Images list, that are of type Guid.

How could we translate this one to a version that (although maybe less readable for us, humans) better reflects the object model and method calls behind the syntax sugar of C# (like static methods, and so on)? If we compile our code into an assembly, and open that assembly with a decompiler tool, like JetBrains dotPeek, the result may help us a step further:

That seems already far better, but what can we do with the lambda expression?

Opening the assembly with another decompiler tool, Reflector reveals further details about the internal working of this functionality:

IQueryable<Field> fieldsQuery = Queryable.Where<Field>(ctx.Web.Lists.GetByTitle("Images").Fields, Expression.Lambda<Func<Field, bool>>(Expression.Equal(Expression.Property(CS$0$0000 = Expression.Parameter(typeof(Field), "f"), (MethodInfo) methodof(Field.get_TypeAsString)), Expression.Constant("Guid", typeof(string)), false, (MethodInfo) methodof(string.op_Equality)), new ParameterExpression[] { CS$0$0000 }));

As this decompiled code suggest, and as learned from this forum answer, we can substitute the lambda expression with an equivalent expression tree built dynamically.

The version we receive after this transformation:

All we now have yet to do is to translate the C# code to PowerShell. The translation is not extremely complicated, it is mainly a mechanical process, although includes a lot of Type objects and Reflection calls, and we need a few tricks as well to achieve our goal.

For example, in the code above we use the static generic Lambda method of the Expression class with parameter types Expression and ParameterExpression[]. If we liked to get this method by name and parameter types directly using the GetMethod method of the Type class, we would receive an “Ambiguous match found” exception. The cause is that the Expression class has two methods with the same name: one generic one, and a non-generic version, and the GetMethod method does not support to restrict the filter to only one of these. Probably we can get the right one using the FindMembers method as well, but I found it easier to use the GetMethods method to get all of the methods of the Expression class and filter the results in my PowerShell code. In the code below we filter the methods, first by name, next we restrict the results to generic methods, and take the override that has two parameters, and the second parameter is of type ParameterExpression[]:

$lambdaMethod = $expressionType.GetMethods() | ? { $_.Name -eq "Lambda" -and $_.IsGenericMethod -and $_.GetParameters().Length -eq 2  -and $_.GetParameters()[1].ParameterType -eq $parameterExpressionArrayType }

I had similar problems with the Where method of the Query class. Although this method has only two overrides, and these ones have different parameter signatures, the parameters are rather complicated generic parameters, like Expression<Func<TSource, int, bool>>, so I decided to filter the methods again from my code, taking the methods called Where, and selecting the one that has a textural representation with the right parameter types:

$whereMethod = [System.Linq.Queryable].GetMethods() | ? { $_.Name -eq "Where" -and $_.ToString() -like "*TSource,System.Boolean*" }

The result of the translation is:

Note: Before using this code, you should load the necessary client object model assemblies via:

Add-Type -Path "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.dll"
Add-Type -Path "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\15\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"

When validating by Fiddler, we can see that the request contains the filter we need:

And when checking the result, only the matching fields are returned, and all without filtering the results on the client side.

Conclusion: I don’t want to suggest at all that you should always use this kind of filtering when requesting data from the server. It may be unnecessary or even inefficient if you have only a few or a few dozens of entities to return from the server. However, if it is really a large number of items, it may be useful to know how to get only the entities you really need.

Note: I’ve tested my solution with SharePoint 2013 on-premise, but it should work with SharePoint 2010 and Office 365 as well. You can however apply this technique not only to the SharePoint object model as well. For example, I used it first with the Project Server 2013 Client Object Model.