That’s the list view threshold, and it’s the reason code that works fine in dev falls over in production two years later. AllowIncrementalResults is the CSOM property that gets you through it.

Why a Ten-Row Query Fails on an 8,000 Item List

The error looks like this:

Microsoft.SharePoint.Client.ServerException: The attempted operation is
prohibited because it exceeds the list view threshold.

CSOM hands you that as a ServerException, but the thing throwing it server-side is SPQueryThrottledException. Worth knowing both names: the message is what you get in the debugger, the class name is what half the search results are filed under. SharePoint REST hits the same wall for the same reason, though not with the same fix: AllowIncrementalResults is a CamlQuery property and there is no REST equivalent of it.

What throws people is that your filter has nothing to do with it. You can ask for ten rows out of 8,000 and still get the exception, because SharePoint is protecting itself against the scan, not against the result set.

Set the Flag, Then Page

var query = new CamlQuery {
    ViewXml = viewXml,               // <RowLimit>5000</RowLimit>, indexed field in <Where>
    AllowIncrementalResults = true,  // the part that gets you past the threshold
};

ListItemCollection items;
do {
    items = list.GetItems(query);
    context.Load(items);
    await context.ExecuteQueryAsync();

    Process(items);
    query.ListItemCollectionPosition = items.ListItemCollectionPosition;
} while (query.ListItemCollectionPosition is not null);

What’s happening here?

1. AllowIncrementalResults = true tells SharePoint it’s allowed to hand back what it has, in chunks, instead of refusing the whole query. Without it, a big list plus a filter is a ServerException and nothing else.
2. <RowLimit> in the view XML decides the chunk size. It’s the same knob you’d use for ordinary paging, and it has to be there - an unbounded query is exactly what the threshold exists to stop. I use 5000.
3. ListItemCollectionPosition is the cursor. Feed it back into the next query, stop when the collection comes back with a null position. That’s the last page.
4. Every iteration is one round trip, so the chunk size is a real trade-off: bigger pages mean fewer trips and heavier payloads. I count round trips out of habit these days, for reasons I’ve written about at length.

Indexed Columns Are Not Optional

This is the part that turns “it doesn’t work” into an afternoon. The flag isn’t a bypass. The fields you filter and sort on still have to be indexed on the list, or you’re back to the same exception with a more confusing story, because now you did set the flag.

One exception, and it’s a useful one: a field you’ve projected across a <Joins> isn’t a column on the list at all, so it can’t be indexed, and the flag carries it anyway. I only found that out by measuring it on a 10,000 item list, where a join filtered on a projected field is refused flat without the flag and returns every matching row with it. So the rule is: real columns need an index, projections need the flag.

Add the index in list settings, and remember SharePoint allows up to 20 per list. And if your first thought is “the list is already too big to index” - that’s the reflex I had too, and it’s wrong. Microsoft’s add an index to a list or library column walks through it and says you can index a list of any size manually. If it does get blocked, do it inside the tenant’s daily time window.

Gotchas

• The flag doesn’t raise the threshold. It changes the failure into incremental delivery. Unindexed <Where> and <OrderBy> columns still fail, with the projected-field exception above.
• No <RowLimit>, no paging. Without it you’re asking for everything again, and the position never comes back. If you’re here because RowLimit “isn’t working”, check where you put it: it belongs in the view XML outside <Query>, and on its own it caps one response rather than giving you the rest of the rows. Paging is what gives you the rest.
• You cannot parallelise the paging. Each page’s cursor comes out of the previous response, so the loop is strictly sequential no matter how many threads you throw at it. Chunk size is your only real lever on total time.
• Calculated columns can’t be indexed. Neither can a few other types. Discovering that late is how a filter design dies, because the column your whole query hangs on turns out to be the one column SharePoint won’t index - and no amount of AllowIncrementalResults saves it.
• ListItemCollectionPosition belongs to the query, not the list. Set it on the CamlQuery you’re about to execute. Reusing a stale position from an earlier query shape gives you nonsense.
• The ceiling moves per tenant and per time of day. Large list throttling is a service-side feature, so “it worked yesterday” is not evidence that it will work in the batch job tonight.
• The other APIs hit the same wall, and only one of them is polite about it. I used to say I’d never got this to work outside CSOM, so I went and measured all three. REST answers an unindexed filter with a bare 500. Graph answers 400 and names the header that would let it through. Neither has an equivalent of this flag, so filtering a big list still ends up back in CSOM, which is one more entry on the pick-one list.
• Folders change the picture. Scoping a query to a folder cuts the item count SharePoint has to consider, which sometimes solves the problem without any of this.

Wrapping Up

If a CAML query is throwing the list view threshold exception, the fix is three things together: AllowIncrementalResults = true, a <RowLimit> to page with, and indexes on the columns you filter and sort on. Two out of three still throws.

It’s also one of the reasons list item work stays in CSOM for me. Picking the right SharePoint API per operation means knowing which one has an answer for the awkward cases, and large lists are as awkward as it gets.

The wider version of that argument is in the two benchmark posts: what a CAML join costs at 1,000 items, and what survives at 10,000. The short version is that past the threshold this stops being a speed question. It becomes a question of whether you get the right answer at all, and of how much of the list you have to hold in memory to find it.

Microsoft.SharePoint.Client.ServerException: The attempted operation is
prohibited because it exceeds the list view threshold.

The code didn’t change. It never does.

CSOM has an escape hatch for this, AllowIncrementalResults, and it gets a post of its own. What I could never answer was the obvious follow-up. Is CSOM the only place with an answer? Every time I’d needed to filter past the threshold I’d ended up back in CSOM, but I’d never actually gone and checked what REST and Graph do. Saying “I don’t know” for two years is a long time.

So I took the harness from the CAML joins benchmark, seeded a second four-list chain at 10,000 items per list, and ran everything again. Ten thousand is not two hundred thousand, but it’s twice the threshold, which is where the behaviour changes.

What actually breaks at the SharePoint list view threshold?

Not reading. I can still pull all 10,000 rows out of any of these APIs, and it costs two requests. What breaks is filtering: the moment a <Where> or a $filter touches a column SharePoint hasn’t indexed, the query is refused, and it’s refused even when the answer would be ten rows. The threshold is about the scan, not the result set. That single sentence explains almost everything below.

Reading Was Never the Problem

The first thing I measured is the thing nobody worries about, and it turns out to be fine:

Two requests each, because 10,000 rows is two pages of 5,000. An unfiltered read past the threshold is allowed on all three. If your job is “give me the whole list”, the threshold barely exists.

That’s worth saying plainly because the error message sends people hunting for the wrong fix. You don’t need to restructure your list to read it.

The Flag That Shouldn’t Work, and Does

Here’s the one I got wrong, and I was confident about it.

The most useful thing a CAML join does is let you filter on a column that lives several lists away, by projecting it and putting the <Where> on the projection. But a projected field is not a column on the list. It doesn’t exist there, so it cannot be indexed. And the documented rule for AllowIncrementalResults is that the fields you filter and sort on still have to be indexed.

I was sure that made the technique impossible past 5,000 items. I wrote the probe expecting to have to publish a correction.

var query = new CamlQuery
{
    ViewXml = viewXml,               // <Joins>, <ProjectedFields>, <Where> on customerNo
    AllowIncrementalResults = true,
};

var all = new List<ListItem>();
do
{
    var items = list.GetItems(query);
    context.Load(items);
    context.Load(items, i => i.ListItemCollectionPosition);
    context.ExecuteQuery();

    all.AddRange(items);
    query.ListItemCollectionPosition = items.ListItemCollectionPosition;
}
while (query.ListItemCollectionPosition is not null);

What’s happening here?

1. Without AllowIncrementalResults, this exact query is refused: SPQueryThrottledException, straight away, no rows.
2. With it, the same query returns all 5,000 matching rows in two requests. The filter is on customerNo, which is projected from a list two hops away and cannot be indexed anywhere.
3. ListItemCollectionPosition is the cursor and it is not optional. A <RowLimit> on its own doesn’t page, it truncates, and truncation is the failure mode you don’t notice.
4. You need both properties loaded. context.Load(items) alone won’t populate the position on every code path, and a null cursor looks exactly like a finished result set.

So the join survives the threshold. My best guess at why is that the projection is resolved after the join rather than as a scan predicate, but that’s inference, not something I can prove from the outside. What I can say is that it returned the right 5,000 rows, ten times in a row.

Then I checked whether the depth matters, because it easily could have. Everything above filters on a value two lists away, which is the shape I’ve written about before. So I moved the query one list further back and made the projection travel three joins instead of two:

Identical, right down to the request count. Whatever the flag is doing, it isn’t running out of road at the second hop, and a filter three lists from the one you’re querying is as viable past the threshold as a filter one list away.

Three APIs, Three Different Refusals

Filter on a column that genuinely isn’t indexed, and all three refuse. How they refuse is where they differ.

Graph wins this one, and it isn’t close:

Field 'Title' cannot be referenced in filter or orderby as it is not
indexed. Provide the 'Prefer: HonorNonIndexedQueriesWarningMayFailRandomly'
header to allow this, but be warned that it may fail randomly.

Add the header and the same request returns 200. Read the header’s name again before you reach for it, though. Somebody at Microsoft went to the trouble of putting “MayFailRandomly” in an API contract, and that is not the kind of thing you build a nightly job on.

REST answering 500 for this is the one that annoys me. Filtering an unindexed column on a large list is a completely predictable, documented, business-as-usual refusal. It is not an internal server error.

The REST Trick That Only Works Under 5,000

In the previous post I found something I didn’t expect: SharePoint REST will let you filter on an expanded lookup’s column, even one that isn’t the lookup’s ShowField.

$filter=CustomerLookUp/customerNo eq 'C-NARROW'&$expand=CustomerLookUp

At 1,000 items that returns 200 and it’s the single thing that got the OData route down to two requests. At 10,000 items it returns 500, in every scenario I threw at it, along with the batched version of the same call.

So it’s a sub-threshold trick. It works beautifully right up until the list is big enough for it to matter, which is a fair description of a lot of SharePoint behaviour. If you have built anything on that shape, it has an expiry date measured in rows.

The One That Lies to You

RenderListDataAsStream is how you run a CAML join over REST, and in the last post it came through with the projected columns intact. Past the threshold it does something worse than failing.

Asked for 100 matching rows out of 10,000, it returned 47. Asked for 5,000, it returned 2,504. One request, 200 OK, no error, no NextHref to follow, no indication that anything is missing. It stops when the scan window closes and hands you what it found so far as though that were the answer.

A truncated result with a 200 on it is the worst outcome in this whole post. Everything else either works or tells you it didn’t.

What It Costs in Memory

This is the part I hadn’t measured before, and it’s the strongest argument in the post. Below, allocated managed bytes while answering the same question: 100 matching rows out of 10,000.

One hundred rows. Two hundred megabytes.

The split is exactly the Filter column. Every approach that filters on the server materialises about a hundred rows. Every approach that filters in C# has to pull the whole list into memory first, then throw away 99% of it, and at 10,000 items that is 200 MB of allocation to produce 100 objects you keep. Your production lists are twenty times bigger than my test list.

Two things surprised me here. Allocation does not rank the same as bytes on the wire: CSOM allocates roughly eleven times its payload because a ListItem is a heavy object with a field dictionary attached, so 17 MB of response becomes 200 MB of garbage. And the join is not the memory winner when you ask for everything, only when you ask for some. Unfiltered, lean OData that fetches nothing but lookup ids allocates 17 MB against the join’s 105 MB, because the join is honestly returning full rows and OData is returning integers.

The Whole Field at 10,000 Items

Filtering to 5,000 rows of 10,000, entering at the order tasks list, two hops to the customer:

The join is still first, and at 10,000 items it is now first by a wider margin than at 1,000, because everything else has to page and it only has to page twice.

Note what happened to Graph’s reverse walk: 62 requests. The or-chains that fitted in a handful of calls at 1,000 items become sixty-two at 10,000, and batching turns 62 round trips into 5 while leaving all 62 queries exactly where they were.

Gotchas

• The exception message is localised. My tenant answers in Danish: Den forsøgte handling er ikke tilladt, fordi den overskrider grænsen for listevisning. My first probe matched on the string “list view threshold” and therefore never detected a single hit. Match on ex.ServerErrorTypeName == "Microsoft.SharePoint.SPQueryThrottledException" instead. Any retry logic or alerting built on the message text is broken in every tenant that isn’t English.
• <RowLimit> without a cursor is a truncation, not a page. This is how you get a wrong answer with no error. One of my own arms returned 47 rows out of 100 because it read the first 5,000 items of a 10,000 item list and then filtered what it happened to have.
• Batching collapses queries, never pages. A cursor only exists once the previous response has arrived, so a batch can carry every list’s page one and then has to go again for page two. That’s true of CSOM’s ExecuteQuery, of SharePoint’s $batch, and of Graph’s. Graph’s docs suggest batching as a way around URL length limits; I measured the same 91 clause ceiling inside a batch as outside it, so that particular workaround doesn’t apply here.
• <In> past 60 values is fine, and I’m one of the people who told you otherwise. I repeated the 60-value claim in the joins post and cited the archived 2013 article it comes from. On 10,000 items with an indexed lookup column I ran 50, 60, 61, 100 and 500 values, and every one of them returned. The 500-value hard cap is real: 501 still throws Value does not fall within the expected range. It’s the 60 that I can’t reproduce.
• The ceiling moves. One reverse-traversal arm hit the threshold in one scenario and sailed through the same shape of query in another. Large list throttling is a service-side feature with a time-of-day component, so “it worked this morning” is not a test result.
• Your own tooling will hit this too. The setup step of my harness crashed while verifying the data it had just seeded, because it ran an <IsNull> check with <RowLimit>10</RowLimit>. Ten rows requested, still refused. I had written that check myself and still didn’t see it coming.

Wrapping Up

Reading a big list is easy on every API. Filtering it is where they separate, and they separate hard: CSOM throws something you can catch, REST throws a 500, Graph throws a 400 with instructions, and RenderListDataAsStream hands you a partial answer with a 200 on it.

The advice from the previous post survives the threshold, with one line added. Make SharePoint do the join, set AllowIncrementalResults = true, and page with the cursor. It stays the fastest correct answer at 10,000 items, and it’s the only approach that filters server-side without resolving the ids yourself first, which past the threshold has stopped being a performance question and become a memory one.

Everything else is holding your whole list in RAM to find a hundred rows in it, and it is doing that whether or not anyone measured it.

And I had never measured it.

Not properly. Never all the reasonable ways of fetching the same data, against the same lists, on the same day. It’s always been case by case: this feels like the better solution here, that one felt slow last time. In my head, one call instead of four has to be faster, and that was enough.

That’s not a benchmark. That’s a hunch I’d been repeating with confidence.

So I built the comparison I should have built a year ago. It moved my numbers, and it also corrected something I’d been saying wrong for a year.

Is a CAML join actually faster than four separate queries?

Yes, and by more than I expected. Fetching 1,000 rows across a three-hop lookup chain, the CAML join needed 1 HTTP request and about 254 ms. The same rows through four separate CSOM queries took 6 requests and 1,192 ms, OData expands over SharePoint REST took 4 requests and 548 ms, and Microsoft Graph took 4 requests and 1,655 ms. The join wasn’t just fewer calls, it was 4.7x faster than the four-query CSOM version and 6.5x faster than Graph. The closest anything got was SharePoint REST with $batch, at 355 ms, and that one deserves its own paragraph rather than a footnote.

Same Rows, Eight Ways

Four lists, each pointing at the next through a lookup column:

Main -> OrderTasks -> Orders -> Customers

Every approach has to return the same thing for each of the 1,000 items in the main list: its id, its title, and the customerNo and customerName from a customer sitting three hops away.

The join version, built with CAMLEX:

const string OrderTasks = "ordertasks", Orders = "orders", Customers = "customers";

CamlexNET.Interfaces.IQuery query = Camlex.Query()
    .LeftJoin(x => x["OrderTaskLookUp"].ForeignList(OrderTasks))
    .LeftJoin(x => x["OrderDetailLookUp"].PrimaryList(OrderTasks).ForeignList(Orders))
    .LeftJoin(x => x["CustomerLookUp"].PrimaryList(Orders).ForeignList(Customers))
    .ProjectedField(x => x["customerNo"].List(Customers).ShowField("customerNo"))
    .ProjectedField(x => x["customerName"].List(Customers).ShowField("customerName"))
    .ViewFields(["ID", "Title", "customerNo", "customerName"]);

var items = mainList.GetItems(query.ToCamlQuery());
context.Load(items);
context.ExecuteQuery();

What’s happening here?

1. Each LeftJoin walks one lookup hop. PrimaryList is the list you’re coming from, ForeignList the one you’re going to, which is what lets hops two and three start somewhere other than the main list. Those arguments are aliases you invent, not list ids: they’re string, and the compiler will tell you so (cannot convert from 'System.Guid' to 'string') if you try otherwise. SharePoint resolves the actual list from the lookup column’s own configuration, so "orders" works exactly as well as a GUID. I checked, because I’d been passing GUIDs for a year without knowing they were only ever labels.
2. ProjectedField pulls a column out of a joined list and gives it a name you can use as if it lived on the main list.
3. ViewFields is not optional, and this is the part I’d been getting away with rather than getting right. A projected field that isn’t listed there comes back absent, with no error. Same query, same joins, one <FieldRef> removed: the column is simply not on the item.
4. ToCamlQuery(), not ToString(). Bare ToString() gives you <Joins> and <ProjectedFields> as two sibling roots with no <View> around them, which isn’t a well-formed document at all. SharePoint accepts it anyway and hands back all 1,000 rows, minus the projected columns, because there’s no <ViewFields> in a fragment. A silent wrong answer is worse than an exception, and I’d shipped that line.
5. The values come back as FieldLookupValue, not string, even when the source column is plain text. The text is in .LookupValue, which trips people up the first time.

Every approach below returned an identical result set once normalised, checked by hashing the rows and comparing against a hash computed from the seed data. The wire shapes differ wildly, as you’ll see. The data doesn’t, and that part matters more than the timings: a fast query that quietly returns 998 rows instead of 1,000 isn’t fast, it’s broken.

One Fat Request Beats Four Lean Ones

1,000 items in the main list, 2 warmup runs discarded, 10 timed runs per approach, interleaved so no approach got a systematically better slot. Every arm asks for the minimum it needs and no more, and every arm is chunked or paged at the ceiling I measured for that specific API rather than at a number I liked the look of. That last part cost me a rerun, and I’ll come back to it.

* RenderListDataAsStream, not GetItems. REST exposes CAML two ways and only one of them returns the projected columns, which is the next section and the single most useful thing I learned building this. The row also carries that endpoint’s view chrome, a couple of dozen fields per row that nothing can trim, so it is paying for bytes the CSOM join never sends.

Graph needs a request per list because it can’t traverse a lookup. It isn’t quite empty-handed: select the lookup’s internal name and you get its display value alongside the id, so OrderDetailLookUp comes back as "Order 00000" next to OrderDetailLookUpLookupId: "1". That’s one column from the related item, for free, and it’s the one the lookup was configured to show. Any other column of that item, or another hop beyond it, needs its own request. My chain wants customerNo and customerName three lists away, and a lookup has exactly one ShowField, so Graph fetches all four lists and joins them in C#.

Four requests, one per list. That number used to be seven, and the three extra ones were mine, not Graph’s: I had paged at $top=999 out of habit, because 999 is the ceiling stuck in my head from /users and directory objects. It isn’t a SharePoint number. Graph’s default here is 200 a page and it honours $top literally, which I only established by seeding a 1,200-item list and counting the first page: $top=999 gives exactly 999, $top=1001 gives exactly 1,001.

Don’t read a 200 OK as a yes, either. $top=100000 is accepted too, and just returns the list. Graph never complains about an over-large $top, so a non-error tells you nothing and the only honest test is to hold more rows than you think the ceiling is. Mine ran out at 1,200 without finding one.

The OData row is the interesting loser. It’s the leanest of the lot on the wire, because odata=nometadata is compact and I only selected the lookup id columns. It still loses on time, because four sequential round trips cost more than one fat one. Being economical doesn’t help when you’re economical four times in a row.

CAML Isn’t a CSOM Feature, and I Had That Wrong

Here’s the correction. I’ve been writing as though CAML joins were a CSOM capability, and framing the choice as CSOM versus REST. That’s wrong twice over.

CAML is SharePoint’s query language, not a CSOM feature, and SharePoint REST will run it. There are two routes, and they don’t behave the same:

POST /_api/web/lists(guid'<list-id>')/GetItems
{ "query": { "ViewXml": "<View>...</View>" } }

POST /_api/web/lists(guid'<list-id>')/RenderListDataAsStream
{ "parameters": { "ViewXml": "<View>...</View>" } }

They don’t share a body shape, which is the first small tax. GetItems takes an SP.CamlQuery under query; RenderListDataAsStream takes a parameter bag under parameters. Send odata=nometadata and drop the __metadata annotation the old docs show, or you get The property '__metadata' does not exist on type 'SP.CamlQuery'.

RenderListDataAsStream runs the join and gives you the projected columns:

{ "Row": [{
    "ID": "1",
    "Title": "Main 00000",
    "customerNo": "C-F0026",
    "customerName": "Filler Company 0026"
}]}

That’s the full three-hop join, over REST, one request. It’s the CAML join, SharePoint REST row in the table above, and it beats every approach that doesn’t join except one, which is the next section.

That snippet is trimmed, though, and the trimming matters. RenderListDataAsStream doesn’t answer in the shape OData or CSOM do. The rows live under Row instead of value, every value is a string ("ID": "1", not "Id": 1), and each row arrives with a couple of dozen fields you never asked for: PermMask, FSObjType, UniqueId, ContentTypeId, SMTotalSize, ScopeId, owshiddenversion, and FileRef in four separate encodings. <ViewFields> doesn’t trim any of it, and neither does RenderOptions: 2. I’d assumed that one was the “just the list data” flag; the docs define ListData as “Return list data (same as None)”, so it’s the default output under a more promising name. It’s a view-rendering endpoint, not an API, and you deserialize it accordingly.

GetItems is where it gets nasty. Send the same join and you get 200 OK, the right number of rows, and no customerNo or customerName anywhere in the response. The columns are simply gone.

Before you write to tell me I forgot <ViewFields>: I didn’t, and I checked that specifically, because it’s the obvious answer and Microsoft is explicit that a projected field has to be named there too. It was in the request. The same <ViewFields>, the same joins, the same projected fields come back fine over RenderListDataAsStream and fine over CSOM. Take the <FieldRef> out over CSOM and the value disappears exactly as documented, so the mechanism works and this isn’t it. GetItems drops the projection with the request fully formed.

I assumed that meant the join had been ignored. It hasn’t, and I made myself prove it rather than infer it from a row count. I put a <Where> on the projected customerNo and compared the returned item ids against the exact set my test data says should match: 10 of 1,000 at one value, 500 of 1,000 at another, and zero for a customer that doesn’t exist. All three came back as exact set matches, not just matching counts. The clincher is that customerNo isn’t a column on that list at all, so there is nothing local for the filter to resolve against; the projection is the only path.

So the join executes server-side, the filter across three lists is correct, and only the projected columns are lost on the way out. GetItems is usable for finding items by a value two lists away. You just can’t read that value back, which makes it a fine filter and a useless projection.

Graph has neither route. Be careful how you state that, though, because “Graph can’t do lookups” is too strong and I had it too strong myself until Rob Windsor’s post made me go and check. Graph reads a lookup’s value perfectly well. What it cannot do is treat that lookup as a path to the rest of the related item.

I checked properly rather than taking anyone’s word for it. Ask Graph to expand a lookup and it tells you exactly what’s wrong:

Parsing OData Select and Expand failed: Could not find a property
named 'OrderDetailLookUp' on type 'microsoft.graph.listItem'.

The metadata says the same thing more permanently. Pull https://graph.microsoft.com/v1.0/$metadata and listItem declares six navigation properties: analytics, documentSetVersions, driveItem, fields, permissions and versions. A lookup column is not among them, and could not be. Your columns live inside fields, declared as <EntityType Name="fieldValueSet" BaseType="graph.entity" OpenType="true" />: an open type, whose properties are whatever your columns happen to be called. fields itself is a navigation property, which is why $expand=fields works at all, but nothing inside it is one, because a column invented at runtime can’t be declared in a schema. There is nothing for $expand to follow. That’s a design decision, not a gap in the documentation.

Which is why you get the display value and nothing more. The lookup’s ShowField is copied into the field bag as a value, so it travels with the item. Everything else stays behind in the other list. One thing I did not test: SharePoint lets you tick extra columns when you create a lookup, under “Add a column to show each of these additional fields”. Those are secondary lookup columns on the list holding the lookup, and Graph models them as ordinary columns with lookup.primaryLookupColumnId pointing back at the primary, so they’d presumably ride along the same way. That only ever spans one hop, so it wouldn’t have rescued a three-list chain, but if your relationship is a single hop it’s worth knowing about before you write a second request.

Batching Gets You the Round Trip, Not the Query Count

This is the nuance I’d have missed if I’d only counted round trips, and it is where the join’s lead is narrowest.

All three stacks can put several queries in one HTTP request. CSOM queues GetItems calls onto one context and sends them in a single ExecuteQuery. SharePoint REST has POST /_api/$batch, multipart and awkward but real. Graph has POST /v1.0/$batch, JSON and pleasant, capped at 20 sub-requests: the 21st comes back Number of requests inside batch exceed the limit.

And batching works. Here is the row I did not want to find:

The REST one is uglier than it should be, because SharePoint’s $batch is multipart rather than JSON. Each part is a whole HTTP request with its own headers, and the blank lines are load-bearing:

POST /_api/$batch
Content-Type: multipart/mixed; boundary=batch_a1b2c3

--batch_a1b2c3
Content-Type: application/http
Content-Transfer-Encoding: binary

GET https://the-tenant.example/sites/x/_api/web/lists(guid'<id>')/items?$select=Id,Title&$top=5000 HTTP/1.1
Accept: application/json;odata=nometadata

--batch_a1b2c3--

Repeat the part per query. The response comes back as multipart/mixed too, one HTTP/1.1 200 block per sub-request in the order you sent them, which you then have to pull the JSON out of yourself. Graph’s version is a JSON array of {id, method, url} and is far nicer to write, but it may reorder its responses, so key them by id rather than by position.

On REST, batching four plain queries beats the join. Not on round trips, which tie at one, but on time and on an order of magnitude of bytes. The join is still the fastest thing overall, but “always join” is the wrong lesson: it’s “always join on CSOM”. The REST join loses because RenderListDataAsStream ships a megabyte of view chrome nobody asked for, and four lean OData queries in one envelope simply carry less.

So batching genuinely buys the round trip. What it does not buy is the query count, and that column is the one to read. The join is one query. The batched arms are three or four, riding together. Whether SharePoint charges you per request or per operation is not something my data settles, and the throttling docs say resource units are counted per operation inside a batch, so I would not assume the envelope is free.

Batching also can’t filter. It drags all four lists back whole, 3,051 rows to answer a question about 1,000 of them, because you don’t know which ids you need until the first response comes back. Which brings me to the claim I’ve been repeating for a year, and which this exercise did not so much confirm as dismantle.

I’ve been saying “6 resource units versus 2”, on the basis that a multi-item query costs 2 resource units. Go back to the throttling page and read which sentence that table sits under: “Microsoft Graph APIs have a predetermined resource unit cost per request.” Much further down, past four more tables and into the “How to handle throttling?” section, the same page says “CSOM and REST don’t have a predetermined resource unit cost, and they usually consume more resource units than Microsoft Graph APIs to achieve the same functionality.”

So the 2-units-per-query figure was never a price for the CSOM queries I was applying it to. My arithmetic borrowed Graph’s price list for a different shop. In fairness the same page does bless the number as an estimate, “you can estimate the request rate using an average of 2 resource units per request”, which is a reasonable way to size a request rate and not a per-call cost you get to multiply by four and quote as a fact. The direction of the argument survives, because fewer queries is still fewer queries. The certainty doesn’t.

And I should say where I said it, because it’s still sitting there in my own writing. The 2025 joins post I linked at the top counts the units out query by query in its code comments and closes on “66% fewer resource units”. The CSOM performance playbook runs the same arithmetic. Both of them are wrong in the same way, this paragraph is the correction, and I’d rather point at them than quietly hope you don’t click through.

I also can’t replace it with a measurement. The documented per-app limits are real and my measurements match them: 1,250 resource units per minute and 1,200,000 per 24 hours, which is the row for tenants up to 1,000 licences. What you can’t easily do is watch the meter. Microsoft’s developer blog says “when the application has consumed 80% of its resource unit quota SharePoint will start to send RateLimit headers”, and it never says which window that 80% is measured against. I found out by pushing: 1,416 requests in 14 seconds and the headers appeared, carrying RateLimit-Limit: 1250. That’s the per-minute budget, not the daily one. It’s also a state that evaporates in about ten seconds, which makes it useless as a measuring instrument.

And here’s the part that made me stop trying. The same throttling page, in a section titled RateLimit headers, currently says: “SharePoint Online does not return or support IETF RateLimit headers.” I have them in a response body from this afternoon. The commit history explains it better than the page does: that section was headed “RateLimit headers - preview” until a docs change on 7 August 2026 deleted it, with the note that there had been a preview and it was no longer there. So what I caught was a preview being switched off underneath me, eight days before I went looking. Two sections higher the page still lists “Use the Retry-After and RateLimit HTTP headers” as a best practice, which it hasn’t got round to retracting.

Honour Retry-After, treat a RateLimit header as a bonus that has already been withdrawn once, and don’t build a measuring instrument on either. It does mean nobody should be quoting per-call resource unit costs at you, including me.

Then I Added a WHERE

The part of the original post I like most is filtering on a field several lists away. So the second scenario queries the OrderTasks list and filters on the customer’s number, two hops out:

CamlexNET.Interfaces.IQuery query = Camlex.Query()
    .Where(x => (string)x["customerNo"] == customerNo)
    .LeftJoin(x => x["OrderDetailLookUp"].ForeignList(Orders))
    .LeftJoin(x => x["CustomerLookUp"].PrimaryList(Orders).ForeignList(Customers))
    .ProjectedField(x => x["customerNo"].List(Customers).ShowField("customerNo"))
    .ProjectedField(x => x["customerName"].List(Customers).ShowField("customerName"))
    .ViewFields(["ID", "Title", "customerNo", "customerName"]);

The generated Where targets the projected field directly, and the value type is Text, not Lookup:

<Where>
  <Eq>
    <FieldRef Name="customerNo" />
    <Value Type="Text">C-BROAD</Value>
  </Eq>
</Where>

Before running it I assumed the join would lose here. Filtering first and walking backwards is the obvious smart move: find the customer, find their orders, find those orders’ tasks. Three tiny, highly selective queries instead of one join across the whole list.

It doesn’t win. Filtering to 500 of 1,000 rows, the join answers in one request and 168 ms; the best reverse walk needs three requests and 365 ms, and the worst needs eight and 1,658 ms. The full table is below, at the shape you’re more likely to ship.

Reverse traversal doesn’t collapse because the idea is bad. It collapses because neither REST nor Graph has an in operator for a set of ids, so 500 ids become a chain of or clauses chopped into pieces that fit inside a query string. Thirteen requests for REST, eight for Graph, which gets to send far longer chains, and I’ll come back to why.

Batching rescues some of that, but less than you’d hope, and it’s worth being precise about why. The stages are sequential: you can’t ask for the orders until the customers have answered. So you can only batch within a stage, which is why REST reverse goes from thirteen requests to two and not to one. The thirteen queries are all still there.

At the other end, filtering down to the 10 rows that match C-NARROW, reverse traversal finally gets somewhere: REST does it in 2 requests and 187 ms, batched or not. Still slower than the same filter as a CAML join over CSOM, which answers in 1 request and 108 ms. And look at what those ten rows cost on the wire: REST reverse moves 1,607 bytes, the CAML join moves 10,229, and CSOM walking forward moves 1,825,534. Same ten rows. That is what client-side filtering means.

More Than One Customer, Which Is What You Actually Write

A single Eq is the demo. Real code usually asks for a set: give me the tasks for these three customers. That’s an <In> over the projected field, and it’s fair to ask whether the operator surface on a projected column is as complete as on a real one.

It is. I checked <In>, <BeginsWith> and <Neq> against the ids my test data says should match, over both CSOM and REST, and all six runs came back as exact set matches:

So here’s the whole field, at the shape you’d actually ship: 520 rows belonging to three customers, and what it costs to get them without a join. The Filter column is the one that repays reading. Only the join and the reverse walks filter on the server. Every client row fetches the entire entry list and discards what doesn’t match with a Contains in C#, because customerNo lives two hops from the list being queried and nothing but the join can express that predicate to SharePoint. A forward arm that filtered server-side would have to resolve the customer ids first, at which point it has become a reverse arm. That isn’t a straw man I built, it’s the shape of the problem.

* RenderListDataAsStream again.

The join answers in 169 ms and one request. The cheapest thing that isn’t a join costs 1.5x that and only by giving up the filter, the honest reverse walk costs 2.2x, and the worst case is 10x and nine round trips.

Worth being precise about why the reverse arms blow up, because “no in operator” is only half of it. They do use or, and it works fine: $filter=(OrderDetailLookUpId eq 2) or (OrderDetailLookUpId eq 3) or .... What kills them is that the chain has to fit in a query string, not a URL, and each stack has its own idea of how long that may be. I bisected both rather than assuming they matched:

• SharePoint REST refuses at 2,048 characters of query string, which is ASP.NET’s maxQueryStringLength default showing through. 46 clauses go through at 2,016 characters, 47 come back 401 at 2,059.
• Graph goes more than twice as far, to somewhere just past 4,625 characters. 91 clauses pass, 92 come back 404 with an empty UnknownError and no explanation at all.

Each escaped clause costs about 43 characters either way, so REST fits about 44 ids per request and Graph about 86. An in operator would cost about 6 characters per id, which is why its absence hurts: same query, seven times the requests.

This is the rerun I owed you. I had originally chunked both stacks against a single 1,900 character budget measured on the whole URL, which is wrong twice: the cap is on the query string, and Graph’s path carries a 90 character site id that was eating into a budget SharePoint never had to pay. Fixing it took Graph’s reverse walk from nineteen requests to nine, which is the sort of correction that makes your own argument weaker and your numbers worth reading.

Three customers become 520 order tasks. REST needs twelve chunked calls to read them, on top of the one that found the orders. Thirteen round trips to answer one question. The more you ask for, the worse not joining gets.

What REST and Graph Actually Refuse

I checked the walls rather than assuming them, and the errors are worth having:

• Two-level $expand in REST: 400. The docs never spell out a depth limit, but they do warn you off the shape, in one sentence you could read past: “Bulk expansion and selection of related items isn’t supported.” In practice you name one field of one lookup and that’s your lot.
• Reaching a lookup id through an expand ($select=OrderTaskLookUp/OrderDetailLookUpId): also 400. If this worked, OData would do the whole chain in two calls.
• $filter=Id in (1,2,3) in REST: 400. No in operator, hence the or chains.
• Graph filtering on a column that lives on the lookup target: 400. You can only filter fields/CustomerLookUpLookupId, the integer.
• Graph in (...) on a lookup id: 400. But or chaining works, which is the only reason Graph’s reverse arm is nine requests and not 500.
• Graph expanding a lookup at all: 400, “Could not find a property named ‘OrderDetailLookUp’ on type ‘microsoft.graph.listItem’”. Try it inside fields instead and the type name changes but the answer doesn’t: $expand=fields($expand=OrderDetailLookUp) is 400, “Could not find a property named ‘OrderDetailLookUp’ on type ‘microsoft.graph.fieldValueSet’”.
• Asking for a path through a lookup, $expand=fields($select=OrderDetailLookUp/Title): 200, which looks like a win until you read it. fields contains exactly one thing, "OrderDetailLookUp": "Order 00000", the lookup’s own display value. The /Title was quietly ignored rather than honoured or rejected.

REST does have one trick I didn’t expect: you can filter on an expanded lookup’s column, even one that isn’t the lookup’s ShowField. $filter=CustomerLookUp/customerNo eq 'C-NARROW' with $expand=CustomerLookUp returns 200. That’s what gets the OData route down to 2 requests on the narrow filter.

Gotchas

• GetItems returns your join without the projected columns. 200 OK, correct row count, columns missing, <ViewFields> present and correct. The filter still works, so if you only need the ids, it’s fine. If you need the values, use RenderListDataAsStream.
• A projected field must also be in <ViewFields>. Leave it out and the column isn’t on the item. No error, no warning, no empty string, just absent. CAMLEX won’t add it for you either.
• ProjectedFields is a whitelist, and it’s shorter than the docs let on. I provisioned one column of each disputed type and tried to project it. Text, Number, DateTime and Currency came through. Choice, person, yes/no, hyperlink and every flavour of multi-line text failed, including the plain one-line Note that Microsoft’s own list says is allowed. The error is Value does not fall within the expected range, which tells you nothing about which column it means. My workaround is to store the value in a plain text column, because I know that projects, and then put column formatting on it so the user still gets the coloured pill or the icon they’d have got from a choice column. Same experience in the list, and the column survives a join.
• LookupId="TRUE" or you’re filtering on display text. Filter a lookup column without it and CAML compares against the shown value, not the id. You get zero rows and no error, which is the worst combination.
• The In clause has two limits, not one. 500 values really is a hard cap: 500 returns, 501 throws Value does not fall within the expected range. The other one, repeated everywhere and traced to an archived 2013 article, is 60 values, past which SharePoint is said to stop treating the column as indexed so a big list throws the list view threshold error instead. My lists are 1,000 items, so I never met that one and I would not take it on faith. At 1,000 items my no-join arm went from 4 requests to 6 purely because of that chunking.
• SharePoint REST pages at 100 by default, Graph at 200. Forget $top and a 1,000 item list is ten round trips before you’ve done anything interesting.
• 999 is not a SharePoint number. It’s the directory-object ceiling from /users, and I paged a whole benchmark at it out of habit. Graph honours $top literally on list items: 999 gives 999, 1,001 gives 1,001. It also returns 200 OK for $top=100000 without honouring anything in particular, so a non-error tells you nothing. If you want to know your real page size, put more rows in the list than you think the cap is and count the first page.
• The query string cap is 2,048 characters on SharePoint, and mine answered 401. Not 414, and not the 400 that ASP.NET’s own docs promise for this. You get “The length of the query string for this request exceeds the configured maxQueryStringLength value” under an Unauthorized status code, which sends you off debugging your token. Push much further past it and the status changes again, to 404. An escaped or clause costs about 43 characters, so plan on roughly 45 ids per request.
• Graph’s ceiling is its own, and roughly twice SharePoint’s. 4,625 characters, so about 86 ids per request, and it announces the limit with a bare 404 and an empty UnknownError. If you chunk both stacks against one number you will silently hand Graph half the requests it needed, which is exactly what I did for the first draft of this post.
• RenderListDataAsStream is not a drop-in swap for the OData shape. Rows under Row, not value. Every value is a string, ids included, so parse rather than cast. And you get the view’s own fields whether you want them or not: RenderOptions: 2 removed nothing for me, and the docs say why, since ListData is defined as “same as None”.
• CAMLEX still works, and ToString() is not the method you want. Camlex.Client.dll 5.4.3 built and ran fine against CSOM on .NET 10, which I mention because the last NuGet release is from July 2024 and people ask. Use ToCamlQuery(); ToString() returns fragments, and SharePoint will run them and quietly leave your projections out.

Wrapping Up

The advice survives contact with a stopwatch, but not in the shape I brought to it. Two things didn’t survive. I’d been selling the join as a CSOM feature and it isn’t: CAML runs over REST too, and comes through intact on RenderListDataAsStream. And “the join always wins” is now false, because on REST a $batch of four ordinary queries beats the REST join on time and on bytes. What’s left is narrower and I think truer: the CAML join over CSOM is the fastest way to do this, and the only one that keeps the filter on the server without walking the chain backwards first.

The honest caveats: one tenant, one geography, warm caches, ten timed runs per approach, so anything inside about 20% is noise. Absolute numbers won’t be yours. The ratios probably will be.

And the gap grows with the ask. One customer, one row set, and the alternatives are merely slower. Three customers and 520 rows, and the reverse walks turn into thirteen and nine round trips, not because or doesn’t work but because 520 ids don’t fit in one query string on either stack.

Those are the numbers after I tuned every arm to the ceiling I measured for it: <In> at its 500-value cap, SharePoint’s or-chains at 2,048 characters, Graph’s at 4,500 just under its measured 4,625, Graph’s pages at a $top that actually returns the list. The first draft of this post had all three set to something I’d guessed, and every one of the guesses happened to favour the join. Fixing them cost the join some of its margin and cost me an afternoon, and it’s the only version of the table I’d defend.

My rule of thumb, now with receipts behind it and one honest amendment: if your lists are connected by lookup columns, make SharePoint do the join. If you can’t, batch. The round trips come back either way, but the join is the only thing that filters server-side without making you resolve the ids yourself first, and that’s the part you’d otherwise have written by hand in C# while holding an entire list in memory to do it.

Every number above is from lists of 1,000 items. Past 5,000 the rules change, one of these approaches starts returning partial answers with a 200 OK, and the REST filter trick stops working entirely. I ran the whole thing again at 10,000: what still works at the list view threshold.

The syntax just looks nothing like what you’d write in CSOM, and the request that gets it wrong doesn’t tell you.

The Shape

PATCH /sites/{site-id}/lists/{list-id}/items/{item-id}/fields

{
  "ProductsLookupId@odata.type": "Collection(Edm.Int32)",
  "ProductsLookupId": [6, 7, 8]
}

What’s happening here?

1. The writable property is the column’s internal name plus LookupId, not the column name. Graph documents this for reads and is silent about writes, but it’s the same convention.
2. The value is a plain array of integer ids from the target list. No objects, no LookupValue, no wrapper.
3. The @odata.type annotation sits in a sibling property with the same name. Some report it isn’t strictly required. Include it anyway - it costs nothing and it’s the difference between a write and a shrug.

Creating an item works the same way, with the body nested under fields. Clearing the column is "ProductsLookupId": [].

The failure mode worth knowing: send the CSOM-flavoured Collection(SP.FieldLookupValue) with a results wrapper and you get 204 No Content and a column that stays empty. Nothing in the response suggests you did anything wrong.

Person Columns Are Just Lookups

A Person or Group column is a lookup into the hidden User Information List, so the write is identical:

{
  "ReviewersLookupId@odata.type": "Collection(Edm.Int32)",
  "ReviewersLookupId": [12, 13, 27]
}

Single-value version is { "ReviewerLookupId": "12" }.

The catch is where those numbers come from. They’re site-collection user ids, not Entra object ids and not Graph user ids. Graph has no /sites/{id}/users endpoint, so you read them out of the hidden list yourself:

GET /sites/{siteId}/lists?$filter=displayName eq 'User Information List'
GET /sites/{siteId}/lists/{uilId}/items?$expand=fields($select=id,EMail,Name,Title)

And then the real gap: if a user has never been referenced on that site, they have no entry in the list and therefore no id, and Graph offers no way to create one. That’s EnsureUser, and it only exists in SharePoint REST and CSOM:

POST https://{site}/_api/web/ensureuser

{ "logonName": "i:0#.f|membership|someone@the-tenant.example" }

So the write path for person columns can’t be pure Graph unless you can guarantee the users already exist on the site. In a backend I skip the dance and do the whole thing in CSOM with Web.EnsureUser() and a FieldUserValue. In a browser client I resolve or create with the REST call, then write the item with Graph.

Reading Them Back

Lookup fields aren’t returned by default, and the display value needs an explicit $select inside the $expand:

GET /sites/{site-id}/lists/{list-id}/items?$expand=fields($select=id,Title,ProductsLookupId,Products)

You get the id and the display value, and that’s it. Any other column from the target list is a second query. Twelve lookup fields per query is the documented ceiling.

Gotchas

• Files.ReadWrite.All alone makes multi-value lookups read back as []. Populated column, empty array, no error. Add Sites.Read.All and the values appear. Single-value lookups are unaffected, which is what makes this one so slow to diagnose.
• Internal names, not display names. Product_x0020_Line, not Product Line. Get them from /lists/{id}/columns.
• 400 on the array means the column is single-value. Check lookup.allowMultipleValues or personOrGroup.allowMultipleSelection on the column definition before blaming the syntax.
• It’s logonName, not loginName. Get it wrong and ensureuser answers with InvalidClientQueryException, which tells you nothing about the spelling.
• ensureuser needs a SharePoint-audience token. https://{tenant}.sharepoint.com/.default, not your Graph token. Two audiences, two sets of permissions to consent.
• Old Graph SDK 5.x chokes on non-string AdditionalData. CurrentDepth (1000) is equal to or larger than the maximum allowed depth of 1000 on anything that isn’t a string, ints and arrays included. Fixed in current packages, so upgrade first and only reach for raw JSON through the request adapter if you’re pinned.

Wrapping Up

Multi-value lookups and person columns are writable from Graph. The rule is <InternalName>LookupId plus an integer array plus Collection(Edm.Int32), and the CSOM-shaped payload you’d expect to work is exactly the one that fails quietly.

The one thing Graph genuinely can’t do here is mint a user id that doesn’t exist yet. That’s still an EnsureUser call, and it’s part of why I keep a playbook for which SharePoint API I reach for instead of committing a whole feature to one of them.

// Outer safety net: retry any activity that throws a transient error (e.g. throttling
// that slips past HTTP-level retry). Exponential backoff: 5s, 10s, 20s, ... up to 5min,
// 10 attempts total. All activities are idempotent, so retrying the full activity is safe.
private static readonly TaskOptions ActivityRetryOptions = new(
    new RetryPolicy(
        maxNumberOfAttempts: 10,
        firstRetryInterval: TimeSpan.FromSeconds(5),
        backoffCoefficient: 2.0,
        maxRetryInterval: TimeSpan.FromMinutes(5)));

Read that comment again: “All activities are idempotent, so retrying the full activity is safe.”

That sentence is either the best thing about the whole engine or a lie that duplicates customer sites at 2am. There’s no middle ground. The retry policy reruns the entire activity, not the line that failed. If your activity created a site and then died resolving its ID, the retry creates the site again - unless the activity was written to survive being run twice.

So this post is a catalog of how I actually make that sentence true. Five patterns, all from real provisioning code.

Pattern 1: Check Before Create

The site creation activity is the highest-stakes one. If a previous attempt created the site but failed a moment later (say, while resolving the Graph site ID), the retry must not create it again. So it probes first:

// Idempotency: if the site already exists (e.g. previous attempt created it but
// failed during Graph ID resolution), skip creation and go straight to Graph lookup.
if (await SiteExistsAsync(newSiteUrl))
{
    logger.LogInformation("Site already exists at {SiteUrl}, skipping creation.", newSiteUrl);
}
else
{
    await CreateSiteAsync(parsedType, site, siteAlias, newSiteUrl);
}

Note what makes the probe possible in the first place: the site URL is deterministic, built from a prefix and alias in the schema. No timestamps, no random suffixes. If the URL contained Guid.NewGuid(), attempt two would probe a different URL, find nothing, and create a second site. Deterministic naming is the quiet prerequisite for check-before-create.

Pattern 2: “Already Exists” Is Success

Sometimes you can’t probe cheaply, so you attempt the operation and translate the failure. Activating a site feature that’s already active comes back from SharePoint as an error - but it means the activity already did its job on a previous run:

catch (InvalidOperationException ex) when (
    // "Feature already activated" comes back as HTTP 500 with odata.error.code
    // "System.Data.DuplicateNameException" - locale-independent, unlike the human message.
    ex.Message.Contains("System.Data.DuplicateNameException", StringComparison.OrdinalIgnoreCase))
{
    logger.LogInformation("{Feature} feature is already active, skipping.", displayName);
}

The comment is a scar. My first version matched on the English error text, which works great until the code runs against a tenant in another language. Match on error codes, not error messages. The OData error code survives localization; “Feature … is already activated” does not.

Graph makes this pattern cleaner because it gives you a real status code. Copying folder structures into the new site:

catch (ODataError ex) when (ex.ResponseStatusCode == 409)
{
    // Item already exists in target - idempotent on retry
    logger.LogInformation("Item '{ItemName}' already exists in target, skipping copy.", itemName);
    return;
}

409 Conflict on a create is not a failure. It’s a receipt from your previous attempt.

Pattern 3: Upsert by Nature

The best idempotency is the kind you get for free by choosing the right API. The registration activity writes the new site’s URL and ID back to an inventory list, and it does so with SharePoint’s ValidateUpdateListItem against a known item ID. Setting the same fields on the same item to the same values twice is indistinguishable from doing it once. There’s nothing to guard because the operation has no failure mode for repetition.

When you’re designing an activity and you get to pick between “add a row” and “set fields on item X”, pick the second. Every Add needs a guard; a keyed Set guards itself.

Pattern 4: Verify After Write

The dark twin of idempotency: some SharePoint writes report success and then don’t stick. Property bag values on a freshly created site are notorious for this. The activity’s answer is to re-read everything it just wrote and throw if reality disagrees:

if (mismatches.Count > 0)
{
    throw new InvalidOperationException(
        $"Property bag verification failed on {siteUrl}: {string.Join("; ", mismatches)}. " +
        "Values did not persist to the server; retrying the activity.");
}

What’s happening here?

1. After writing the property bag values, the activity loads Web.AllProperties fresh from the server.
2. Each expected key/value is compared against what actually persisted.
3. Any mismatch throws a plain transient exception, which hands the problem to the orchestrator’s retry policy.
4. The rerun is safe precisely because setting a property bag key is an upsert - pattern 3 again. Verification and idempotency are two halves of the same loop: writes are repeatable, so failed verification can simply demand a repeat.

This inverts the usual relationship with retries. Instead of retries being something inflicted on the activity, the activity uses a throw to request one.

Pattern 5: Idempotent Deletes Too

Removal flows have the same problem in the other direction. Removing a user from a site’s members group when they’re already gone throws a ServerException, and treating that as failure would wedge every cleanup retry:

catch (ServerException ex)
{
    logger.LogWarning(ex, "Could not remove {LoginName} from {SiteUrl} ({ErrorType}); treating as already removed.",
        loginName, siteUrl, ex.ServerErrorTypeName);
    return;
}

Desired state: user is not in the group. User is not in the group. Done. Idempotency is about converging on a state, not performing an action.

Gotchas

• One non-idempotent activity poisons the whole pipeline. I had a security activity that created SharePoint groups without checking for existing ones. Every retry of that step risked duplicate groups, which meant retries weren’t safe, which meant the retry policy comment was a lie for the whole orchestration. It’s disabled until it earns its way back in. Idempotency is all-or-nothing per pipeline.
• The failure window is between success and checkpoint. The orchestrator records completed steps, but a crash can land after the activity finished and before the record was written. The completed-steps list narrows how often reruns happen; idempotency is what makes the remaining reruns harmless. You need both.
• catch the narrowest thing you can. The 409 handler above catches ODataError with status 409, nothing else. A broad catch { return; } also “makes retries pass”, by swallowing real failures. Idempotency guards should be precise enough that a genuinely broken call still throws.
• Test by running it twice, literally. My smoke test for a new activity is: run the provisioning, then immediately queue the exact same provisioning again. Zero errors and zero duplicates or it doesn’t ship.

Wrapping Up

A Durable Functions retry policy is a contract: it promises to rerun your activities, and your activities promise not to care. Check before create, read “already exists” as success, prefer upserts, verify writes that lie. Write every activity as if it will run twice, because on a long enough timeline, it will.

If retry policies and replay are new to you, Durable Functions: A Function That Sleeps for a Week covers what an orchestration actually is, and why “it will run twice” is a feature rather than a bug.

Then you hit the one field, the one property, the one checkbox Graph doesn’t do. And you hit it on day three, with the feature already shaped around Graph.

Sometimes rewiring it is cheap. Sometimes it’s a rewrite. Either way you paid.

I’ve walked into that wall enough times now that I stopped calling it bad luck and started writing down which tool I reach for when. This post is that list.

The 200 That Meant Nothing

The worst one was metadata on uploaded documents. My add-in uploads a file with Graph, then applies the library’s columns to the resulting list item with a PATCH to /listItem/fields. Text columns, choice columns, dates, all good.

Then a managed metadata column.

200 OK. Empty field.

Not a 400. Not “column type not supported”. Graph accepted the payload, answered like everything was fine, and wrote nothing. The Learn page for updating a listItem shows you Color and Quantity and never mentions which column types don’t make it through.

A thrown error costs you an hour. A silent success costs you until someone notices the taxonomy column is blank on documents that were archived weeks ago, and then it costs you a data migration.

That’s the real argument for having a playbook. Not elegance. The cases where the wrong choice fails quietly.

Be careful about which columns you write off, though. I had multi-value lookups and person columns on my “Graph can’t” list for a long time, and both were wrong. They work fine, they just don’t take the shape you’d guess coming from CSOM, and that’s its own post. Managed metadata is the one that genuinely has no Graph write path.

Graph First Is a Default, Not a Plan

To be clear, “Graph first” is a good instinct. PnP Core builds it in: the SDK favours Graph when reading SharePoint data and falls back to SharePoint REST when the requested properties aren’t available there, and you can flip GraphFirst off if you disagree. I’ve written before about why I’m still on PnP.Framework, and that per-operation routing is the thing I most want from PnP Core.

Until I have it, I’m the router. So here’s how I route.

CSOM Owns List Items

Roughly 95% of my list item CRUD is CSOM, and it’s not nostalgia. Three capabilities keep it there.

Batching without a hard ceiling. CSOM queues operations until you call ExecuteQueryAsync(), and around 100 operations per batch is the reliable sweet spot. Graph’s $batch caps at 20 requests, and a failed batch needs picking apart before you retry it. When I’m updating 500 items, that difference is 5 round trips versus 25.

CAML joins. One query across lists connected by lookup columns, filtered on a field two lists away, merged server-side. There’s no Graph equivalent, and I’ve never found a way to fake it that didn’t end in merging rows in C#.

A way past the list view threshold. Query a big list and CSOM throws The attempted operation is prohibited because it exceeds the list view threshold. There’s a flag on CamlQuery that gets you through it, combined with paging and indexed columns. That one deserves its own post and it’s on my list.

The rest of my CSOM reasoning is in the CSOM performance playbook, and the join pattern has its own post.

Graph Owns Files

For files it flips completely. Uploading and downloading through Graph is fewer requests and noticeably faster in every project I’ve measured it in, and large files are the clearest case: createUploadSession gives you a resumable, chunked upload with a pre-authenticated URL, and the chunks don’t carry an Authorization header at all.

// Small files: straight PUT to the content endpoint
if (file.size <= SIMPLE_UPLOAD_LIMIT) {
  return folder.concat(`:/${filename}:/content`).put(file);
}

// Large files: a session, then sequential chunks
const session = await folder.createUploadSession({ name: filename });
for (const chunk of chunks) {
  await session.resumableUpload.upload(chunk.length, chunk, contentRange);
}

The rules that matter: chunk sizes must be a multiple of 320 KiB, they go up sequentially, and each PUT extends the session expiry. Get the multiple wrong and the upload fails at the last chunk, which is a fun way to spend an afternoon.

Downloads get the same treatment - zipping a whole folder out of SharePoint is a Graph job for me, not a CSOM one. Graph’s JSON batching maps cleanly onto “fetch these 20 files’ content”, though matching responses back to requests has its own quirks that I covered in Graph batching for file content.

SharePoint REST Is the Escape Hatch

I almost never choose SharePoint REST. I end up there when it’s the only thing that works, which turns out to be more often than the modern-API story suggests. From my current projects:

• Setting a navigation link to open in a new tab. It’s a checkbox in the UI, it’s not in the PnP provisioning schema, and it’s not on CSOM’s NavigationNode. It’s MenuState/SaveMenuState. I have a whole post coming about that one.
• Reading the sites a user follows: /_api/social.following/my/followed(types=4).
• Joining a hub site, activating site features, applying a site design, adding an available content type to a library. All _api calls in my provisioning engine.
• ValidateUpdateListItem, which is what rescues the metadata story from the top of this post.
• And /_api/web/ensureuser, because Graph has no EnsureUser and no /sites/{id}/users endpoint at all.

The pattern is consistent: the older and more SharePoint-specific the concept, the more likely REST is the only place it lives.

ValidateUpdateListItem is worth knowing by name. It takes form values in SharePoint’s own wire format - taxonomy as Label|GUID, people as a JSON array of claim keys - and it’s the same payload the classic edit form posts, which is exactly why it accepts the column types Graph won’t touch. In a C# backend it’s a method on ListItem in the client library; in a browser add-in with no CSOM available, it’s the REST endpoint. Either way, read the response: rejected fields come back with HasException: true inside an otherwise successful call, so you can recreate the silent-200 problem from the other direction if you don’t look.

The EnsureUser gap is the more interesting one, because it’s what stops person columns from being a pure Graph story. The ids in ReviewersLookupId are site-collection user ids from the User Information List. Not Entra object ids, not Graph user ids. If a user has never been referenced on that site, they simply have no id, and Graph gives you no way to create one. So you POST /_api/web/ensureuser with a logonName first, then write the item with Graph. Or you stay in CSOM and let Web.EnsureUser() plus a FieldUserValue do both in one place, which is what I usually do in a backend.

One nice asymmetry on the taxonomy side: reading the term store works fine over Graph (/sites/{id}/termStore/sets/{id}, with TermStore.Read.All). It’s only writing a term onto a list item that Graph won’t do. So my term picker is Graph and my term save is not.

Gotchas

• A 200 is not proof. This is the whole post in one bullet. Graph accepts unsupported column types and writes nothing. Assert on the value you wrote, at least once per column type, in a real library. Otherwise you find out at migration time.
• Mixing APIs means mixing tokens. A Graph token does not work against /_api. SharePoint REST wants an audience of https://<tenant>.sharepoint.com/.default, so a flow that uses both acquires two tokens and your app registration needs both sets of permissions. Plan the consent, not just the code.
• Lookups are FieldNameLookupId, not FieldName. Graph names the writable property differently from the column. Writing to the column name silently does nothing. Yes, silently.
• Switching to ValidateUpdateListItem needs the list item id. Drive item ids don’t work, so $select=sharepointIds on the Graph item first and use sharepointIds.listItemId.
• It’s logonName, not loginName. The ensureuser parameter is spelled the unintuitive way, some docs get it wrong, and the wrong spelling gets you an InvalidClientQueryException that says nothing useful.
• Chunk sizes are a multiple of 320 KiB or nothing. Graph upload sessions fail on the final commit, not on the offending chunk, so the error points at the wrong place.
• “Unsupported” is sometimes just undocumented. Multi-value lookups and person columns sat on my can’t-do list for far too long. Before you route an operation to the older API, check whether the modern one only lacks a doc page.

Wrapping Up

There’s no winner here. CSOM, SharePoint REST and Graph are three drawers in the same toolbox, and picking per operation beats picking per project.

My rule of thumb: Graph for files, CSOM for list items, SharePoint REST when nothing else can do it - and verify the write whenever you cross a boundary. The failure mode that actually hurt me wasn’t choosing the slower API. It was choosing the API that said yes and did nothing.

That kind of library is exactly the moment to pick your foundation on purpose. So I sat down and looked hard at PnP Core.

And then I stayed on PnP.Framework.

What Makes PnP Core Tempting

The thing I actually want from PnP Core isn’t a feature. It’s that somebody already made a decision I keep having to make myself.

With CSOM and raw calls, every time I read something out of SharePoint I have to think about how: SharePoint REST, or Microsoft Graph? Which one has this property? Which one is faster here? PnP Core removes that question. Per the docs, the SDK “by default is configured to favor the Microsoft Graph API when you’re reading SharePoint data assuming the requested properties are available via Graph”, and it falls back to SharePoint REST when they aren’t. If you disagree, you flip GraphFirst to false on the PnPContext and it prefers REST instead.

Someone with more context than me picked the better call per operation, and I get to write the intent instead of the plumbing. Add async-first, a real domain model, and modern .NET hosting, and it’s not a close comparison on paper.

The problem is that my code doesn’t run on paper.

Blocker 1: The Provisioning Engine Isn’t There

Almost everything I build for SharePoint eventually provisions something. Lists, content types, pages, navigation. That’s ApplyProvisioningTemplate, and it lives in PnP.Framework. PnP Core has no provisioning engine.

So a migration wasn’t “rewrite my queries”. It was “rewrite my queries and keep PnP.Framework around anyway for the part that does the heavy lifting”. Two SDKs, two mental models, no win.

That’s the part I’d already made peace with, until pnpframework issue #1237 showed up in June: a roadmap for moving both the Provisioning Engine and the Modernization Engine into PnP Core as dedicated projects, with the PnP.Framework versions deprecated afterwards. Target is Q4 2026.

That single issue turns my biggest objection into a waiting game. I don’t need PnP Core to grow a provisioning engine as a favor to me. It’s on the roadmap, in the open, with a phase 2 already referenced for the wider PnP.Framework deprecation. Everything I’ve written about splitting templates into modular files for retry resilience survives that move, because it’s a property of how you structure templates, not of which SDK applies them.

Blocker 2: CAML Joins Come Back Empty

The second one I couldn’t wait out, because it’s the query pattern I lean on hardest.

SharePoint’s CAML supports joining lists and pulling fields across the join with <Joins> and <ProjectedFields>. It’s the difference between one query and four, and I’ve written a whole post on doing it with CAMLEX in CSOM because it saves resource units and spares you from merging rows in C#.

Run that same view XML through PnP Core’s LoadItemsByCamlQueryAsync and nothing breaks. No exception, no error, no warning. You get items back, the join is honored for filtering, and the projected fields simply aren’t in the result. The REST GetItems endpoint doesn’t serialize them, so they never make it into the response for the SDK to map.

A silent hole in the data is worse than a thrown exception, and it’s a hole in the one query shape I most wanted to keep.

So I filed issue #1799 and then PR #1802 with an implementation: two new methods on IList that run the query through CSOM’s List.GetItems(CamlQuery) instead of REST, because CSOM does return projected fields.

/// <summary>
/// Loads list items based up on a CAML query executed via CSOM, which also returns
/// fields projected from a joined list (CAML Joins/ProjectedFields)
/// </summary>
public Task<ICamlQueryCsomResult> LoadItemsByCamlQueryViaCsomAsync(CamlQueryOptions queryOptions);

public ICamlQueryCsomResult LoadItemsByCamlQueryViaCsom(CamlQueryOptions queryOptions);

Using it looks like any other PnP Core call. This is the shape from the integration test in the PR:

var page1 = await list.LoadItemsByCamlQueryViaCsomAsync(new CamlQueryOptions()
{
    ViewXml = viewXml,
    DatesInUtc = true
});

var firstItem = page1.Items[0];
var projected = firstItem["ProjectedText"] as IFieldLookupValue;
Console.WriteLine(projected.LookupValue);

// Next page, without hand-building a paging string
var page2 = await list.LoadItemsByCamlQueryViaCsomAsync(new CamlQueryOptions()
{
    ViewXml = viewXml,
    DatesInUtc = true,
    PagingInfo = page1.PagingInfo
});

What’s happening here?

1. LoadItemsByCamlQueryViaCsomAsync takes the exact same CamlQueryOptions as the REST-based method, so the view XML with <Joins> and <ProjectedFields> is unchanged. Only the transport differs.
2. The projected field comes back typed as an IFieldLookupValue, not as a raw string. That’s how CSOM represents it and it’s what you’d expect from a lookup coming across a join, so LookupId and LookupValue are both there.
3. The returned ICamlQueryCsomResult carries Items plus PagingInfo, taken from CSOM’s ListItemCollectionPosition. Feed PagingInfo back into the next call and you get the next page. It’s null when there are no more pages.
4. The loaded items are also merged into the list’s Items collection, the same as the existing CAML methods, so nothing about the surrounding model changes.

Gotchas

• The failure mode is silence, not an error. No exception on a join query with projected fields means your code happily maps a null and moves on. If you’re evaluating PnP Core for join-heavy work, assert on the projected value in a test, don’t eyeball the item count.
• A roadmap is a plan, not a shipped release. Issue #1237 has a Q4 2026 target and it’s still open. Same for my PR: open, no reviewer assigned yet. I’m building on the current state of both SDKs, not on the version I hope exists in six months.
• No batch variant for the CSOM path, on purpose. The items are materialized immediately, so it doesn’t fit PnP Core’s batching model. If you’re used to queueing everything into a batch, this one call stands apart.
• Deprecation warnings are part of the plan. Once the engines land in PnP Core, the PnP.Framework equivalents get marked deprecated and stop taking new features. So if you’re searching for whether PnP.Framework is deprecated and whether the provisioning engine is moving to PnP Core: yes and yes, on a published roadmap, and the two blockers above are what stand between “announced” and “usable”. Staying put is a decision with an expiry date, which is exactly why I’d rather have the blockers resolved than keep postponing.

Wrapping Up

I’m not avoiding PnP Core because I prefer PnP.Framework. I’m on PnP.Framework because two specific things weren’t there: the provisioning engine, and CAML joins that actually return the joined data. Both now have an issue number attached, and one of them has my name on the pull request.

Rule of thumb: pick the SDK by what your workload actually needs today, then go make the gap smaller instead of waiting for someone else to. Filing #1799 took an evening. It moved my migration date more than another month of reading release notes would have.

When the provisioning engine lands in PnP Core and projected fields come back from a join, I’m switching and I’m not looking back.

First instinct: call the API straight from the web part. It works in twenty minutes. Then you press F12, open the network tab, click any request… and there it is. The company’s master API key, in plain text, readable by every single user who ever loads that intranet page.

And it’s not “they can see the data the web part shows anyway” - the web part shows a filtered slice. The key unlocks the whole API: every customer, every record, every write operation the key is licensed for. Anyone who copies it out of the inspector can call the API from Postman on their couch.

Why the Browser Can’t Keep a Secret

It’s worth being blunt about this, because the temptation to “just obfuscate it a bit” is real:

• Everything the browser sends is in the network tab.
• Everything the bundle contains is in the sources tab (source maps or not, strings are strings).
• Everything the app holds in memory is one breakpoint away.

There is no hiding place in the client. And “it’s only our internal SharePoint” doesn’t help - internal users are exactly the people who shouldn’t be walking around with the master key to a system they’re only supposed to see a corner of.

The Proxy Function

The fix is a small reverse proxy. One catch-all Azure Function fronts the entire third-party API:

private static readonly HashSet<string> _hopByHop = new(StringComparer.OrdinalIgnoreCase)
{
    "Connection", "Keep-Alive", "Proxy-Authenticate", "Proxy-Authorization",
    "TE", "Trailer", "Transfer-Encoding", "Upgrade"
};

[Function("ApiProxy")]
public async Task<IActionResult> Run(
    [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", "put", "delete",
                 Route = "api/{*path}")] HttpRequest request,
    string? path,
    CancellationToken ct)
{
    var targetUri = _settings.BaseUrl.TrimEnd('/') + "/" + (path ?? "")
                  + (request.QueryString.Value ?? "");

    using var upstream = new HttpRequestMessage(new HttpMethod(request.Method), targetUri);

    if (request.Method is "POST" or "PUT" || request.ContentLength is > 0)
    {
        upstream.Content = new StreamContent(request.Body);
        if (request.ContentType is not null)
            upstream.Content.Headers.TryAddWithoutValidation("Content-Type", request.ContentType);
    }

    foreach (var (key, values) in request.Headers)
    {
        // Never forward the client's Host, any API key they try to sneak in,
        // or hop-by-hop headers that belong to *this* connection only.
        if (key.Equals("Host", StringComparison.OrdinalIgnoreCase) ||
            key.Equals("X-Api-Key", StringComparison.OrdinalIgnoreCase) ||
            _hopByHop.Contains(key))
            continue;

        if (!upstream.Headers.TryAddWithoutValidation(key, (IEnumerable<string?>)values!))
            upstream.Content?.Headers.TryAddWithoutValidation(key, (IEnumerable<string?>)values!);
    }

    // The swap. The secret exists only here, on the outbound leg.
    upstream.Headers.Add("X-Api-Key", _settings.ApiKey);

    var client = _httpClientFactory.CreateClient("upstream-api");
    using var response = await client.SendAsync(upstream, HttpCompletionOption.ResponseHeadersRead, ct);

    _logger.LogInformation("Proxy: {Method} {Path} → {StatusCode}", request.Method, path, (int)response.StatusCode);

    var httpResponse = request.HttpContext.Response;
    httpResponse.StatusCode = (int)response.StatusCode;

    foreach (var (key, values) in response.Headers.Concat(response.Content.Headers))
    {
        if (_hopByHop.Contains(key)) continue;
        httpResponse.Headers.Append(key, values.ToArray());
    }

    await response.Content.CopyToAsync(httpResponse.Body, ct);
    return new EmptyResult();
}

What’s happening here?

1. Route = "api/{*path}" is a catch-all. GET /api/customers/123/tasks becomes GET https://the-api.example/customers/123/tasks, query string included. One function fronts the whole API surface - when the vendor adds endpoints, the proxy needs zero changes.
2. The header loop deliberately strips any inbound X-Api-Key. A caller can’t inject their own key or override yours - whatever they send in that header dies at the proxy.
3. The hop-by-hop set (Connection, Transfer-Encoding, …) is the detail naive proxies get wrong. Those headers describe one connection, not the request; forwarding them causes wonderfully confusing breakage.
4. The real key is added to the outbound request only. Response headers get copied back to the browser, but request headers are never echoed - so the key physically cannot appear in the inspector. It’s not hidden; it’s absent.
5. ResponseHeadersRead + CopyToAsync streams the upstream response straight through without buffering it in the function’s memory, and upstream status codes pass through untouched so the web part can react to a 404 or a 429 honestly.

Where the Key Lives

Server-side config, strongly typed, validated at boot:

services.AddOptions<ApiSettings>()
    .Bind(configuration.GetSection("UpstreamApi"))
    .Validate(s => IsConfigured(s.BaseUrl), "UpstreamApi:BaseUrl must be configured.")
    .Validate(s => IsConfigured(s.ApiKey), "UpstreamApi:ApiKey must be configured.")
    .ValidateOnStart();

private static bool IsConfigured(string value) =>
    !string.IsNullOrWhiteSpace(value) && !value.StartsWith('<');

The value itself comes from Function App settings - ideally as a Key Vault reference - never from a committed file. ValidateOnStart() plus the <placeholder> guard means a misconfigured secret fails the deployment at boot, not at the first user’s click three days later.

Hiding ≠ Authorizing

Here’s the part that’s easy to skip and shouldn’t be: as shown so far, the proxy hides the key from the browser, but anyone who discovers the function URL can call it - and burn your API quota with your key. We’ve moved the secret, not secured the door.

Lock the function to Entra ID. Create an app registration for the function API, expose a scope (the convention is access_as_user), and have the function validate incoming JWTs - signature, issuer, and audience, not just parsing the claims out. Parsing is reading the name tag; validating is checking the ID.

On the SPFx side, this is pleasantly little work. Declare the permission in package-solution.json:

"webApiPermissionRequests": [
  {
    "resource": "my-function-api",
    "scope": "access_as_user"
  }
]

And call the function with AadHttpClient - SPFx acquires and attaches the user’s token for you:

const client = await this.context.aadHttpClientFactory
  .getClient("api://<function-app-registration-client-id>");

const response = await client.get(
  `https://my-function.azurewebsites.net/api/customers/${customerCode}/tasks`,
  AadHttpClient.configurations.v1
);

Now open the inspector again. There is a token in the request - but it’s the user’s own token: short-lived, scoped to your function only, and useless against the third-party API. That’s the whole difference between a secret and an identity. A stolen API key is everyone’s master key forever; a stolen user token is one person’s function access for an hour.

Gotchas

• Watch what you log. Log method, path, and status - never headers. Otherwise the key you carefully kept out of the browser ends up in Application Insights, readable by everyone with portal access.
• You now front someone else’s quota. The vendor’s rate limits hit your key for all users combined. If the web part is chatty, add caching or throttling in the proxy before the vendor does it for you.
• CORS is on you. The browser is calling your function from a SharePoint origin - configure CORS on the Function App for your tenant’s SharePoint domain, not *.
• Consider narrowing the surface. A catch-all proxy forwards everything, including endpoints the web part never needed. If the API has destructive operations, whitelist the methods and paths you actually use.

Wrapping Up

A shared API key belongs on a server. Full stop. The moment it ships to a browser, every user has it, and you can’t take it back - most vendors will happily rotate the key for you, but you’ll be doing that dance on their schedule, not yours.

The proxy costs about fifty lines: catch-all route, header hygiene, one server-side header swap, and Entra ID on the front door. That turns “every intranet user carries the master key” into “every request is a named user, calling a locked endpoint, seeing exactly what the web part shows them”. 🔒

The first time step seven failed, I got to watch my engine try to create a site that already existed.

That run taught me the rule this post is about: in a long provisioning flow, you don’t roll back when something fails. You checkpoint, and you resume forward.

If Durable Functions are new to you, start with my intro to what they are and what they can do - this post builds on it.

Why Rollback Is the Wrong Instinct

When provisioning fails at step seven, you have a half-built site. The textbook answer is a compensation saga: undo steps six through one in reverse order. Delete the folders, detach the template, deactivate the features, delete the site, then recreate everything from scratch on the next attempt.

I started sketching that and stopped halfway through the list, because every line was either dangerous or absurd. Deleting a site collection to work around a failed navigation tweak is using a crane to hang a picture. And some steps don’t even have an undo - you can’t meaningfully “unapply” a provisioning template that merged fields into existing lists.

Here’s the thing rollback ignores: the six completed steps aren’t damage. They’re progress. The site is fine; what’s missing is the steps that haven’t run yet. So the only recovery that makes sense is forward: figure out where the run stopped, and continue from there.

That reframes the problem completely. I don’t need compensation logic. I need to know, reliably, which steps finished.

Carry the Progress in the State

Durable Functions makes this natural, because an orchestrator already passes state to each activity and gets state back. The trick is to make “what’s done” part of that state. My orchestration state looks something like this:

public sealed record SiteSetupState(
    string CustomerId,
    string SiteUrl,
    string[] CompletedSteps
);

And the orchestrator walks its steps like this:

var state = context.GetInput<SiteSetupState>() ?? await CreateSite(context, customerId);

string[] pipeline =
[
    nameof(ActivateFeatures),
    nameof(ApplyTemplate),
    nameof(SeedFolders),
    nameof(AddMemberGroups),
    nameof(RegisterSite),
];

foreach (var step in pipeline)
{
    if (state.CompletedSteps.Contains(step))
    {
        logger.LogInformation("{Step} already completed, skipping.", step);
        continue;
    }

    await context.CallActivityAsync(step, state, retryOptions);

    state = state with { CompletedSteps = [.. state.CompletedSteps, step] };
    context.SetCustomStatus(state);
}

What’s happening here?

1. The pipeline is just an ordered list of activity names. Nothing clever, and that’s the point - the interesting machinery is around the calls, not in them.
2. Before each step, the orchestrator checks whether this step already ran. On a fresh run the array is empty and nothing is skipped. On a resumed run, this check is what fast-forwards past the finished work.
3. After each successful step, the state is copied with the step name appended. An immutable with copy, so nothing mutates in place, which keeps replays honest.
4. SetCustomStatus publishes the updated state on the orchestration instance. This is the checkpoint. It costs one line.

That last line is doing more work than it looks like. Custom status is readable from outside the orchestration - through the management API, without touching the orchestration history. So the same call gives you two things: anyone polling the instance sees live progress (“three of six steps done”), and if the run fails, the last published state is sitting right there, telling you exactly where it stopped.

Resume Is Just Input

Because the checkpoint is a plain serializable record, resuming a failed run doesn’t need any special framework support. You read the failed instance’s custom status, and you start a new orchestration with that state as input:

var failed = await client.GetInstanceAsync(failedInstanceId, getInputsAndOutputs: true);
var checkpoint = failed.ReadCustomStatusAs<SiteSetupState>();

await client.ScheduleNewOrchestrationInstanceAsync(
    nameof(SiteSetupOrchestrator),
    checkpoint);

The new run enters the same loop, finds five steps in CompletedSteps, skips them in about a millisecond, and picks up at step six. No site deletion, no re-creation, no duplicate template apply. The half-built site becomes a five-sixths-built site, then a finished one.

I like how little there is to this. The “resume feature” is the skip-check in the loop plus the fact that the input type and the checkpoint type are the same type. That’s it.

The Gap That Idempotency Covers

One honest caveat. The checkpoint is written after the activity succeeds, so there’s a window: the activity finishes, the process dies before the checkpoint lands. On resume, that step’s name isn’t in CompletedSteps, and it runs again.

You can’t close that window - it’s inherent to doing the work and recording the work as two operations. What you do instead is make every activity safe to run twice: check before create, treat “already exists” as success, write with upserts. That’s a full topic on its own, but the division of labor is worth stating plainly: the checkpoint decides how often steps rerun, idempotency decides whether reruns hurt. You need both. The checkpoint alone has a crash window; idempotency alone means re-executing ten minutes of finished work on every hiccup.

And never checkpoint before the call to close the window from the other side - then a crashed step gets skipped on resume, which is far worse. A step that runs twice is a wasted minute; a step that runs zero times is a broken site that says “Completed”. Ask me how I know.

Gotchas

• Use the replay-safe logger. Orchestrator code replays from history every time the instance wakes up. context.CreateReplaySafeLogger(...) keeps your logs from repeating every completed step on each replay. A regular ILogger in an orchestrator will gaslight you.
• No clocks, no GUIDs in the orchestrator. DateTime.UtcNow, Guid.NewGuid(), and Random produce different values on replay and corrupt the history. Anything nondeterministic belongs inside an activity, including generated names and timestamps you want in the state.
• Custom status has a size limit (16 KB of JSON). A record with a customer ID, a URL, and an array of step names fits hundreds of times over, but don’t stuff a whole template or file manifest in there. Checkpoint the position, not the payload.
• Step names are a contract. The moment CompletedSteps is persisted anywhere - a failed instance you might resume next week - renaming an activity breaks the match and the step silently reruns (fine, if idempotent) or the resume misbehaves (not fine). Rename with the same care you’d give a database column.
• Report progress somewhere humans look. Custom status is great for machines; my engine also writes the current step name to a status column in the site inventory list, inside a try/catch that logs and swallows. A cosmetic status write must never kill a provisioning run.

Wrapping Up

Long provisioning flows fail in the middle, so design for resuming instead of undoing: carry a list of completed steps in the orchestration state, publish it as custom status after every step, skip completed steps on rerun, and feed the saved state back in to resume. Rollback is for databases. Provisioning goes forward.

For a long time, slow CSOM was just annoying. Users waited, someone made coffee, life went on. Then I sat down with a client to calculate the cost of Service Prioritization in SharePoint, where every CSOM call has an actual price tag: USD $1.00 per 1,000 calls. Suddenly all those wasted calls weren’t just slow. They were an invoice.

That changed how I write CSOM. I count round trips now, the way you’d count database queries in a hot loop. Over the past year I’ve written several posts about the specific techniques that came out of that habit, and this post ties them together into one playbook.

Every ExecuteQuery Is a Road Trip

Here’s the mental model that makes all four techniques click: every ExecuteQueryAsync() is a road trip to the SharePoint server. Doesn’t matter if you’re delivering one package or a hundred, the drive takes the same time. Network latency, authentication, server processing - the overhead is per trip, not per operation.

So the whole playbook boils down to four questions:

1. Can I deliver more packages per trip? (batching)
2. Can one trip cover several destinations? (CAML joins)
3. Is this trip even necessary? (change detection)
4. Can I avoid a second trip when something goes wrong? (ExceptionHandlingScope, and the taxonomy trick)

Let’s take them one at a time.

Rule 1: Batch or Suffer

The most common sin. A loop that loads items one by one:

// ❌ 50 items = 50 round trips = 3-5 seconds
foreach (var id in itemIds)
{
    var item = list.GetItemById(id);
    context.Load(item);
    await context.ExecuteQueryAsync();
    ProcessItem(item);
}

CSOM happily queues up operations until you call ExecuteQueryAsync(). Around 100 operations per batch is the reliable sweet spot:

// ✅ 50 items = 1 round trip = ~200-500ms
var items = chunk.Select(id => {
    var item = list.GetItemById(id);
    list.Context.Load(item);
    return item;
}).ToList();

await context.ExecuteQueryAsync();

In my testing, that’s a 10x+ improvement for the price of restructuring a loop. It works for reads, writes, deletes, all of it.

Full post with the reusable ProcessInChunks helper: Batch Your CSOM Operations Instead of Looping One by One.

Rule 2: Join Lists, Don’t Query Them One by One

Related data across multiple lists is where round trips multiply quietly. Customers in one list, orders in another, order items in a third, so you write three queries and merge the results in C#. Every multi-item query costs 2 resource units toward your throttling budget, so three lists cost 6 units plus three round trips plus the merging code nobody wants to maintain.

CAML supports joins, even though the SharePoint UI never hints at it. One query, one trip, 2 resource units, and the server does the merging. I use CAMLEX instead of raw CAML XML because lambda expressions are readable and the XML it generates is not:

var query = Camlex.Query()
    .Where(x => (string)x["CustomerName"] == "Contoso") // filter on a field 2 lists away!
    .LeftJoin(x => x["OrderTaskLookUp"].ForeignList(ListGuidOrderTasks))
    .LeftJoin(x => x["OrderDetailLookUp"].PrimaryList(ListGuidOrderTasks).ForeignList(ListGuidOrders))
    .LeftJoin(x => x["CustomerLookUp"].PrimaryList(ListGuidOrders).ForeignList(ListGuidCustomers))
    .ProjectedField(x => x["CustomerName"].List(ListGuidCustomers).ShowField("CustomerName"));

var camlQuery = new CamlQuery { ViewXml = query.ToString() };
var items = ordersList.GetItems(camlQuery);

That’s 66% fewer resource units than the three-query version, and you can even filter on a value several lists away. The lists must be connected via lookup columns, and ProjectedFields only supports simple field types (text, number, date - not user, choice, or managed metadata fields).

Full post with the raw CAML comparison and the field type list: Efficient Multi-List Queries in CSOM: Using CAML Joins with CAMLEX.

Rule 3: Did Anything Actually Change?

Users are save-happy. They open a form, change nothing, and click save anyway. Automated syncs are worse. When I dug into a typical day of API logs on one project, roughly 70% of our SharePoint update calls weren’t changing anything. SharePoint accepts identical data with a smile and bills you for the privilege.

The fix is change detection before the update:

var differences = GetDifferences(listItem, newValues, treatEmptyStringAsNull: true);
if (differences.Any())
{
    foreach (var (fieldName, newValue) in differences)
    {
        listItem[fieldName] = newValue;
    }
    listItem.Update();
    clientContext.ExecuteQuery();
}
// Nothing changed? Do absolutely nothing.

The hard part is the comparison itself. SharePoint field types fight you: user fields where only LookupId matters, multi-choice fields that come back in random order, taxonomy values, dates with kind mismatches. Naive oldValue == newValue doesn’t survive contact with any of them, and JSON serialization comparison turned out 3-5x slower than a proper normalizing comparer when I benchmarked both.

Full post with the complete comparison engine: SharePoint CSOM: Prevent Unnecessary Updates and API Calls.

Rule 4: Let the Server Do the Catching

This one came straight out of that Service Prioritization cost analysis. The client’s solution made about 90,000 CSOM calls per month, and 88% of them were EnsureUser calls for users who were already on the site. The classic defensive pattern - always ensure the user before setting a user field - was two round trips per assignment, and the first one was almost always pointless.

ExceptionHandlingScope lets you ship try-catch logic to the server and resolve it in one round trip:

var scope = new ExceptionHandlingScope(clientContext);
using (scope.StartScope())
{
    using (scope.StartTry())
    {
        // Optimistic: works for users already known to the site
        listItem["AssignedTo"] = FieldUserValue.FromUser("user@company.com");
        listItem.Update();
    }
    using (scope.StartCatch())
    {
        // Fallback: only runs server-side if the try failed
        var user = clientContext.Web.EnsureUser("user@company.com");
        listItem["AssignedTo"] = FieldUserValue.FromUser("user@company.com");
    }
}
clientContext.ExecuteQuery(); // one trip

Result for that client: a 75-85% reduction in monthly calls. One honest caveat, and it’s important: the scope reduces round trips, not server requests. Each operation inside still counts toward throttling. The call reduction came from the optimistic-first logic, and ExceptionHandlingScope is what made that logic affordable.

Full post with the cost breakdown: SharePoint’s Server-Side Try-Catch: ExceptionHandlingScope.

Rule 5: Keep Taxonomy Out of Your ViewFields

Taxonomy fields are the slowest thing you can put in a CAML query. On a list where items carried 5 to 50 terms across multiple managed metadata fields, loading a few hundred items took 90-120 seconds. Users literally walked away from their computers.

The fix has two parts. First, query the list without the taxonomy fields in ViewFields, which is where most of the cost hides. Then load the taxonomy data separately through FieldValuesForEdit, where SharePoint stores it as a raw string you can parse directly:

// Raw format in FieldValuesForEdit: "Label1|GUID1;Label2|GUID2"
foreach (ListItem[] chunk in items.Cast<ListItem>().Chunk(100))
{
    foreach (var item in chunk)
    {
        clientContext.Load(item, i => i.FieldValuesForEdit);
    }
    await clientContext.ExecuteQueryAsync();
    // parse Label|GUID pairs per item
}

That took the same load from 90-120 seconds down to 25-35 seconds, a 70-75% improvement. Fair warning: this parses an internal, undocumented format. It’s been reliable for me, but test it in your environment and keep the traditional approach as a fallback.

Full post with the parser: Load SharePoint Taxonomy Fields Fast With FieldValuesForEdit.

Which One First?

If your workload is write-heavy (forms, syncs, integrations), start with change detection. It’s the only technique that eliminates entire operations instead of making them cheaper, and it’s invisible to users.

If your workload is read-heavy, start with batching. It’s the smallest code change for the biggest win, and the chunking helper is reusable everywhere. And if those reads span related lists, add joins next: they cut resource units, not just latency, which stretches your throttling budget further.

ExceptionHandlingScope is for when a specific fallback pattern (like EnsureUser) dominates your call logs. Check your logs first - I only found the 88% figure because I went looking. And the taxonomy trick is a targeted weapon: only reach for it when managed metadata is measurably your bottleneck, because it carries the undocumented-format risk.

The good news is they stack. Batched updates that skip unchanged items, with server-side fallbacks, is exactly how my current projects run.

Gotchas

• 100 operations per batch is the sweet spot. CSOM reliably handles around that many. Bigger batches mean bigger payloads and less predictable behavior across environments. Ask me how I know.
• ExceptionHandlingScope doesn’t reduce throttling pressure by itself. Round trips go down, but every operation inside the scope still counts as a request. The savings come from the optimistic-first logic it enables.
• Your numbers will differ from mine. The 70%, 75-85%, and 70-75% figures are from real projects, but latency, item complexity, and tenant load all move them. Measure before and after in your own environment.
• Test under throttling before you trust any of it. Dev Proxy can simulate 429 responses locally so you find out how your batches behave under pressure before production does.
• The taxonomy trick gives you label and GUID, nothing more. If you need full term paths or custom properties, you’re back to the taxonomy API for those items.
• Joins need lookup columns and GUIDs. CAML joins only work across lists connected by lookup columns, and referencing lists by GUID instead of title saves you from “list does not exist” surprises.

If you’re also downloading files in the same solution, the round-trip mindset applies there too - I’ve covered that in downloading multiple files from SharePoint.

Wrapping Up

Count your round trips before SharePoint counts them for you. That’s the whole playbook in one sentence: fewer trips (batching), combined trips (CAML joins), no pointless trips (change detection), no second trips (ExceptionHandlingScope), and lighter trips (taxonomy loading).

Each technique stands on its own, so grab the one that matches your bottleneck:

• Batch your SharePoint operations
• Join multiple lists in one CAML query
• Prevent unnecessary updates
• Server-side try-catch with ExceptionHandlingScope
• Fast taxonomy loading

Both approaches work. Both have real trade-offs. And for a long time I did not have a strong opinion either way. Then I started testing with Dev Proxy and suddenly the answer became very clear to me.

What Is a PnP Provisioning Template?

If you have not worked with PnP provisioning before: a PnP Provisioning Template is an XML (or JSON) file that describes the structure of a SharePoint site. It can contain:

• Lists and libraries — custom columns, views, and settings
• Content types — site or list content types with their fields
• Pages and web parts — modern pages, hero sections, text blocks
• Navigation and branding — site navigation structure, themes, logos

You apply a template to a site using the PnP Framework library for .NET, which reads the file and provisions everything described in it. The library handles the order of operations, dependencies between objects, and a lot of edge cases you would rather not think about.

The typical entry point looks like this:

web.ApplyProvisioningTemplate(template, applyInfo);

One call. One template. PnP handles the rest — until throttling enters the picture.

The Monolith Approach

The simplest way to structure your provisioning is one template that contains everything. All lists, all content types, all pages, all navigation — in a single file.

Pros:

• ✅ Simple to manage — one file, one version, one deployment artifact
• ✅ Easy to apply — a single ApplyProvisioningTemplate call
• ✅ PnP handles dependencies between objects internally

Cons:

• ❌ All-or-nothing retry — if anything fails, you restart from the beginning
• ❌ Long templates take a long time to apply, which means long retries too
• ❌ Hard to reason about what failed and where

The all-or-nothing behavior is the killer. SharePoint throttles aggressively under load, and ApplyProvisioningTemplate does not pick up where it left off. Under a sustained throttle — the kind where even 15 retries at the CSOM call level are not enough to get through — the template application eventually throws a MaximumRetryAttemptedException — a PnP-specific exception signalling that all retry attempts were used up. At that point your outer retry logic has no choice but to start the whole template over from the very first list.

The Modular Approach

The alternative is splitting your template into multiple smaller files, each representing a logical feature or concern. For example:

• 01-lists-and-libraries.xml
• 02-content-types.xml
• 03-pages.xml
• 04-navigation.xml

You apply them in sequence, and after each one completes successfully you record that fact in memory. If a template fails, you retry only from the failed template — not from the beginning.

Pros:

• ✅ Granular retry — only the failed feature is re-applied
• ✅ Faster recovery from throttling
• ✅ Easier to reason about failures (“pages failed, lists are fine”)
• ✅ Easier to test individual features in isolation

Cons:

• ❌ More files to manage and version
• ❌ Ordering matters — you need to think about dependencies
• ❌ Slightly more orchestration code on your end

The trade-off is real. Modular means more moving parts. But for resilience, the payoff is significant.

Testing With DevProxy — Where It Got Interesting

I wanted to put both approaches through their paces under realistic throttling conditions. For that I used Dev Proxy with two plugins:

• GenericRandomErrorPlugin — injects random 429 responses to simulate SharePoint throttling unpredictably
• RateLimitingPlugin — caps requests per second to trigger organic throttling at volume

With these configured, I ran my provisioning engine against both template structures.

The result with the monolithic template: PnPClientContext did its job — it absorbed a lot of 429s silently and kept going. But under sustained throttling, the retries ran out, PnP threw a MaximumRetryAttemptedException, and my outer retry had to restart it from scratch. From list number one. Even though we were deep into page provisioning when it failed.

The result with modular templates: the same CSOM-level retry absorbed the same throttling. But when a sustained burst finally exhausted PnP’s retries on the pages template, my retry loop skipped straight to 03-pages.xml — lists and content types had already been marked done in memory and were left alone.

This is exactly the kind of thing that looks fine in a happy-path test, but silently destroys your provisioning SLA in production.

PnPClientContext: Bump the Retry Count for Provisioning

The PnP Framework’s internal CSOM calls already use ExecuteQueryRetry() — an extension method that handles 429 throttling with exponential backoff, respecting the Retry-After header SharePoint returns. You get this for free with a plain ClientContext. The default is 10 retries.

If you expect heavier throttling during provisioning — and a large site with lots of lists, content types, and pages is a reasonable place to expect it — you can easily raise that limit by converting to a PnPClientContext:

using var siteCtx = CSOMClientFactory.Create(provCtx.NewSiteUrl, credential);
var pnpCtx = PnPClientContext.ConvertFrom(siteCtx, retryCount: 15);

That is the entire change. The provisioning code picks up the higher retry count automatically. Your outer retry logic only comes into play if throttling is so sustained that even 15 attempts on a single CSOM call are not enough — at which point PnP throws a MaximumRetryAttemptedException.

Idempotency Strategy

The key property that makes modular templates safe to retry is idempotency — applying a template that has already been applied should not cause errors or duplicate data. PnP handles most of this for you: before creating a list, content type, or page, it checks whether it already exists.

A very simple example of how this looks in practice:

foreach (var source in config.Templates)
{
    var template = LoadTemplate(source);
    pnpCtx.Web.ApplyProvisioningTemplate(template, applyInfo);
}

If this loop is re-invoked after a failure (for example by an Azure Function retry policy), templates that already completed will be re-applied — but because each template is idempotent, that is safe. The already-provisioned lists and pages are left untouched. The only cost is the time it takes to re-apply them.

That wasted time is the whole point. With a monolithic template, a retry means re-applying everything from scratch. With modular templates, it means re-applying a small number of already-done features before reaching the one that actually failed.

Comparison

Verdict

Neither approach is universally correct. If you are provisioning simple sites with low template complexity and you are not worried about throttling, a monolithic template is perfectly fine and a lot less work to maintain.

But if you are building a production provisioning engine that needs to be resilient — where throttling is a real risk, where retries need to be efficient, and where you care about how long a failure takes to recover from — modular templates win clearly.

The combination that works best for me is: modular templates + PnPClientContext with conservative retry settings. PnPClientContext handles the low-level CSOM throttling so most 429s never surface at all. When they do surface (and eventually they will), modular templates ensure your retry loop does the minimum amount of work to get back on track.

Conclusion

The biggest takeaway for me is not even the monolith-vs-modular debate. It is that I never would have found this out without DevProxy. Happy-path testing showed no difference between the two approaches. It was only when I introduced realistic throttling that the behavior diverged in a way that actually matters.

If you are building any kind of SharePoint automation that makes CSOM or Graph calls under load, DevProxy should be a standard part of your testing setup. It is free, it integrates with your local development environment, and it reveals exactly the kind of subtle failure modes that only appear in production.

References

• PnP Framework on GitHub
• Dev Proxy overview
• SharePoint throttling and retry guidance
• PnPClientContext — ExecuteQueryRetry docs

News Link pages in SharePoint — those cards that link to external news articles — are only available through the Graph API’s beta endpoint. That’s fine, beta APIs are part of life.

What’s not fine is that Microsoft’s official documentation covers the simple case well, but the moment you want to add a banner image, the C# code snippet disappears and is replaced with:

“Snippet not available.”

Great. Thanks.

The banner image upload requires a multipart request. Figuring out how to do that with the Graph Beta SDK means piecing together documentation about MultipartBody, some Kiota internals, and a few quirks you’ll only discover by actually trying it.

This post covers the full working solution.

Prerequisites

Install these two NuGet packages:

Microsoft.Graph.Beta
Microsoft.Kiota.Serialization.Multipart

Microsoft.Graph.Beta is the Beta SDK — it contains NewsLinkPage and all the beta models. Microsoft.Kiota.Serialization.Multipart provides the MultipartBody class needed to construct multipart requests. Without it, you don’t have the types to send binary data alongside the JSON payload.

The Simple Case: No Banner Image

If you just want a News Link without a banner image, the Beta SDK fluent API handles it cleanly:

var page = new NewsLinkPage
{
    OdataType = "#microsoft.graph.newsLinkPage",
    Title = "Contoso Unveils First Self-Driving Car",
    NewsWebUrl = "https://someexternalnewssite.com/article",
    Description = "A brief description of the article."
};

var result = await GraphClient.Sites[siteId].Pages.PostAsync(page, requestConfiguration =>
{
    requestConfiguration.Headers.Add("prefer", "include-unknown-enum-members");
});

Two things to note:

The Prefer: include-unknown-enum-members header is required. Without it, the API won’t return newsLink as a valid pageLayoutType value — it’s an evolvable enum that hasn’t been promoted to v1.0 yet, so Graph treats it as unknown by default.

The page is also created as a draft. You still need to publish it separately before it shows up in the news feed.

The Full Solution: With Banner Image

Here’s the complete implementation including banner image upload and publishing:

internal async Task CreateNewsLink(
    string siteId,
    string? title,
    string? url,
    string? description,
    Stream? bannerImageContent)
{
    var page = new NewsLinkPage
    {
        OdataType = "#microsoft.graph.newsLinkPage",
        Title = title,
        NewsWebUrl = url,
        Description = description,
        AdditionalData = new Dictionary<string, object>
        {
            ["@microsoft.graph.bannerImageWebUrlContent"] = "name:content"
        }
    };

    var multipartBody = new MultipartBody();
    multipartBody.AddOrReplacePart("request", "application/json", page);
    multipartBody.AddOrReplacePart("content", "image/jpeg", bannerImageContent, fileName: "banner.jpg");

    var requestInfo = new RequestInformation
    {
        HttpMethod = Method.POST,
        UrlTemplate = $"https://graph.microsoft.com/beta/sites/{siteId}/pages"
    };

    requestInfo.Headers.Add("prefer", "include-unknown-enum-members");
    requestInfo.SetContentFromParsable(GraphClient.RequestAdapter, "multipart/form-data", multipartBody);

    var response = await GraphClient.RequestAdapter.SendAsync<NewsLinkPage>(
        requestInfo,
        NewsLinkPage.CreateFromDiscriminatorValue
    );

    var createdNewslink = await GraphClient.Sites[siteId].Pages[response.Id].GetAsync(
        requestConfiguration =>
        {
            requestConfiguration.Headers.Add("prefer", "include-unknown-enum-members");
        }
    );

    var publishRequestInfo = new RequestInformation
    {
        HttpMethod = Method.POST,
        UrlTemplate = $"https://graph.microsoft.com/beta/sites/{siteId}/pages/{createdNewslink.Id}/microsoft.graph.newsLinkPage/publish"
    };

    await GraphClient.RequestAdapter.SendNoContentAsync(publishRequestInfo);
}

It looks like a lot, but each part has a specific reason for being there. Let me explain the non-obvious bits.

Breaking Down the Code

The @microsoft.graph.bannerImageWebUrlContent Annotation

AdditionalData = new Dictionary<string, object>
{
    ["@microsoft.graph.bannerImageWebUrlContent"] = "name:content"
}

This is the glue between the JSON part and the image bytes. The value "name:content" tells the Graph API: “find the image bytes in the multipart part named ‘content’”.

When the API processes the request, it reads this annotation, locates the content part in the multipart body, saves the image to the site’s assets library, and sets the bannerImageWebUrl property on the created page. The naming has to match — the part you add with AddOrReplacePart("content", ...) is what the annotation references.

Why MultipartBody Instead of the Fluent API

The normal fluent API — GraphClient.Sites[siteId].Pages.PostAsync(...) — only supports JSON payloads. There’s no overload that accepts binary data or constructs a multipart request.

MultipartBody fills that gap. You compose the request from named parts, each with their own content type:

var multipartBody = new MultipartBody();
multipartBody.AddOrReplacePart("request", "application/json", page);
multipartBody.AddOrReplacePart("content", "image/jpeg", bannerImageContent, fileName: "banner.jpg");

The "request" part carries the JSON, the "content" part carries the image bytes. The names are what you reference in @microsoft.graph.bannerImageWebUrlContent.

Why RequestInformation Directly

Since the fluent API can’t send multipart requests, we drop down one level to RequestInformation — the underlying request abstraction that all Kiota-generated clients use internally:

var requestInfo = new RequestInformation
{
    HttpMethod = Method.POST,
    UrlTemplate = $"https://graph.microsoft.com/beta/sites/{siteId}/pages"
};

requestInfo.SetContentFromParsable(GraphClient.RequestAdapter, "multipart/form-data", multipartBody);

SetContentFromParsable serializes the MultipartBody and sets the correct Content-Type header — including the boundary parameter that multipart requests require. This is one of those things you have to discover by reading the Kiota source code rather than the docs.

The Extra GET After Creation

You might notice the code fetches the page again right after creating it:

var createdNewslink = await GraphClient.Sites[siteId].Pages[response.Id].GetAsync(...);

This is a quirk of the multipart POST. The response from a raw RequestInformation-based call doesn’t go through the same deserialization pipeline as the fluent API, which means the returned NewsLinkPage object isn’t fully populated. Rather than fighting with it, a quick GET on the newly created page — with the proper prefer header — gives you a cleanly deserialized object with the correct Id to use for publishing.

Publishing the News Link

Pages are created as drafts. To make the News Link appear in the SharePoint news feed, you have to explicitly publish it:

var publishRequestInfo = new RequestInformation
{
    HttpMethod = Method.POST,
    UrlTemplate = $"https://graph.microsoft.com/beta/sites/{siteId}/pages/{createdNewslink.Id}/microsoft.graph.newsLinkPage/publish"
};

await GraphClient.RequestAdapter.SendNoContentAsync(publishRequestInfo);

Again, RequestInformation directly — the Graph Beta SDK’s fluent API doesn’t expose a typed publish method for newsLinkPage.

What About the v1.0 SDK?

Not supported yet. The NewsLinkPage type, the pageLayout: newsLink enum value, and the publish endpoint are all beta-only. When the API graduates to v1.0, the approach should be almost identical — just swap Microsoft.Graph.Beta for Microsoft.Graph.

Until then, you’re on beta. Microsoft officially cautions against using beta APIs in production, but in practice this particular API has been stable for a while. Use your own judgment.

When building automated site provisioning, every Microsoft 365 group creation triggers a welcome email to each member by default. In an automated scenario where multiple groups are created at once, that quickly turns into a flood of emails your users didn’t ask for — and they’re going to call it spam.

Microsoft documents WelcomeEmailDisabled as a supported option, but doesn’t show you how to actually set it from the Graph SDK in C#. That’s what this post is about.

The Weird Part: AdditionalData

If you look at the typed Group object in the SDK, you won’t find a ResourceBehaviorOptions property anywhere. It has to be set through AdditionalData — a catch-all dictionary for properties that exist in the Graph API but aren’t modeled as first-class typed properties in the SDK.

It works fine, but it does mean you lose IntelliSense and compile-time safety. The same pattern applies to owners@odata.bind and members@odata.bind, which let you assign owners and members at creation time without separate follow-up calls.

The Solution

Here’s the complete group creation request with welcome emails disabled, owners set, and members added — all in a single API call:

var ownerId = $"https://graph.microsoft.com/v1.0/users/{ownerObjectId}";
var groupMembers = new List<string>
{
    $"https://graph.microsoft.com/v1.0/users/{memberObjectId1}",
    $"https://graph.microsoft.com/v1.0/users/{memberObjectId2}"
};

Group requestBody = new()
{
    Description = description,
    DisplayName = displayName,
    GroupTypes = ["Unified"],
    MailEnabled = false,
    MailNickname = siteName,
    SecurityEnabled = true,
    AdditionalData = new Dictionary<string, object>
    {
        { "resourceBehaviorOptions", new List<string> { "WelcomeEmailDisabled" } },
        { "owners@odata.bind", new List<string> { ownerId } },
        { "members@odata.bind", groupMembers }
    }
};

Group? group = await _graphClient.Groups.PostAsync(requestBody);

resourceBehaviorOptions

resourceBehaviorOptions controls specific group behaviors at creation. WelcomeEmailDisabled simply stops Microsoft 365 from sending the welcome email to members.

You can also combine multiple options in the same list if needed:

{ "resourceBehaviorOptions", new List<string> { "WelcomeEmailDisabled", "HideGroupInOutlook" } }

Other supported values are documented here.

owners@odata.bind and members@odata.bind

The @odata.bind syntax binds users to the group by their full resource URL at creation time, so you don’t need separate POST /groups/{id}/members calls afterward. The URL format must be the full Graph v1.0 path:

https://graph.microsoft.com/v1.0/users/{objectId}

I was working on a SharePoint solution where we had a list with items containing multiple taxonomy fields. Each item could have anywhere from 5 to 50 taxonomy terms across different fields. The client needed to load hundreds of these items regularly.

The traditional CSOM approach was… well, let’s just say coffee breaks became very popular during load times.

The loading process was taking 90-120 seconds for a few hundred items. Users were literally walking away from their computers while waiting for data to load.

That’s when I realized we needed to completely rethink how we approach taxonomy loading in CSOM.

The Traditional (Slow) Way

Here’s what most developers do, and what I was doing initially:

// The traditional approach - including taxonomy fields in ViewFields
CamlQuery query = new CamlQuery();
query.Query = @"
    <View>
        <ViewFields>
            <FieldRef Name='ID'/>
            <FieldRef Name='Title'/>
            <FieldRef Name='CategoryField'/>     <!-- Taxonomy field -->
            <FieldRef Name='TagField'/>          <!-- Taxonomy field -->
            <FieldRef Name='OtherTaxField'/>     <!-- Taxonomy field -->
        </ViewFields>
    </View>";

ListItemCollection items = list.GetItems(query);
clientContext.Load(items);
await clientContext.ExecuteQueryAsync();

// Then access taxonomy fields normally
foreach (ListItem item in items)
{
    var categoryField = item["CategoryField"] as TaxonomyFieldValueCollection;
    var tagField = item["TagField"] as TaxonomyFieldValueCollection;
    // Process taxonomy values...
}

Problems with this approach:

• Including taxonomy fields in ViewFields is extremely slow
• SharePoint has to load and process all taxonomy data upfront
• Multiple expensive taxonomy service calls during the initial load
• No control over how taxonomy data is loaded

Result: Coffee break time. Lots of it.

The Optimization Journey

After digging into SharePoint’s internals and some experimentation, I discovered two key things:

1. Including taxonomy fields in ViewFields is what kills performance. SharePoint has to load and process all taxonomy data during the initial query, which is extremely slow.

2. SharePoint stores taxonomy data in a raw format that can be accessed via FieldValuesForEdit and parsed directly. This bypasses the expensive taxonomy service calls entirely.

The breakthrough: Separate regular field loading from taxonomy field loading. Load items with only the fields you need immediately, then load taxonomy fields separately using the faster FieldValuesForEdit approach.

Key Insights:

1. Exclude taxonomy fields from ViewFields - This is the biggest performance gain. Load regular fields first, taxonomy fields separately.
2. FieldValuesForEdit contains raw taxonomy data in the format: Label|GUID;Label|GUID - much faster to parse than TaxonomyFieldValue API
3. Batch loading is crucial - load all items’ FieldValuesForEdit in one call per chunk
4. Chunking prevents timeouts - process items in manageable batches
5. Dictionary lookups are fast - map terms back to items efficiently

The Optimized Solution

Here’s the approach that cut our load times by 70-75%:

Main Loading Logic

public async Task<List<MyItemDTO?>?> GetItemsByOptions(string[] options)
{
    // Step 1: Load list items WITHOUT taxonomy fields in ViewFields
    // This is much faster than including taxonomy fields in the initial query
    List list = clientContext.Web.Lists.GetByTitle("YourListName");
    
    CamlQuery query = new CamlQuery();
    query.Query = @"
        <View>
            <ViewFields>
                <FieldRef Name='ID'/>
                <FieldRef Name='Title'/>
                <FieldRef Name='OtherRegularFields'/>
                <!-- Notice: NO taxonomy fields here -->
            </ViewFields>
            <Query>
                <!-- Your where clause here -->
            </Query>
        </View>";
    
    ListItemCollection items = list.GetItems(query);
    clientContext.Load(items);
    await clientContext.ExecuteQueryAsync();

    if (items == null || items.Count == 0)
        return null;

    // Step 2: Process in chunks and load taxonomy fields separately
    List<MyItemDTO> result = new List<MyItemDTO>();
    
    foreach (ListItem[] chunk in items.Cast<ListItem>().Chunk(100))
    {
        // Load taxonomy fields separately using FieldValuesForEdit
        var categoryTerms = await LoadTaxonomyField(chunk, "CategoryField");
        var tagTerms = await LoadTaxonomyField(chunk, "TagField");

        // Step 3: Map everything together
        foreach (var item in chunk)
        {
            var dto = new MyItemDTO
            {
                Id = item.Id,
                Title = item["Title"]?.ToString(),
                // Map regular fields normally
            };

            // Add taxonomy terms from our separate loading
            if (categoryTerms.TryGetValue(item.Id.ToString(), out var terms))
                dto.CategoryTerms = terms;
                
            if (tagTerms.TryGetValue(item.Id.ToString(), out var tags))
                dto.TagTerms = tags;

            result.Add(dto);
        }
    }

    return result;
}

Batch Taxonomy Loading

private async Task<Dictionary<string, List<TaxonomyTerm>>> LoadTaxonomyField(
    IEnumerable<ListItem> items, 
    string taxonomyFieldInternalName)
{
    // Load FieldValuesForEdit for all items in one batch
    foreach (var item in items)
    {
        clientContext.Load(item, i => i.FieldValuesForEdit);
    }
    await clientContext.ExecuteQueryRetryAsync();

    // Get the field ID for internal format matching
    var list = clientContext.Web.Lists.GetById(listGuid);
    var field = list.Fields.GetFieldByInternalName(taxonomyFieldInternalName);
    clientContext.Load(field);
    await clientContext.ExecuteQueryRetryAsync();

    string fieldId = field.Id.ToString();

    // Parse taxonomy data from each item's FieldValuesForEdit
    return items.ToDictionary(
        item => item.Id.ToString(),
        item => ParseTaxonomyFromEditValues(item, fieldId)
    );
}

private List<TaxonomyTerm> ParseTaxonomyFromEditValues(ListItem item, string fieldId)
{
    // Find the correct field key (SharePoint uses shortened GUIDs)
    var fieldValues = item.FieldValuesForEdit.FieldValues
        .Where(x => x.Key.EndsWith(fieldId.Replace("-", "").Substring(4)));
    
    if (!fieldValues.Any())
        return new List<TaxonomyTerm>();

    string actualFieldKey = fieldValues.First().Key;
    
    if (!item.FieldValuesForEdit.FieldValues.ContainsKey(actualFieldKey))
        return new List<TaxonomyTerm>();

    // Parse the raw taxonomy string
    string rawTaxonomyData = item.FieldValuesForEdit[actualFieldKey];
    return ParseTaxonomyEditString(rawTaxonomyData);
}

Raw Format Parser

private List<TaxonomyTerm> ParseTaxonomyEditString(string editString)
{
    if (string.IsNullOrWhiteSpace(editString))
        return new List<TaxonomyTerm>();

    // Format: "Label1|GUID1;Label2|GUID2;Label3|GUID3"
    return editString
        .Split(new[] { ';' }, StringSplitOptions.RemoveEmptyEntries)
        .Select(entry => 
        {
            var parts = entry.Split('|');
            if (parts.Length == 2 && Guid.TryParse(parts[1], out Guid guid))
            {
                return new TaxonomyTerm 
                { 
                    Label = parts[0].Trim(), 
                    TermGuid = guid 
                };
            }
            return null;
        })
        .Where(t => t != null)
        .Select(t => t!)
        .ToList();
}

public class TaxonomyTerm
{
    public string Label { get; set; }
    public Guid TermGuid { get; set; }
}

Performance Impact

The results were dramatic:

Before optimization:

• 90-120 seconds for a few hundred items with multiple taxonomy fields
• Multiple ExecuteQueryRetryAsync() calls per item
• Heavy taxonomy service usage

After optimization:

• 25-35 seconds for the same dataset
• ~70-75% performance improvement
• Minimal API calls (2 per chunk: one for items, one for field metadata)

The exact savings depend on the number of items and terms, but the pattern held consistently across different datasets.

Should You Try This Approach?

If you’re experiencing slow loading times with taxonomy fields, this might be worth testing.

I can’t give you specific numbers for when this optimization makes sense - every environment and scenario is different. What I can tell you is that in my case, it made a dramatic difference.

My recommendation:

• If your current taxonomy loading is painfully slow, try this approach in a test environment
• Measure the before and after performance in your specific scenario
• Keep the complexity vs. benefit trade-off in mind
• Always have a fallback to the traditional approach

Remember the limitations:

• This is an undocumented approach that works with SharePoint’s internal formats
• You only get basic term information (label + GUID)
• You’ll need to test thoroughly in your environment

The traditional CSOM approach works fine for many scenarios. But if you’re hitting performance walls with taxonomy fields, this optimization might be exactly what you need.

Important Disclaimers

⚠️ This is an undocumented approach that relies on SharePoint’s internal storage format for taxonomy fields. While it has worked reliably in my experience, it’s not officially supported by Microsoft.

Key Takeaways

1. Batch operations are crucial - Load multiple items’ data in single API calls
2. Chunking prevents timeouts - Process large datasets in manageable pieces
3. Raw formats can be faster - Sometimes bypassing official APIs improves performance
4. Know when to optimize - Don’t add complexity unless you really need the performance
5. Document your risks - Be transparent about using undocumented approaches

The traditional CSOM taxonomy approach works fine for simple scenarios. But when you’re dealing with lots of items and lots of terms, sometimes you need to think outside the box.

Fast taxonomy loading is one of five techniques in The SharePoint CSOM Performance Playbook, which is where I’d start if taxonomy isn’t the only thing slowing you down.

I demoed this approach on the Microsoft 365 & Power Platform community demos call in January 2026, under the same title as this post. Those calls are worth an hour of your month if you build on this stack.

So there I was, building what I thought was a pretty straightforward API endpoint. Users fill out a form, click save, and boom—SharePoint list item gets updated. Simple, right?

Wrong.

It was during one of those routine maintenance windows when I decided to review some API logs that I noticed something odd. The numbers just didn’t add up. We were making way more SharePoint API calls than seemed necessary for the amount of actual data changes happening in the system.

Every operation was hitting SharePoint. Even when nothing had actually changed.

That’s when it hit me. Our application was treating every potential update as an actual update, regardless of whether the data was different from what SharePoint already had. Users could open a form, change nothing, click save, and we’d still fire off an API call to “update” the item with identical values.

But SharePoint doesn’t care if you’re setting a field to the exact same value it already has. It still counts that as an API call. It still hits your throttling limits. And if you’re paying for “Service Prioritization in SharePoint”? It still costs you money.

The “Why Did I Even Build This?” Moment

I spent some time digging deeper into the patterns, and the numbers were… enlightening.

This wasn’t just a user behavior issue. It was happening everywhere:

• Users frequently save forms without making actual changes
• Automated processes periodically “sync” data that’s already current
• Applications send complete object states rather than just changed fields
• Integration systems perform routine updates as part of larger workflows

When I looked at a typical day’s worth of operations, I found that roughly 70% of our SharePoint update calls weren’t actually changing anything. We were essentially paying SharePoint to accept identical data and politely say “thanks for the update!”

The more I thought about it, the more I realized this pattern was probably happening in systems everywhere. How many developers have built the same “just update everything” approach without stopping to ask whether anything actually changed?

SharePoint’s “Sure, I’ll Take Your Money” Attitude

Here’s the thing about SharePoint’s CSOM that nobody really talks about: it doesn’t care if you’re being wasteful. You want to set a field to the exact same value it already has? “No problem!” says SharePoint. “That’ll be one API call, please.”

And if you’re using Service Prioritization in SharePoint? That’s real money walking out the door.

Let me show you what I mean with some concrete numbers from a client I worked with recently:

The Head-Scratching Cases

// These should be considered equal, but .NET says "nope"
currentValue: "John Doe"
newValue: "John Doe"  // Easy case, no problem here

currentValue: null
newValue: ""  // Is empty string the same as null? Your call!

currentValue: 42
newValue: "42"  // Different types, same value. Fun times.

The SharePoint Special Cases

// User fields (oh boy...)
currentValue: FieldUserValue { LookupId = 15, LookupValue = "John Doe" }
newValue: FieldUserValue { LookupId = 15, LookupValue = "John Doe" }

// Multi-choice fields that hate you
currentValue: ["Option A", "Option C", "Option B"]
newValue: ["Option B", "Option A", "Option C"]  // Same options, different order

// Taxonomy fields (because why not make it complicated?)
currentValue: TaxonomyFieldValue { TermGuid = "abc-123", Label = "Technology" }
newValue: TaxonomyFieldValue { TermGuid = "abc-123", Label = "Technology" }

After an hour of testing different scenarios, I realized I needed something more sophisticated than oldValue == newValue.

Building the “Actually Smart” Comparison Engine

Alright, time to get serious. I needed a comparison function that could handle all of SharePoint’s quirky field types and still be fast enough to not slow down my API.

Here’s what I ended up building (and yes, it’s a bit of a beast):

internal static List<(string InternalName, object? NewValue)> GetDifferences(
    ListItem item,
    IDictionary<string, object?> newValues,
    bool treatEmptyStringAsNull)
{
    var diffs = new List<(string, object?)>();

    foreach (var kvp in newValues)
    {
        var name = kvp.Key;
        var newVal = kvp.Value;

        object? currentVal = null;
        if (item.FieldValues != null && item.FieldValues.TryGetValue(name, out var cv))
        {
            currentVal = cv;
        }

        if (!ValuesEqual(currentVal, newVal, treatEmptyStringAsNull))
        {
            diffs.Add((name, newVal));
        }
    }

    return diffs;
}

private static bool ValuesEqual(object? a, object? b, bool treatEmptyStringAsNull)
{
    if (ReferenceEquals(a, b))
    {
        return true;
    }

    if (a is null || b is null)
    {
        if (treatEmptyStringAsNull)
        {
            if ((a is null && IsEmptyStringLike(b)) || (b is null && IsEmptyStringLike(a)))
            {
                return true;
            }
        }
        return false;
    }

    a = Normalize(a, treatEmptyStringAsNull);
    b = Normalize(b, treatEmptyStringAsNull);

    if (a is IStructuralEquatable seA && b is IStructuralEquatable seB)
    {
        return StructuralComparisons.StructuralEqualityComparer.Equals(seA, seB);
    }

    return Equals(a, b);
}

private static bool IsEmptyStringLike(object? x)
{
    return x is string s && string.IsNullOrWhiteSpace(s);
}

private static object? Normalize(object? v, bool treatEmptyStringAsNull)
{
    if (v == null)
    {
        return null;
    }

    switch (v)
    {
        case string s:
            return (treatEmptyStringAsNull && string.IsNullOrWhiteSpace(s)) ? null : s;

        case bool b:
            return b;

        case byte or sbyte or short or ushort or int or uint or long or ulong or float or double or decimal:
            return Convert.ToDecimal(v, CultureInfo.InvariantCulture);

        case DateTime dt:
            var utc = (dt.Kind == DateTimeKind.Utc ? dt : DateTime.SpecifyKind(dt, DateTimeKind.Unspecified)).ToUniversalTime();
            return new DateTime(utc.Year, utc.Month, utc.Day, utc.Hour, utc.Minute, utc.Second, DateTimeKind.Utc);

        case FieldUserValue u:
            return u.LookupId;

        case FieldLookupValue l:
            return l.LookupId;

        case IEnumerable<FieldLookupValue> multiLookup:
            return multiLookup.Select(x => x?.LookupId ?? 0).OrderBy(x => x).ToArray();

        case TaxonomyFieldValue tx:
            return tx.TermGuid?.Trim().ToLowerInvariant();

        case TaxonomyFieldValueCollection txc:
            return txc
                .Where(x => x != null && !string.IsNullOrEmpty(x.TermGuid))
                .Select(x => x.TermGuid.Trim().ToLowerInvariant())
                .OrderBy(g => g)
                .ToArray();

        case FieldGeolocationValue geo:
            return new ValueTuple<double, double, double, double>(
                Math.Round(geo.Latitude, 6),
                Math.Round(geo.Longitude, 6),
                Math.Round(geo.Altitude, 2),
                Math.Round(geo.Measure, 2));

        case string[] ss:
            return ss
                .Select(x => treatEmptyStringAsNull && string.IsNullOrWhiteSpace(x) ? null : x)
                .OrderBy(x => x, StringComparer.Ordinal)
                .ToArray();

        case IEnumerable enumerable when v is not string:
        {
            var list = new List<object?>();
            foreach (var e in enumerable)
            {
                list.Add(Normalize(e, treatEmptyStringAsNull));
            }
            return list.ToArray();
        }

        default:
            return v;
    }
}

The Magic Behind the Scenes

1. The Normalization Dance

The Normalize method is where the real magic happens. It takes SharePoint’s various field types and converts them into something we can actually compare:

• Numbers: Everything becomes a decimal because SharePoint loves to mix integers and strings
• DateTime: Converted to UTC and truncated to seconds (because who needs millisecond precision for list items?)
• User/Lookup Fields: We only care about the LookupId, not the display text that might change
• Collections: Sorted alphabetically because SharePoint doesn’t guarantee order
• Strings: Can optionally treat empty/whitespace as null (trust me, you want this)

2. The “It’s Not You, It’s SharePoint” Handling

For arrays and complex objects, I use IStructuralEquatable to do deep comparisons. Because yes, SharePoint will absolutely give you arrays that contain the same items but in different orders.

3. The Business Logic Escape Hatch

The treatEmptyStringAsNull parameter saved my sanity. Different parts of your application might handle empty values differently, and this lets you define your own rules for what counts as “unchanged.”

But Wait, Why Not Just Use JSON Comparison?

Before you ask (because I know you’re thinking it): “Why build this elaborate comparison engine when you could just serialize both objects to JSON and compare the strings?”

Great question. I actually thought the same thing initially. How hard could it be, right? Just JsonSerializer.Serialize() both the old and new values, compare the resulting strings, and call it a day.

Here’s what the “simple” JSON approach looks like:

internal static List<(string InternalName, object? NewValue)> GetDifferencesJson(
    IDictionary<string, object?> oldValues,
    IDictionary<string, object?> newValues,
    bool treatEmptyStringAsNull)
{
    var diffs = new List<(string, object?)>();

    foreach (var kvp in newValues)
    {
        var name = kvp.Key;
        var newVal = kvp.Value;

        object? currentVal = null;
        oldValues?.TryGetValue(name, out currentVal);

        var newJson = JsonSerializer.Serialize(newVal);
        var oldJson = JsonSerializer.Serialize(currentVal);

        if (newJson != oldJson)
        {
            diffs.Add((name, newVal));
        }
    }

    return diffs;
}

Looks clean and simple, right? That’s what I thought too. So I built both approaches and put them head-to-head with BenchmarkDotNet. The results were… eye-opening.

The Performance Reality Check

Here’s what I found when comparing the performance of my custom approach versus JSON string comparison:

Single List Item Comparison:

• Custom Dictionary Approach: 749,625 operations/second (1.334 μs per operation)
• JSON String Approach: 201,109 operations/second (4.972 μs per operation)
• Performance difference: 3.7x faster with the custom approach

Batch Operations (300 items):

• Custom Dictionary Approach: 3,705 operations/second (269.91 μs per batch)
• JSON String Approach: 712 operations/second (1,404.06 μs per batch)
• Performance difference: 5.2x faster with the custom approach

Why JSON Comparison Falls Short

The numbers tell the story, but here’s what’s actually happening under the hood:

1. Serialization Overhead: Converting SharePoint field values to JSON strings requires multiple allocations and string operations
2. Memory Pressure: JSON approach allocated nearly 50% more memory (4.67KB vs 3.17KB per operation)
3. Type Conversion Issues: JSON serialization doesn’t handle SharePoint’s quirky field types the way we need
4. Garbage Collection: More allocations = more GC pressure = slower overall performance

When you’re processing hundreds or thousands of list items in batch operations, that 3-5x performance difference really adds up. Plus, the custom approach gives us complete control over how different field types are compared, which JSON comparison simply can’t provide.

Putting It All Together: The Real-World Implementation

Here’s how I actually use this in my APIs:

public async Task<bool> UpdateListItemAsync(int itemId, Dictionary<string, object?> newValues)
{
    var clientContext = GetClientContext();
    var list = clientContext.Web.Lists.GetByTitle("YourList");
    var listItem = list.GetItemById(itemId);
    
    clientContext.Load(listItem);
    clientContext.ExecuteQuery();

    // The moment of truth - what actually changed?
    var differences = GetDifferences(listItem, newValues, treatEmptyStringAsNull: true);
    
    if (!differences.Any())
    {
        // Nothing changed! Skip the update entirely
        return false; // "Thanks for playing, but you didn't actually change anything"
    }

    // Only update the fields that actually changed
    foreach (var (fieldName, newValue) in differences)
    {
        listItem[fieldName] = newValue;
    }

    listItem.Update();
    clientContext.ExecuteQuery();
    
    return true; // "Something actually happened!"
}

The Numbers Don’t Lie (And They’re Pretty Great)

After implementing this change detection across our applications, here’s what we typically see:

Before implementing change detection:

• Applications making thousands of update requests per day
• Every single one triggered a SharePoint ExecuteQuery()
• High API call volumes and occasional throttling issues

After implementing change detection:

• Same number of update requests from users/systems
• 60-80% contained no actual changes and were skipped
• Dramatically reduced SharePoint API consumption
• Much lower throttling risk

The typical impact:

• Significant reduction in monthly API calls
• Cost savings with Service Prioritization in SharePoint (varies by usage)
• Reduced throttling risk during busy periods
• Faster API responses because skipped operations are basically instant

But here’s what really matters: the performance improvement isn’t just about the numbers. Users notice when applications feel more responsive, and developers notice when they stop getting throttling alerts.

The Gotchas (Because There Are Always Gotchas)

When This Approach Might Not Be For You

1. Audit Requirements: If you need to log every single “save” action regardless of whether anything changed, this might not work for your use case.

2. Always Update Timestamps: Some business requirements dictate that “LastModified” should always be updated when a user clicks save, even if the content is identical.

3. Super Simple Scenarios: If you’re only updating one field and it’s a simple string comparison, the overhead of this elaborate comparison might not be worth it.

The SharePoint Field Types That Made Me Question My Life Choices

Some field types need special handling:

// Calculated fields - don't even try to update these
if (field.ReadOnlyField || field.FieldTypeKind == FieldType.Calculated)
{
    continue; // SharePoint will just ignore you anyway
}

// Attachment fields - these need completely different logic
if (field.FieldTypeKind == FieldType.Attachments)
{
    // Handle separately - attachments are their own special nightmare
    continue;
}

Wrapping Up (And Why This Matters More Than You Think)

Look, I know this might seem like over-engineering for a simple “update SharePoint item” operation. But here’s the thing: those small inefficiencies add up fast.

In my case, this one change:

• Improved user experience with faster response times
• Reduced throttling headaches during busy periods
• Made me feel like a responsible developer (this one’s important too!)

The pattern I’ve shown you isn’t just about SharePoint or just about API optimization. It’s about asking the fundamental question: “Is this operation actually necessary?”

That question has made me a better developer across all the platforms I work with, not just SharePoint.

So next time you’re building any kind of update operation—whether it’s SharePoint, a database, or that JSON file you’re definitely not using as a database—pause for a moment and ask: “Did anything actually change?”

Your future self (and your API bills) will thank you.

Pro tip: Start implementing this pattern on your highest-traffic endpoints first. That’s where you’ll see the biggest impact, and it’ll give you the confidence to roll it out everywhere else.

Now go forth and eliminate those unnecessary API calls. Your SharePoint environment will purr like a happy cat.

Change detection is one of five techniques in The SharePoint CSOM Performance Playbook. If you want the whole picture of where CSOM calls leak, start there.

Recently, I was working with a client to calculate the cost of enabling Service Prioritization in SharePoint on one of their app registrations. The pricing model is straightforward: USD $0.50 per 1,000 Graph calls and USD $1.00 per 1,000 SP REST/CSOM calls.

During the analysis, I discovered something that caught my attention: their solution was making approximately 90,000 calls per month, and 88% of all their CSOM calls were EnsureUser operations.

This was particularly interesting because the users being “ensured” were already associated with the site where the data was being created. It seemed like there should be a more efficient approach - similar to what’s possible with Graph API.

The Problem with User Field Assignment

If you’ve worked with SharePoint user fields, you’re familiar with this common pattern:

// The standard approach
var user = clientContext.Web.EnsureUser("user@company.com");
clientContext.Load(user);
clientContext.ExecuteQuery(); // Network call #1

listItem["AssignedTo"] = new FieldUserValue()
{
    LookupId = user.Id
};
listItem.Update();
clientContext.ExecuteQuery(); // Network call #2

This requires two network calls for each user assignment. Microsoft provides FieldUserValue.FromUser("user@company.com") as an alternative, but it throws an exception if the user isn’t already “known” to the site. This leads to either making two calls to be safe, or handling exceptions with additional calls.

Discovering ExceptionHandlingScope

While researching optimization approaches, I came across a section in Microsoft’s documentation for “Complete basic operations using SharePoint client library code” about ExceptionHandlingScope.

ExceptionHandlingScope allows you to implement try-catch logic that executes on the SharePoint server rather than requiring multiple round trips from your client application. Instead of this painful dance:

1. Try operation → Network call
2. Handle exception → Network call
3. Retry operation → Network call

You get this beautiful symphony:

1. Send try-catch logic to server → Network call
2. Server handles everything internally
3. Get result → You’re done

Here’s the magic in action:

var scope = new ExceptionHandlingScope(clientContext);

using (scope.StartScope())
{
    using (scope.StartTry())
    {
        // Try the optimistic approach first
        listItem["CaseResponsible"] = FieldUserValue.FromUser("user@company.com");
        listItem.Update();
    }
    
    using (scope.StartCatch())
    {
        // If that fails, ensure the user exists
        var user = clientContext.Web.EnsureUser("user@company.com");
        listItem["CaseResponsible"] = FieldUserValue.FromUser("user@company.com");
    }
}

// Execute once - the server handles all the logic
clientContext.ExecuteQuery();

One network call. That’s it.

The server receives your entire try-catch block, attempts the optimistic operation first, and only falls back to EnsureUser if needed. No round trips. No guessing. No waste.

Real-World Impact: Significant Call Reduction

Looking back at the client scenario with ExceptionHandlingScope:

• Before: ~79,200 EnsureUser calls + ~10,800 actual operations = 90,000 total calls
• After: ~10,800-21,600 operations (depending on how many users need ensuring) = substantial reduction

This represents a potential 75-85% reduction in API calls, with corresponding improvements in both performance and Service Prioritization in SharePoint costs.

The Cost Breakdown

Let’s translate this into actual Service Prioritization in SharePoint costs:

Before optimization:

• 90,000 CSOM calls per month
• At USD $1.00 per 1,000 calls
• Monthly cost: $90

After ExceptionHandlingScope optimization:

• Best case: 10,800 calls (if most users are already known) = $10.80/month
• Worst case: 21,600 calls (if many users need ensuring) = $21.60/month
• Monthly savings: $68.40 - $79.20

Annual savings: $820 - $950

These savings become even more significant at scale. For organizations with multiple solutions or higher call volumes, the cost difference can easily reach thousands of dollars annually.

More importantly, the performance improvements—reduced network latency, faster operations, and lower throttling risk—often provide value that far exceeds the direct cost savings.

Throttling: The Hidden Performance Killer

Beyond cost savings, ExceptionHandlingScope provides significant throttling benefits that are often more impactful than the financial savings.

Understanding SharePoint Throttling

SharePoint applies throttling limits to prevent abuse and ensure service stability. When you exceed these limits, you’ll encounter:

• HTTP 429 (Too Many Requests) responses
• Exponential backoff delays (sometimes minutes)
• User experience degradation as operations slow down
• Potential service interruptions during peak usage

The Throttling Math

In our client scenario:

Before ExceptionHandlingScope:

• 90,000 calls per month = ~3,000 calls per day
• During peak hours, this could easily trigger throttling
• Each throttled request requires retry with exponential backoff
• A single throttled operation can delay your entire batch

After ExceptionHandlingScope:

• 10,800-21,600 calls per month = ~360-720 calls per day
• 10x reduction in throttling risk
• Smoother operation during peak business hours
• More predictable performance for end users

Real-World Throttling Impact

Consider a typical business day where multiple users are creating items simultaneously:

• Without optimization: 88% of your throttling budget consumed by redundant EnsureUser calls
• With ExceptionHandlingScope: 88% of your throttling budget available for actual business operations

The beauty of ExceptionHandlingScope is that it doesn’t just reduce the number of calls—it makes your remaining calls more valuable and less likely to be throttled.

Important: ExceptionHandlingScope Is NOT a Magic Bullet

Critical Disclaimer: ExceptionHandlingScope itself does NOT reduce the number of requests to SharePoint. Each operation within the scope still counts as a separate request for throttling purposes.

The reduction in our scenario comes from changing the approach, not from using ExceptionHandlingScope:

What Actually Reduces Calls

• Before: Always calling EnsureUser + setting the field = 2 calls per user
• After: Using FieldUserValue.FromUser() directly, falling back to EnsureUser only when needed

What ExceptionHandlingScope Actually Does

ExceptionHandlingScope reduces network round trips, not server requests:

• Network benefit: 1 round trip instead of potentially 2-3
• Throttling impact: Each operation still counts toward throttling limits
• Performance gain: Reduced latency, not reduced server load

The Real Magic

The 75-85% reduction in our client scenario comes from this logic change:

// This approach reduces actual requests
// Most users are already known to the site
// So most operations only need 1 request instead of 2
using (scope.StartTry())
{
    // This succeeds for ~80% of users (1 request)
    listItem["Field"] = FieldUserValue.FromUser("user@company.com");
}
using (scope.StartCatch())
{
    // This only runs for ~20% of users (1 request)
    var user = clientContext.Web.EnsureUser("user@company.com");
    listItem["Field"] = FieldUserValue.FromUser("user@company.com");
}

ExceptionHandlingScope simply makes this pattern efficient by handling the try-catch logic server-side instead of requiring multiple network round trips to determine which approach to use.

Bottom line: ExceptionHandlingScope optimizes network efficiency, but the request reduction comes from smarter business logic, not from the scope itself.

The Broader Lesson

This experience highlighted an important principle: before optimizing an existing approach, it’s worth questioning whether there’s a fundamentally different way to solve the problem. In this case, server-side exception handling provided a cleaner solution than client-side optimization or caching strategies.

ExceptionHandlingScope has been available since SharePoint 2013, but it’s not widely discussed in the SharePoint development community. It’s a good reminder to periodically review Microsoft’s documentation for features that might address current challenges in new ways.

Your Next Steps

Before you write your next SharePoint operation that involves potential exceptions:

1. Pause and calculate: How many network calls will your approach generate?
2. Question the pattern: Could server-side logic handle the complexity?
3. Explore ExceptionHandlingScope: Can your try-catch logic run on the server?
4. Measure the impact: Compare network calls before and after implementation

Sometimes the most powerful optimizations come from using the tools that were there all along. ExceptionHandlingScope isn’t just a performance optimization—it’s a reminder that the platform often provides solutions we didn’t know we were looking for.

Next time you’re facing a SharePoint performance challenge, remember: the answer might not be in the latest framework or cutting-edge technique. Sometimes, it’s hiding in plain sight in the documentation you scrolled past.

ExceptionHandlingScope is one of five techniques in The SharePoint CSOM Performance Playbook, alongside batching, CAML joins, change detection and fast taxonomy loading.

This is exactly the problem I ran into while building a system where SharePoint list changes needed to trigger updates to site titles. Users would edit customer information, and I wanted the customer’s SharePoint site title to reflect those changes. But I definitely didn’t want to rename a site five times in thirty seconds just because someone couldn’t decide between “Acme Corp” and “ACME Corporation.”

The solution? Debouncing. Just like the useDebounce hook that React developers know and love, but for SharePoint webhooks.

The Business Problem

Let me paint you the picture. We have a UI that updates SharePoint list items every time a user makes a change. When they modify a customer’s name, we want to update the title of that customer’s SharePoint site. Simple enough, right?

Wrong.

Users don’t make one clean edit and walk away. They edit the name, realize they want to change the format, edit it again, spot a typo, fix that too, and maybe adjust the capitalization for good measure. Each of these changes triggers our SharePoint webhook, which in turn tries to update the site title.

Not only is this inefficient, but it’s also potentially problematic. Site title updates aren’t instant operations, and we definitely don’t want them queuing up and executing out of order.

What we really want is to wait until the user is done making changes, then process all their edits as one logical operation.

Enter the Debounce Pattern

The concept is borrowed straight from the front-end world. Instead of reacting to every single change immediately, we wait. If another change comes in within our debounce window, we cancel the previous operation and reset the timer.

Here’s the entry point to my implementation:

public class WebhookProcessor() {
    public async Task HandleListChangeAsync(string itemId, string listId) {
        using (DebounceScheduler debouncer = new DebounceScheduler()) {
            var message = new ServiceBusMessage() {
                ContentType = "application/json",
            };

            await debouncer.DebounceAsync(itemId, listId, message, TimeSpan.FromMinutes(5));
        }
    }
}

Simple on the surface, but there’s quite a bit happening under the hood.

The Architecture: Azure Service Bus + Table Storage

I built this debounce system using two Azure services:

1. Azure Service Bus for scheduled message delivery
2. Azure Table Storage for tracking sequence numbers

Here’s why this combination works so well:

Azure Service Bus Scheduled Messages

Service Bus has a fantastic feature called scheduled messages. You can schedule a message to be delivered at a specific time in the future, and critically, you can cancel that scheduled message if needed.

This is perfect for debouncing. When a webhook fires, I schedule a message to be processed in 5 minutes. If another webhook fires for the same item before those 5 minutes are up, I cancel the previously scheduled message and schedule a new one.

Table Storage for State Management

The tricky part is remembering which messages you’ve scheduled so you can cancel them later. That’s where Table Storage comes in.

The core logic looks something like this:

internal async Task DebounceAsync(string entityId, string listId, ServiceBusMessage message, TimeSpan delay) {
    // Get any existing scheduled message for this entity
    var oldSeq = await store.GetSequenceNumberAsync(entityId);
   
    if (oldSeq.HasValue) {
        // Cancel the previous scheduled message
        await _sender.CancelScheduledMessageAsync(oldSeq.Value);
        await store.DeleteSequenceNumberAsync(entityId);
    }

    // Schedule a new message
    var scheduledTime = DateTimeOffset.UtcNow.Add(delay);
    long newSeq = await _sender.ScheduleMessageAsync(message, scheduledTime);

    // Store the new sequence number for future cancellation
    await store.SetSequenceNumberAsync(entityId, newSeq, listId);
}

For each item being debounced, I store:

• The sequence number of the scheduled Service Bus message
• The list ID

When a new change comes in, I can look up the previous sequence number, cancel that message, and schedule a new one.

The Flow in Action

Let’s walk through what happens when a user edits a customer:

1. User changes customer name → SharePoint webhook fires
2. My webhook Azure Function receives the notification and queues it
3. The webhook queue processor calls WebhookProcessor.HandleListChangeAsync()
4. DebounceScheduler checks Table Storage for any existing scheduled message
5. If one exists, it gets cancelled via the Service Bus sequence number
6. A new message is scheduled for 5 minutes from now
7. The new sequence number is saved to Table Storage

If the user makes another change within 5 minutes, steps 4-7 repeat, effectively resetting the countdown.

Once 5 minutes pass without any changes, the scheduled message is delivered to our debounce queue, and the actual processing begins.

When SharePoint Gets Slow (Really Slow)

Here’s where things got interesting. During testing, I discovered that SharePoint webhooks aren’t always as snappy as you’d expect.

One evening, I made a change to a list item and noticed that both my webhook and a Power Automate flow (each set up for the same list) didn’t fire until 22 minutes later. The timing was unusual, especially since this happened right when the customer’s backup solution started running.

I can’t say for sure what caused the delay, and I don’t want to speculate or draw any conclusions based on this single incident. However, it does highlight an important consideration: webhook delivery isn’t guaranteed to be immediate.

Trade-offs and Considerations

Like any architectural decision, this approach comes with trade-offs:

Pros:

• Dramatically reduces unnecessary processing
• Handles the “fidgety user” problem elegantly
• Resilient to webhook delivery delays
• Scales well (Service Bus handles the heavy lifting)
• Works across multiple function instances

Cons:

• Adds latency (minimum 5-minute delay)
• Requires additional Azure services (cost)
• More complex than direct processing
• Potential for message loss (though Service Bus is pretty reliable)

Alternative Approaches I Considered:

1. In-memory debouncing: Would work for a single instance, but doesn’t scale across multiple Azure Function instances
2. Redis-based debouncing: Would work, but adds another dependency and requires more custom logic
3. Database polling: Could work, but feels clunky and less efficient than Service Bus scheduled messages
4. Simple delays: Just wait X seconds before processing, but this doesn’t handle multiple rapid changes

The Service Bus + Table Storage approach won because it’s distributed by nature, leverages existing Azure services we were already using, and handles cancellation elegantly.

Wrapping Up

Debouncing SharePoint webhooks with Azure Service Bus and Table Storage has made our operations more efficient and resilient to user behavior and platform quirks.

Picture this: It’s 2 AM, you’re on your third cup of coffee, and you’re watching your perfectly crafted Microsoft Graph batch request fail spectacularly. Again.

You’ve got 25 files to download from SharePoint. Your batch processes 24 of them perfectly, then one lonely file decides to throw a throttling tantrum. What does your retry logic do? It throws away all 24 successful downloads and starts over. From scratch. Like a digital Groundhog Day, but less amusing and more soul-crushing.

Sound familiar? Welcome to the “retry everything” club – where perfectly good API calls go to die unnecessarily. 😅

The Grocery Cart Problem (Or: Why We’re Doing This Wrong)

Let me paint you a picture. You’re at the grocery store with a cart full of 20 items. You get to checkout, and the cashier says, “Sorry, we’re out of milk.”

What would you do?

• Option A: Put back everything, go home, and come back later to shop for all 20 items again
• Option B: Buy the 19 items you can get, then come back just for the milk

If you picked Option A, congratulations – you think like most API retry logic! If you picked Option B, you’re ready to learn about smart retries.

The “retry everything” approach is like Option A, and here’s why it’s bonkers:

• 🔄 Wasted effort: You’re re-requesting stuff that already worked perfectly
• 🐌 Slower performance: Users wait longer while you redo successful work
• 📈 Throttling amplification: You’re actually making the problem worse by hitting successful endpoints again
• 🔍 Poor debugging: Can’t easily identify which specific requests are the real troublemakers

I learned this the hard way when I watched a simple file sync turn into an API call avalanche. 20 requests became 40, then 80, then… well, let’s just say Microsoft’s throttling system got very acquainted with my application.

The “Aha!” Moment (It’s Simpler Than You Think)

The solution hit me during one of those 2 AM debugging sessions: What if we only retry the stuff that actually failed?

Revolutionary, right? 😏

Here’s the beautiful thing – this isn’t some PhD-level computer science. It’s just common sense applied to code. Keep the winners, retry the losers. Simple.

But (there’s always a “but”), there’s one sneaky technical challenge that makes this trickier than it sounds. Microsoft’s Graph SDK has a helpful method called NewBatchWithFailedRequests(), but it has a quirk: it generates brand new request IDs. This breaks your ability to map responses back to your original data.

Think of it like this: You order pizza for table 5, but when they bring the replacement slice, they call it table 23. Good luck figuring out who ordered what!

If you’re new to Graph batching or request mapping, I’d recommend checking out my post on Graph Batching for File Content: Mapping Requests to Responses first. It’s like the prequel to this story – explains how to keep track of what’s what when dealing with batch responses.

Quick Win Summary (For the Impatient Developers)

The Problem: Your retry logic is like that friend who starts the entire conversation over when they missed one word. Inefficient and annoying.

The Solution: A drop-in extension method that only retries the actual failures while keeping successful responses safe and sound.

The Payoff:

• ⚡ Faster operations (no more re-downloading working files)
• 📉 Fewer API calls (your rate limits will thank you)
• 🎯 Less throttling (stop beating dead endpoints)
• 😌 Happier users (and happier you at 2 AM)

The Catch: You need to understand request-to-response mapping. Don’t worry, it’s not rocket science, and I’ve got a whole post about it.

Time Investment: About 5 minutes to implement, countless hours of frustration saved.

Ready for the nitty-gritty? Let’s dive in! 👇

The Hero of Our Story: The Smart Retry Extension

Okay, here’s where we get our hands dirty. The main challenge isn’t just filtering out successful requests – it’s that pesky NewBatchWithFailedRequests method that scrambles your request IDs like eggs at Sunday brunch.

Here’s the extension method that saves the day:

internal static class GraphServiceClientExtensions 
{
    internal static async Task<(IReadOnlyDictionary<string, HttpStatusCode> Statuses, Dictionary<string, HttpResponseMessage> BatchResponse)> 
        PostBatchWithFailedDependencyRetriesAsync(this GraphServiceClient graphClient, BatchRequestContentCollection originalBatch) 
    {
        const int maxRetries = 5;
        TimeSpan delay = TimeSpan.FromSeconds(1);

        Dictionary<string, HttpResponseMessage> allResponses = new Dictionary<string, HttpResponseMessage>();
        Dictionary<string, HttpStatusCode> allStatuses = new Dictionary<string, HttpStatusCode>();

        BatchRequestContentCollection batchToSend = originalBatch;

        for (int attempt = 1; attempt <= maxRetries; attempt++) 
        {
            BatchResponseContentCollection batchResponse = await graphClient.Batch.PostAsync(batchToSend);
            Dictionary<string, HttpStatusCode> responses = await batchResponse.GetResponsesStatusCodesAsync();

            // Filter out failures (excluding redirects which are normal for file content)
            Dictionary<string, HttpStatusCode> failedRequests = responses
                .Where(kvp => !BatchResponseContent.IsSuccessStatusCode(kvp.Value) && kvp.Value != HttpStatusCode.Found)
                .ToDictionary(kvp => kvp.Key, kvp => kvp.Value);

            // Collect all responses from this attempt
            foreach (var kvp in responses) 
            {
                var response = await batchResponse.GetResponseByIdAsync(kvp.Key);
                allResponses[kvp.Key] = response;
                allStatuses[kvp.Key] = kvp.Value;
            }

            if (failedRequests.Count == 0 || attempt == maxRetries) 
            {
                break;
            }

            await Task.Delay(delay);
            delay = TimeSpan.FromSeconds(delay.TotalSeconds * 2);

            // The key problem: NewBatchWithFailedRequests creates new request IDs!
            batchToSend = batchToSend.NewBatchWithFailedRequests(responses);
            
            // This is why we need this method - restore the original request IDs
            await RestoreOriginalRequestIdsAsync(batchToSend, originalBatch);
        }

        return (allStatuses, allResponses);
    }

    private static async Task RestoreOriginalRequestIdsAsync(
        BatchRequestContentCollection newBatch, 
        BatchRequestContentCollection originalBatch) 
    {
        var stepsSnapshot = newBatch.BatchRequestSteps.ToArray();

        foreach (var kvp in stepsSnapshot) 
        {
            var oldStepId = kvp.Key;
            var step = kvp.Value;
            var requestPath = step.Request.RequestUri!.AbsolutePath;

            // Find the original request ID by matching the request path
            var matchingOriginal = originalBatch.BatchRequestSteps
                .First(x => x.Value.Request.RequestUri!.AbsolutePath == requestPath);

            var originalStepId = matchingOriginal.Key;

            if (oldStepId != originalStepId) 
            {
                newBatch.RemoveBatchRequestStepWithId(oldStepId);

                var newStep = new BatchRequestStep(
                    requestId: originalStepId,
                    httpRequestMessage: step.Request,
                    dependsOn: step.DependsOn);

                newBatch.AddBatchRequestStep(newStep);
            }
        }
    }
}

What’s happening here? Think of it as a diplomatic negotiator for your API calls:

1. The ID Shuffle Problem: NewBatchWithFailedRequests gives failed requests shiny new IDs, like witness protection for HTTP requests
2. The Detective Work: RestoreOriginalRequestIdsAsync plays detective, matching requests by their paths to find their original identities
3. The Happy Reunion: Failed requests get their original IDs back, so your mapping dictionary doesn’t break down in tears

It’s like having a really good wedding planner who makes sure everyone sits at the right table, even after the venue changes.

Showtime: Watching Smart Retries in Action

Now let’s see our smart retry logic work its magic in a real-world scenario. Imagine you’re building a document sync tool and need to download 25 files from SharePoint. Some will work perfectly, others might throw tantrums due to throttling or network hiccups.

Here’s how the new approach handles it like a champ:

// Let's say you need to download content from 25 SharePoint files
var batch = new BatchRequestContentCollection(graphClient);
var fileMapping = new Dictionary<string, FileContentRequest>();

// Build the batch for file content downloads
var filesToDownload = await GetFilesToProcess(); // Your method to get file list

foreach (var file in filesToDownload) 
{
    var requestInfo = graphClient.Sites[siteId]
                                .Drives[driveId]
                                .Items[file.DriveItemId]
                                .Content
                                .ToGetRequestInformation();
    
    var requestId = await batch.AddBatchRequestStepAsync(requestInfo);
    
    // Map the request ID to your file info (same pattern as previous post)
    fileMapping[requestId] = new FileContentRequest 
    { 
        DriveItemId = file.DriveItemId,
        FileName = file.Name,
        ExpectedSize = file.Size,
        DownloadStartTime = DateTime.Now
    };
}

// 🎯 Here's where the magic happens - just one line change!
var (statuses, responses) = await graphClient.PostBatchWithFailedDependencyRetriesAsync(batch);

// Process results - this is where the retry really shines
var successfulDownloads = new List<FileDownloadResult>();
var failedDownloads = new List<string>();

foreach (var kvp in fileMapping) 
{
    var requestId = kvp.Key;
    var fileRequest = kvp.Value;
    
    if (responses.TryGetValue(requestId, out var response)) 
    {
        var statusCode = statuses[requestId];
        
        if (BatchResponseContent.IsSuccessStatusCode(statusCode)) 
        {
            // Success! Handle the file content
            var contentBytes = await response.Content.ReadAsByteArrayAsync();
            
            // Save to your desired location
            var localPath = Path.Combine(downloadFolder, fileRequest.FileName);
            await File.WriteAllBytesAsync(localPath, contentBytes);
            
            successfulDownloads.Add(new FileDownloadResult 
            { 
                FileName = fileRequest.FileName,
                LocalPath = localPath,
                ActualSize = contentBytes.Length,
                ExpectedSize = fileRequest.ExpectedSize,
                DownloadTime = DateTime.Now - fileRequest.DownloadStartTime
            });
        } 
        else 
        {
            // Even after smart retries, this file failed
            failedDownloads.Add($"{fileRequest.FileName} ({statusCode})");
            
            // Log the specific failure for debugging
            Console.WriteLine($"Failed to download {fileRequest.FileName}: {statusCode}");
        }
        
        // Always clean up the response
        response.Dispose();
    }
}

Console.WriteLine($"Successfully downloaded: {successfulDownloads.Count} files");
if (failedDownloads.Any())
{
    Console.WriteLine($"Failed downloads: {string.Join(", ", failedDownloads)}");
}

The beautiful part? Look at that line with the magic emoji 🎯. That’s literally the only change you need to make to your existing batch processing code. Everything else stays exactly the same.

Here’s what’s happening behind the scenes:

1. First batch attempt: Say 20 files succeed, 5 fail due to throttling
2. Smart filtering: Keep those 20 successful responses safe
3. Targeted retry: Build a new batch with just the 5 failures
4. ID preservation: Make sure those 5 retries still map to your original file info
5. Rinse and repeat: Maybe 4 of the 5 succeed on retry, leaving just 1 persistent troublemaker

The result? Instead of making 250 API calls (25 files × 5 retry attempts for the unlucky ones), you might only make 35 total calls. Your throttling problems become manageable, and files download way faster.

// Supporting classes for the example above
public class FileContentRequest 
{
    public string DriveItemId { get; set; } = string.Empty;
    public string FileName { get; set; } = string.Empty;
    public long ExpectedSize { get; set; }
    public DateTime DownloadStartTime { get; set; }
}

public class FileDownloadResult 
{
    public string FileName { get; set; } = string.Empty;
    public string LocalPath { get; set; } = string.Empty;
    public long ActualSize { get; set; }
    public long ExpectedSize { get; set; }
    public TimeSpan DownloadTime { get; set; }
}

The Method to the Madness (What’s Really Happening)

I know that extension method looks intimidating – like trying to read assembly instructions in a foreign language. But once you break it down, it’s actually pretty logical. Let me walk you through the step-by-step dance:

Step 1: The First Attempt “Let’s try everything once and see what happens”

Sends your original batch of 25 files and carefully captures every single response and status code. No throwing anything away yet.

Step 2: The Great Sorting “Okay, who succeeded and who’s being difficult?”

Separates the winners from the losers, but (and this is important) ignores redirect responses. Why? Because when you’re downloading large files, redirects are totally normal – SharePoint often redirects you to the actual storage location.

Step 3: The Preservation Society “Keep the good stuff safe while we deal with the troublemakers”

All successful responses get stored in a safe place while we build a new, smaller batch containing only the failed requests. It’s like having a really good filing system for your API responses.

Step 4: The Identity Crisis Resolution “Wait, who are you again? Let me check your original ID…”

This is the tricky bit! The NewBatchWithFailedRequests method gives everyone new IDs, like a witness protection program for HTTP requests. Our RestoreOriginalRequestIdsAsync method plays detective, matching requests by their URL paths to restore their original identities.

Step 5: The Polite Wait “Let’s not be pushy – maybe try again in a second?”

Implements exponential backoff – starts with a 1-second wait, then 2 seconds, then 4 seconds, etc. This prevents your app from being that annoying person who keeps knocking on the door every second.

Step 6: The Safety Net “Okay, we tried 5 times. Some files just aren’t meant to be downloaded today.”

Gives up after 5 attempts to prevent infinite retry loops. Because sometimes you need to know when to walk away from the poker table.

Why This Actually Works (The Science Behind the Magic)

Here’s what makes this approach so much better than the “retry everything” strategy:

🎯 Surgical Precision Only retry what actually failed – it’s like having a really good therapist who focuses on the actual problems instead of rehashing everything from childhood.

No wasted API calls on requests that already succeeded. If 24 out of 25 files downloaded perfectly, why punish them with another round trip?

⚡ Speed Demon Successful requests don’t get repeated, so everything finishes faster – sometimes dramatically faster.

Users see their successful downloads immediately while you quietly retry the problematic ones in the background.

🤝 Throttling-Friendly Fewer total requests means you’re less likely to hit Microsoft’s rate limits, and when you do, recovery is faster.

Instead of amplifying throttling issues, you’re actually helping to resolve them by reducing load on the endpoints that are already struggling.

🔄 Drop-in Simplicity Change literally one line of code and you’re done. No architectural rewrites, no complex state management – just swap out the method call.

Your existing error handling, logging, and business logic all stay exactly the same.

🔍 Debug Paradise Easy to see exactly which requests are consistently failing, making troubleshooting a breeze instead of a nightmare.

When file “ImportantDocument.pdf” fails on every retry attempt, you know there’s something specific about that file, not your entire batch logic.

The “Before and After” Moment

Let me paint you a picture of how this changes your life:

Before Smart Retries:

• 25 file batch fails on 3 files due to throttling
• Retry all 25 files → now 5 files fail due to increased throttling
• Retry all 25 files again → now 8 files fail
• You’re now in the throttling spiral of doom
• Users are staring at loading spinners
• You’re questioning your career choices

After Smart Retries:

• 25 file batch fails on 3 files due to throttling
• Keep the 22 successful files, retry only the 3 failures
• Maybe 2 of the 3 succeed on retry, leaving 1 stubborn file
• Final retry gets the last file, or you log it as a persistent issue
• Users get 24/25 files quickly, you sleep better at night

It’s the difference between being stuck in traffic because one lane is blocked (and everyone keeps switching to that lane), versus just using the open lanes and going around the problem.

The Bottom Line (And Why Your Future Self Will Thank You)

I’ll be real with you – when I first started working with Microsoft Graph batching, I thought the built-in retry policies were enough. “How hard could it be?” I thought. “APIs fail sometimes, just retry them!”

Then I built my first real-world document sync application. Suddenly, I was dealing with users uploading hundreds of files, enterprise throttling limits, and the occasional network hiccup that would bring the whole operation to a screeching halt.

That’s when I learned the hard way that “retry everything” is like using a sledgehammer to hang a picture frame. Sure, it might work, but you’re probably going to break some stuff in the process.

This selective retry approach has been a game-changer. Not just for performance (though users definitely notice when their bulk operations actually complete), but for debugging too. When you can see that ImportantReport_v23_FINAL_REALLY_FINAL.docx is the file that keeps failing, you can actually do something about it.

The best part? Once you have this extension method in your toolkit, it becomes muscle memory. You’re not adding complexity to your day-to-day development – you’re just swapping out one method call for a smarter one. It’s like upgrading from a flip phone to a smartphone – you wonder how you ever lived without it.

Pro tip: After implementing this, keep an eye on your application logs. You’ll start to notice patterns in failures that you never saw before. Maybe certain file types are more prone to issues, or maybe there’s a specific time of day when throttling gets worse. This kind of insight is pure gold for optimization.

The moral of the story? Sometimes the biggest performance improvements come not from doing things faster, but from doing fewer unnecessary things. And sometimes, the best debugging tool is just… not breaking the working stuff while you fix the broken stuff.

Your 2 AM debugging sessions will never be the same. 😌

Want to Learn More? (The Reading List)

• Microsoft Graph JSON batching - The official documentation (surprisingly readable!)
• Graph Batching for File Content: Mapping Requests to Responses - My previous post that sets up the foundation for this one
• Microsoft Graph throttling guidance - Understanding what makes Microsoft’s APIs cranky
• Graph Explorer - Test your batch requests interactively (great for experimenting)
• Exponential Backoff Pattern - The polite way to retry things

Now go forth and batch smarter, not harder! 🚀

Picture this: you need to download 50 files from SharePoint using Microsoft Graph. Being a good developer, you decide to use batching instead of making 50 individual API calls (because nobody wants to wait that long, and Microsoft’s throttling limits aren’t going anywhere).

You set up your batch request, send it off, and get your responses back. Great! Except… now you’re staring at a bunch of file content with absolutely no way to tell which file is which. 😅

Unlike other Graph operations that return nice JSON objects with IDs and metadata, file content responses are just raw bytes. No file name, no path, no ID - nothing to help you figure out which response belongs to which original request.

I learned this the hard way when I first tried Graph batching for file downloads. Spent way too much time trying to correlate responses by file size or content patterns before realizing there was a much cleaner solution.

The Mapping Solution (It’s Simpler Than You Think)

The trick is surprisingly straightforward: use the batch request ID as your bridge between the original file info and the response content. Every batch request gets a unique ID, and that same ID comes back with the response.

Here’s the game plan:

1. Create a dictionary mapping request IDs to your original file information
2. Build your batch requests and store the mappings
3. Process responses using the request ID to look up the original file info

Let me show you exactly how this works.

Setting Up Your File Information

First, let’s create a simple model to hold our file details:

public class FileInfoDTO {
    public string? Path { get; set; }
    public string? Name { get; set; }
    public string? RelativePath { get; set; }
    public string? UniqueFileName { get; set; }
}

Nothing fancy here - just the basics we need to identify and process each file.

The Complete Batch Download Method

Here’s the full implementation. Don’t worry, I’ll break down the important parts afterward:

private async Task<Dictionary<string, byte[]>> DownloadFilesBatchAsync(
    FileInfoDTO[] fileInfos, 
    string siteId, 
    string driveId) {
    
    using BatchRequestContent batchRequestContent = new BatchRequestContent();
    Dictionary<string, FileInfoDTO> requestMapping = new Dictionary<string, FileInfoDTO>();
    Dictionary<string, byte[]> results = new Dictionary<string, byte[]>();

    // Build batch requests with mapping
    foreach (FileInfoDTO fileInfo in fileInfos) {
        if (string.IsNullOrEmpty(fileInfo.RelativePath) || string.IsNullOrEmpty(fileInfo.UniqueFileName))
            continue;

        string requestId = batchRequestContent.AddBatchRequestStep(
            GraphClient.Sites[siteId]
                      .Drives[driveId]
                      .Root
                      .ItemWithPath(fileInfo.RelativePath)
                      .Content
                      .Request());

        // This is the magic - storing the mapping!
        requestMapping[requestId] = fileInfo;
    }

    // Execute batch request
    BatchResponseContent batchResponse = await GraphClient.Batch.Request().PostAsync(batchRequestContent);
    Dictionary<string, HttpResponseMessage> responses = await batchResponse.GetResponsesAsync();

    // Process responses using our mapping
    foreach ((string requestId, HttpResponseMessage response) in responses) {
        FileInfoDTO originalFile = requestMapping[requestId]; // Look up the original file info
        
        try {
            switch (response.StatusCode) {
                case HttpStatusCode.OK:
                    byte[] content = await response.Content.ReadAsByteArrayAsync();
                    results[originalFile.UniqueFileName!] = content;
                    break;
                    
                case HttpStatusCode.Redirect:
                    // Handle redirect for large files (more on this below)
                    byte[] redirectContent = await DownloadFromRedirectAsync(response.Headers.Location);
                    results[originalFile.UniqueFileName!] = redirectContent;
                    break;
                    
                case HttpStatusCode.TooManyRequests:
                    // Handle throttling
                    throw new Exception($"Throttled request for {originalFile.Name}");
                    
                default:
                    throw new Exception($"Failed to download {originalFile.Name}: {response.ReasonPhrase}");
            }
        }
        finally {
            // Always dispose - learned this one the hard way after some memory leak hunting
            response.Content.Dispose();
        }
    }

    return results;
}

The Key Parts Explained

The Mapping Dictionary

Dictionary<string, FileInfoDTO> requestMapping = new Dictionary<string, FileInfoDTO>();

This is your lifeline. For every request you add to the batch, you store the request ID and link it to your original file information. When responses come back, you can instantly look up which file each response belongs to.

Building Requests with Mapping

string requestId = batchRequestContent.AddBatchRequestStep(...);
requestMapping[requestId] = fileInfo;

The AddBatchRequestStep method returns a unique request ID. Store this immediately - you’ll need it to match responses later.

Handling the Redirect Curveball

Here’s something that caught me off guard initially: large files don’t return content directly. Instead, Graph gives you a redirect to an Azure Blob Storage URL where the actual file lives. Microsoft doesn’t specify the exact file size threshold, but in practice, I’ve observed this happening with files larger than a few MB.

private async Task<byte[]> DownloadFromRedirectAsync(Uri? redirectUri) {
    if (redirectUri == null)
        throw new ArgumentException("Redirect URI is null");

    using HttpClient httpClient = new HttpClient();
    return await httpClient.GetByteArrayAsync(redirectUri);
}

This happens because Microsoft doesn’t want to push huge files through the Graph API unnecessarily. The redirect URL is temporary and works great - just make sure you handle it properly.

Note: The exact file size that triggers a redirect isn’t officially documented by Microsoft, so always handle both direct content (200 OK) and redirect (302) responses in your code.

Error Handling That Actually Helps

When things go wrong (and they will), you want meaningful error messages. Here’s how to handle the common scenarios:

private async Task ProcessBatchResponseAsync(
    string requestId, 
    HttpResponseMessage response, 
    FileInfoDTO originalFile,
    Dictionary<string, byte[]> results) {
    
    try {
        switch (response.StatusCode) {
            case HttpStatusCode.OK:
                byte[] content = await response.Content.ReadAsByteArrayAsync();
                results[originalFile.UniqueFileName!] = content;
                break;
                
            case HttpStatusCode.Redirect:
            case HttpStatusCode.Found:
                byte[] redirectContent = await DownloadFromRedirectAsync(response.Headers.Location);
                results[originalFile.UniqueFileName!] = redirectContent;
                break;
                
            case HttpStatusCode.NotFound:
                Console.WriteLine($"File not found: {originalFile.Name} at {originalFile.RelativePath}");
                // Maybe the file was moved or deleted
                break;
                
            case HttpStatusCode.TooManyRequests:
                Console.WriteLine($"Throttled request for: {originalFile.Name}");
                // This is where retry logic would go (coming in the next post!)
                break;
                
            case HttpStatusCode.Forbidden:
                Console.WriteLine($"Access denied for: {originalFile.Name}");
                // Check your permissions
                break;
                
            default:
                Console.WriteLine($"Unexpected error downloading {originalFile.Name}: {response.StatusCode} - {response.ReasonPhrase}");
                break;
        }
    }
    finally {
        response.Content?.Dispose(); // Don't leak memory!
    }
}

Important Things to Remember

Batch Size Limits

Graph batching has a hard limit of 20 requests per batch. If you have more files, you’ll need to chunk them:

const int BATCH_SIZE = 20;
for (int i = 0; i < fileInfos.Length; i += BATCH_SIZE) {
    var batch = fileInfos.Skip(i).Take(BATCH_SIZE).ToArray();
    var batchResults = await DownloadFilesBatchAsync(batch, siteId, driveId);
    // Merge results...
}

Memory Management

Always dispose of HttpResponseMessage.Content. File downloads can be large, and forgetting to dispose will cause memory leaks that are painful to debug.

Request ID Uniqueness

Request IDs are unique within a single batch, but not across different batches. Don’t try to reuse mappings between different batch operations.

Why This Pattern Works So Well

This approach has several advantages that make it my go-to solution:

• Dead Simple: No complex logic, just a straightforward mapping pattern
• Reliable: Works consistently regardless of file sizes or response order
• Memory Efficient: Proper cleanup prevents memory leaks
• Debuggable: Easy to trace issues when something goes wrong
• Extensible: Perfect foundation for adding retry logic later

The beauty is in its simplicity. You’re not trying to guess which response belongs to which request - you know exactly because you mapped it from the start.

What’s Next?

This mapping pattern solves the core problem of correlating Graph batch responses with your original requests. But what happens when some requests fail due to throttling or temporary errors?

In my next post, I’ll show you how to build retry logic on top of this foundation that automatically handles failed requests without losing track of which files still need to be downloaded.

Have you run into this mapping challenge before? Let me know in the comments how you solved it - I’m always curious about different approaches! 🚀

Following up on my previous post about DevProxy and throttling testing, there’s another critical performance issue I see regularly in SharePoint CSOM code: executing operations one by one instead of batching them.

Consider this common pattern that I see everywhere:

// ❌ This approach is slow and inefficient
foreach (var id in itemIds)
{
    var item = list.GetItemById(id);
    context.Load(item);
    await context.ExecuteQueryAsync(); // Executing for EACH item!
    
    // Process the item
    ProcessItem(item);
}

This code makes a server round-trip for every single item. If you’re processing 50 items, that’s 50 separate calls to SharePoint. Each call has network latency and server processing time.

The Solution: Batch Operations

SharePoint CSOM reliably handles batches of around 100 operations before calling ExecuteQueryAsync(). This means you can dramatically reduce the number of server round-trips.

Microsoft’s official documentation also emphasizes this pattern in their performance guidelines, showing how grouping data retrieval operations significantly improves performance.

Here’s the pattern I use in most of my SharePoint projects:

The Helper Methods

public static async Task<List<T>> GetItemsByIds<T>(this List list, IEnumerable<int> ids) 
    where T : new()
{
    if (ids == null || !ids.Any())
    {
        return new List<T>();
    }

    var listItems = await CSOMHelpers.ProcessInChunks(ids.ToList(), 100, async chunk => {
        var items = chunk.Select(id => {
            var item = list.GetItemById(id);
            list.Context.Load(item);
            return item;
        }).ToList();

        await ExecuteQuery(list.Context, () => { /* Load calls already done above */ });
        
        return items;
    });

    return listItems.ToList();
}

internal static async Task<List<TOut>> ProcessInChunks<TIn, TOut>(List<TIn> source, int chunkSize, Func<List<TIn>, Task<List<TOut>>> action)
{
    var result = new List<TOut>();
    foreach (var chunk in source.Chunk(chunkSize))
    {
        result.AddRange(await action(chunk.ToList()));
    }
    return result;
}

Performance Impact: The Numbers

Let me show you the difference with a real-world example. Loading 50 SharePoint list items:

Before (One-by-One)

• 50 server calls
• Network latency: 50ms × 50 = 2.5 seconds
• Actual total time is usually higher due to server-side processing, hence: ~3-5 seconds

After (Batched)

• 1 server call (all 50 items in one batch)
• Network latency: 50ms × 1 = 50ms
• Total time: ~200-500ms

⚠️ Performance Disclaimer

The numbers shown above are from a specific example scenario and are meant to illustrate the concept of batching benefits. Your actual performance gains will vary significantly based on:

• Network latency and bandwidth
• SharePoint server load and location
• Item complexity and field count
• Query complexity and filtering
• Authentication overhead
• Time of day and concurrent users

While batching almost always improves performance, the exact improvement factor can range from marginal gains to dramatic improvements. The key takeaway is the pattern and approach, not the specific numbers.

Applying the Pattern to CRUD Operations

This pattern works for all CSOM operations, not just reading, but for all the CRUD operations.

Why 100 Items?

SharePoint has practical limits on how many operations you can batch:

• CSOM limit: Around 100 operations per batch
• REST API limit: Different limits depending on operation type
• Network payload: Larger batches mean bigger HTTP requests

I’ve found 100 items to be the sweet spot that:

• Maximizes performance gains
• Stays well within SharePoint limits
• Keeps HTTP payloads manageable
• Works reliably across different SharePoint environments

Push past those limits and SharePoint tells you, in one of two ways. Too many operations in a single ExecuteQuery gives you The request uses too many resources. Too much payload in one request gives you The server does not allow messages larger than 2097152 bytes - that’s 2 MB, and in SharePoint Online it’s a server-side cap you can’t raise from client code. No configuration setting, no header, no retry gets you past either one. Splitting the batch is the fix, which is what the helper above is for.

Combining with Throttling Protection

When you combine this batching approach with the throttling protection patterns from my DevProxy post, you get robust, high-performance SharePoint applications.

Best Practices

1. Batch whenever you can - even for small numbers of items, batching usually helps.
2. Use 100 as your chunk size for most scenarios
3. Combine with retry logic for production resilience
4. Test with DevProxy to ensure your batching works under throttling conditions
5. Monitor performance before and after implementing batching

The Bottom Line

Batching CSOM operations is one of the easiest wins for SharePoint performance optimization. The code pattern is straightforward to implement and reuse, but the performance impact is dramatic.

Stop executing SharePoint operations one by one. Your users (and SharePoint servers) will thank you.

Batching is the first of five techniques I use to keep CSOM cheap. The rest, and the order I reach for them in, are in The SharePoint CSOM Performance Playbook.

Resources

• SharePoint CSOM Best Practices
• DevProxy for Testing Throttling
• SharePoint Performance Guidelines

If you’ve ever optimized your C# API calls with parallel programming, you might know this story.

Your client wants faster performance. You run multiple API requests in parallel. Locally, everything flies — especially late at night when traffic is low. But the next morning, on the final pre-deployment test, disaster strikes.

Random errors. Requests delayed for 45 seconds.
Microsoft Graph or SharePoint has slammed you with throttling and rate limits. Your blazing-fast local solution crumbles under production-like conditions.

How CRUD Operations Can Secretly Burn Through Your Limits — and How to Catch Them with DevProxy

Here’s the thing:
SharePoint Online charges “Resource Units” (RUs) for every request you make. Think of RUs as an invisible currency — every API call you send deducts from your allowance. Run out too fast, and throttling kicks in.

And those “simple” operations? They’re not as cheap as you think.

From Microsoft’s RU table:

Source: Microsoft’s official SharePoint throttling documentation

Permission expansions are just one type of “expensive” call. Another common scenario that can impact RU consumption is making multiple separate queries instead of using CAML joins — something I explored in my CAML join post.
While joining multiple lists in a single query is actually more efficient than separate calls, poorly structured queries or retrieving unnecessarily large datasets can still consume RUs quickly if not optimized properly.

Why This Sneaks Up on You

When coding locally, it’s easy to run a handful of queries without noticing.
But in production, with real concurrency and volume, these calls pile up in seconds.

Imagine:

• 10 parallel updates (2 RUs each) = 20 RUs in one burst
• Add a few joins/expansions or large list queries, and you’re suddenly burning through limits 5x faster.

How DevProxy Can Show You the Pain Before Production

Here’s where DevProxy shines.
If you configure it to simulate throttling based on these RU-heavy calls, you’ll see the impact locally:

1. Enable throttling plugins in your devproxy.json (as shown earlier).
2. Point urlsToWatch at your SharePoint endpoints.
3. Run your CRUD-heavy code.

DevProxy will start throwing 429s (Too Many Requests) once your “fake” RU budget is exhausted — just like SharePoint would in production.

The beautiful part?
You can crank the limits down during testing to make expensive patterns obvious. Even a single large query or unbatched Update will light up your logs.

Example DevProxy log when hitting a throttling simulation:

[Warning] Throttling triggered: 2 parallel Create calls exceeded RU limit (RateLimitingPlugin)
Retry after: 30 seconds

Pro tip:
Use DevProxy as a budget meter for your API calls. Treat every 2-RU and 5-RU operation as “spending big” — and redesign those spots before they become a production outage.

What is DevProxy?

DevProxy is an open-source HTTP/HTTPS proxy server that can simulate real-world network issues, including:

• Rate limiting
• Throttling
• Network delays
• Intermittent errors

It’s free, open source, and designed for developers who want to catch performance bottlenecks before they reach production.

Installing DevProxy

Follow the official setup guide here:
DevProxy Installation Documentation

Basic Usage

Run DevProxy with the default configuration:

devproxy

It will begin intercepting all HTTP/HTTPS requests.

Tip: Start DevProxy before running your own code — otherwise, it won’t capture the traffic.

Simulating Microsoft Graph Throttling

Here’s how to configure DevProxy to reproduce Microsoft Graph and SharePoint throttling issues locally.

1. Create a devproxy.json configuration file

This is the exact configuration I used when testing my problematic parallel code:

{
  "$schema": "https://raw.githubusercontent.com/dotnet/dev-proxy/main/schemas/v0.27.0/rc.schema.json",
  "rate": 25,
  "plugins": [
    {
      "name": "RetryAfterPlugin",
      "enabled": true,
      "pluginPath": "~appFolder/plugins/dev-proxy-plugins.dll"
    },
    {
      "name": "RateLimitingPlugin",
      "enabled": true,
      "pluginPath": "~appFolder/plugins/dev-proxy-plugins.dll",
      "configSection": "rateLimitingPlugin"
    },
    {
      "name": "GraphRandomErrorPlugin",
      "enabled": true,
      "pluginPath": "~appFolder/plugins/dev-proxy-plugins.dll",
      "configSection": "graphRandomErrorPlugin"
    }
  ],
  "urlsToWatch": [
    "https://graph.microsoft.com/v1.0/*",
    "https://graph.microsoft.com/beta/*",
    "https://graph.microsoft.us/v1.0/*",
    "https://graph.microsoft.us/beta/*",
    "https://dod-graph.microsoft.us/v1.0/*",
    "https://dod-graph.microsoft.us/beta/*",
    "https://microsoftgraph.chinacloudapi.cn/v1.0/*",
    "https://microsoftgraph.chinacloudapi.cn/beta/*",
    "https://*.sharepoint.*/*_api/web/GetClientSideComponents",
    "https://*.sharepoint.*/*_api/*",
    "https://*.sharepoint.*/*_vti_bin/*",
    "https://*.sharepoint-df.*/*_api/*",
    "https://*.sharepoint-df.*/*_vti_bin/*"
  ],
  "graphRandomErrorPlugin": {
    "$schema": "https://raw.githubusercontent.com/dotnet/dev-proxy/main/schemas/v0.27.0/graphrandomerrorplugin.schema.json",
    "allowedErrors": [
      429,
      503,
      504
    ]
  },
  "rateLimitingPlugin": {
    "$schema": "https://raw.githubusercontent.com/dotnet/dev-proxy/main/schemas/v0.27.0/ratelimitingplugin.schema.json",
    "costPerRequest": 2,
    "rateLimit": 120
  },
  "logLevel": "information",
  "newVersionNotification": "stable",
  "showSkipMessages": true,
  "showTimestamps": true
}

2. Start DevProxy with your configuration

devproxy --config-file devproxy.json

Example of Problematic Code

Here’s the snippet that caused my throttling nightmare. It uses Parallel.ForEachAsync to query SharePoint in bulk:

// ❌ This will likely trigger throttling in production
var customers = new ConcurrentBag<CustomerDTO>();

await Parallel.ForEachAsync(departmentIds, async (departmentId, token) => 
{
    var clonedContext = _clientContext.Clone(_clientContext.Url);

    var query = Camlex.Query()
        .ViewFields(new CustomerDTO().ViewFields().ToArray().Append("DepartmentId"))
        .LeftJoin(x => x["DepartmentLookup"].ForeignList(DEPARTMENT_LIST_GUID))
        .ProjectedField(x => x["DepartmentId"].List(DEPARTMENT_LIST_GUID).ShowField("ID"))
        .Where(x => x["DepartmentId"] == departmentId);

    var departmentCustomers = await SharePointService.GetItemsFromListByQuery<CustomerDTO>(
        CUSTOMER_LIST_GUID,
        clonedContext,
        query);

    if (departmentCustomers?.Any() == true)
    {
        foreach (var customer in departmentCustomers)
        {
            customers.Add(customer);
        }
    }
});

Why This Code Fails in Production

1. No throttling safeguards — Multiple parallel requests overwhelm SharePoint.
2. No retry logic — Requests fail instead of recovering.
3. No request limiting — All departments are processed simultaneously.
4. Silent failures — Errors are ignored without logging or fallback.

The Better Way

Instead of hard-coding fixes here, I recommend Bert Jansen’s detailed guide on throttling and rate limit handling:
Throttling & Rate Limit Handling Patterns

Key principles:

• Limit concurrency with SemaphoreSlim.
• Use exponential backoff for retries.
• Monitor rate limit headers to adjust requests dynamically.
• Handle errors explicitly to prevent silent failures.

Conclusion

DevProxy has become an essential tool in my Microsoft 365 development workflow. It helps me:

• Catch throttling issues before production.
• Test error handling and retry logic locally.
• Deliver applications that can survive real-world API limits.

If I’d used DevProxy from the start, I could have avoided the last-minute throttling meltdown entirely.

💡 Pro tip: Make DevProxy part of your early development process, not your emergency toolkit.

Additional Resources

• DevProxy GitHub Repository
• Microsoft Graph Throttling Guidelines
• DevProxy Documentation

When working with related data across multiple SharePoint lists, developers often fall into the trap of making multiple individual queries:

// ❌ Bad approach - Multiple API calls
var customers = context.Web.Lists.GetByTitle("Customers");
var orders = context.Web.Lists.GetByTitle("Orders");
var orderItems = context.Web.Lists.GetByTitle("OrderItems");

// Each query consumes 2 resource units
var customerItems = customers.GetItems(camlQuery1);  // 2 units
var orderItems = orders.GetItems(camlQuery2);        // 2 units  
var itemDetails = orderItems.GetItems(camlQuery3);   // 2 units
// Total: 6 resource units + processing overhead

This approach has several issues:

• Higher resource consumption: Each query consumes 2 resource units per multi-item request
• Increased throttling risk: More API calls mean hitting limits faster
• Network overhead: Multiple round trips to SharePoint
• Complex data merging: Manual joining of results in C#

How to fix it?

I have worked alot with MSSQL where it is pretty easy to just join a table on a table on a table. But I did not know it was possible in CSOM. Eg. in the UI of SharePoint you can only expand one table - NOT multiple. But one day I deep dived into CAML and joins, and found out it was possible!

Resource Unit Comparison:

Let’s break down the actual cost difference. According to Microsoft’s throttling documentation, each multi-item query consumes 2 resource units.

Multiple separate queries (❌ Bad approach):

// Query 1: Get order items from main list
var orderItems = ordersList.GetItems(query1);        // 2 resource units

// Query 2: Get customer details  
var customers = customersList.GetItems(query2);      // 2 resource units

// Query 3: Get order details
var orderDetails = orderDetailsList.GetItems(query3); // 2 resource units

// Total: 6 resource units + network overhead + manual C# joining

Single join query (✅ Good approach):

// One query with joins gets ALL the data
var items = ordersList.GetItems(joinQuery);          // 2 resource units total

The math:

• Multiple queries: 2 units × 3 lists = 6 units
• Single join query: 2 units total
• Savings: 66% fewer resource units!

This becomes even more significant when you consider that SharePoint throttling limits are measured in resource units per time window. With joins, you can query 3x more data within the same throttling limits.

The Solution: CAML Joins

CAML actually supports joining multiple lists in a single query! This means you can get data from related lists without multiple API calls.

First, install CAMLEX via NuGet:

Install-Package Camlex.Client.dll

Then build your join query:

var list = context.Web.Lists.GetByTitle("YourMainList");

CamlexNET.Interfaces.IQuery query = Camlex.Query();

query = query
.LeftJoin(x => x["OrderTaskLookUp"].ForeignList(ListGuidOrderTasks))
.LeftJoin(x => x["OrderDetailLookUp"].PrimaryList(ListGuidOrderTasks).ForeignList(ListGuidOrders))
.LeftJoin(x => x["CustomerLookUp"].PrimaryList(ListGuidOrders).ForeignList(ListGuidCustomers))
.ProjectedField(x => x["CustomerNo"].List(ListGuidCustomers).ShowField("CustomerNo"))
.ProjectedField(x => x["CustomerName"].List(ListGuidCustomers).ShowField("CustomerName"));

var camlQuery = new CamlQuery();
camlQuery.ViewXml = query.ToString();

var items = list.GetItems(camlQuery);
context.Load(items);
context.ExecuteQuery();

Important: Your lists need to be connected via lookup columns for joins to work!

Why CAMLEX Instead of Raw CAML?

I don’t write raw CAML myself - it’s verbose and error-prone. Instead, I use CAMLEX because:

• Cleaner syntax: C# lambda expressions instead of XML
• IntelliSense support: Catch errors at compile time
• Easier for the next developer: Self-documenting code
• Less mistakes: No more XML typos or missing tags

What else?

One thing is to join multiple lists, but to filter on a value 4 lists away - That is neat!

For example, filtering orders by the customer’s name, which is 2 lists away:

CamlexNET.Interfaces.IQuery query = Camlex.Query();

query = query
.Where(x => (string)x["CustomerName"] == "Contoso") // <---- Filtering on join field
.LeftJoin(x => x["OrderTaskLookUp"].ForeignList(ListGuidOrderTasks))
.LeftJoin(x => x["OrderDetailLookUp"].PrimaryList(ListGuidOrderTasks).ForeignList(ListGuidOrders))
.LeftJoin(x => x["CustomerLookUp"].PrimaryList(ListGuidOrders).ForeignList(ListGuidCustomers))
.ProjectedField(x => x["CustomerNo"].List(ListGuidCustomers).ShowField("CustomerNo"))
.ProjectedField(x => x["CustomerName"].List(ListGuidCustomers).ShowField("CustomerName"));

var camlQuery = new CamlQuery();
camlQuery.ViewXml = query.ToString();
var items = ordersList.GetItems(camlQuery);

This CAMLEX query automatically generates the following CAML XML - notice how complex the raw XML is compared to the clean C# syntax above:

<View>
  <Query>
    <Where>
      <Geq>
        <FieldRef Name="customerName" />
        <Value Type="Text">Contoso</Value>
      </Geq>
    </Where>
  </Query>
  <ViewFields>
    <FieldRef Name="Id" />
    <FieldRef Name="customerNo" />
    <FieldRef Name="customerName" />
  </ViewFields>
  <Joins>
    <Join Type="LEFT" ListAlias="0f0cfc71-1c6e-4fd4-b6f2-279d0e3862f4">
      <Eq>
        <FieldRef Name="OrderTaskLookUp" RefType="Id" />
        <FieldRef List="0f0cfc71-1c6e-4fd4-b6f2-279d0e3862f4" Name="Id" />
      </Eq>
    </Join>
    <Join Type="LEFT" ListAlias="ce57b9a2-5052-482c-a8c8-150c7d59ced3">
      <Eq>
        <FieldRef List="0f0cfc71-1c6e-4fd4-b6f2-279d0e3862f4" Name="OrderDetailLookUp" RefType="Id" />
        <FieldRef List="ce57b9a2-5052-482c-a8c8-150c7d59ced3" Name="Id" />
      </Eq>
    </Join>
    <Join Type="LEFT" ListAlias="e204ed0f-7c25-432f-9228-3eb438c527e2">
      <Eq>
        <FieldRef List="ce57b9a2-5052-482c-a8c8-150c7d59ced3" Name="CustomerLookUp" RefType="Id" />
        <FieldRef List="e204ed0f-7c25-432f-9228-3eb438c527e2" Name="Id" />
      </Eq>
    </Join>
  </Joins>
  <ProjectedFields>
    <Field 
      Name="customerNo" 
      Type="Lookup" 
      List="e204ed0f-7c25-432f-9228-3eb438c527e2" 
      ShowField="customerNo" 
    />
    <Field 
      Name="customerName" 
      Type="Lookup" 
      List="e204ed0f-7c25-432f-9228-3eb438c527e2" 
      ShowField="customerName"
    />
  </ProjectedFields>
</View>

Imagine having to write and maintain that XML manually! This is exactly why CAMLEX is so valuable - you get all the power of CAML joins with readable C# syntax.

Performance Benefits

Before (Multiple queries):

• 🔴 6+ resource units
• 🔴 Multiple network calls
• 🔴 Complex C# merging logic

After (Single join):

• ✅ 2 resource units only
• ✅ One network call
• ✅ Server-side joining

Key Takeaways

• Use joins instead of multiple queries to reduce resource consumption
• CAMLEX makes CAML readable for you and the next developer
• You can filter on joined data even multiple lists away
• SharePoint UI limitations ≠ API limitations - joins work even if the UI doesn’t show it

Common Issues & Solutions

❌ “List does not exist” error

// Use list GUIDs instead of names for reliability
.ForeignList(new Guid("12345678-1234-1234-1234-123456789012"))

❌ “Field not found” error

// Use internal field names, not display names
.ShowField("Title")        // ✅ Internal name
.ShowField("Customer Name") // ❌ Display name

❌ Join returns no data

• Verify lookup columns exist and are properly configured
• Check that you’re joining on the correct fields
• Ensure the lookup field contains valid IDs

❌ “Cannot project this field type” error

Only specific field types can be included in ProjectedFields:

✅ Supported ProjectedFields types:

• Calculated (treated as plain text)
• ContentTypeId
• Counter
• Currency
• DateTime
• Guid
• Integer
• Note (one-line only)
• Number
• Text

❌ NOT supported in ProjectedFields:

• Multi-line text fields
• Rich text fields
• Choice fields
• Lookup fields (use joins instead)
• User/Person fields
• Managed metadata fields

💡 Pro tip: If you need data from unsupported field types, query them separately after getting your joined results.

CAML joins are one of five techniques in The SharePoint CSOM Performance Playbook, which covers when a join is the right answer and when batching or change detection would serve you better.

Microsoft Graph is the unified REST API for Microsoft 365. Think of it as a single gateway to access data across SharePoint, Teams, OneDrive, Outlook, Entra ID, and more, all through one consistent API.

Graph vs CSOM/PnP.Framework:

• Graph: Modern REST API, works across all Microsoft 365 services
• CSOM/PnP: SharePoint-specific, more SharePoint features available

When to Use Graph SDK vs PnP.Framework

Reach for the Graph SDK when you need Teams, OneDrive, Exchange or Entra ID data, when you want the built-in retry and batching, or when the feature spans several Microsoft 365 services.

Reach for PnP.Framework when you need the SharePoint-specific surface: site templates, provisioning, taxonomy, search refiners, managed metadata, content types.

Graph won’t help you with SharePoint classic features like master pages and web parts, its search is thinner than the SharePoint Search API, and it can’t talk to SharePoint Server on-premises at all.

Which SDK Version Are You On?

This is the part that costs people an afternoon, so it comes before the code.

The .NET SDK was rewritten in v5. It moved to Kiota-generated clients, and .Request() was removed from every call. That one change means a v4 sample does not compile on v5, and a v5 sample does not compile on v4. Most of the samples you’ll find online, including the first version of this post, are v4.

v6 is the current major, and v5 code compiles on it unchanged. I checked: the same file builds clean against 5.105.0 and 6.5.0. So there are really two dialects to care about, not three.

# current: v5 syntax, v6 package
dotnet add package Microsoft.Graph --version 6.5.0

# legacy: only if you're pinned to v4 already
dotnet add package Microsoft.Graph --version 4.54.0

Pin it. Install-Package Microsoft.Graph with no version gives you the newest major, and if you then paste a v4 sample into it you get a wall of does not contain a definition for 'Request'.

One thing to get out of the way: there is no Microsoft.Graph.Auth package. It never had a stable release and the namespace doesn’t exist in v4, v5 or v6. If a sample tells you to add using Microsoft.Graph.Auth;, that sample predates all of this. You want Azure.Identity.

dotnet add package Azure.Identity

Add Graph Permissions

In your existing app registration, go to “API permissions” then “Add a permission” then “Microsoft Graph” then “Application permissions”:

• Sites.ReadWrite.All: SharePoint sites and lists access

Click “Grant admin consent” after adding permissions, or every call comes back 403.

The Code, v5 and v6

Same certificate and app registration as the PnP.Framework post.

using Azure.Identity;
using Microsoft.Graph;
using Microsoft.Graph.Models;
using System.Security.Cryptography.X509Certificates;

class Program
{
    private static readonly string TenantId = "<tenant-id>";
    private static readonly string ClientId = "<client-id>";
    private static readonly string CertificatePath = @"C:\Temp\cert\appcert.pfx";
    private static readonly string CertificatePassword = "<password>";

    static async Task Main(string[] args)
    {
        var certificate = X509CertificateLoader.LoadPkcs12FromFile(CertificatePath, CertificatePassword);
        var options = new ClientCertificateCredentialOptions
        {
            AuthorityHost = AzureAuthorityHosts.AzurePublicCloud,
        };

        var credential = new ClientCertificateCredential(TenantId, ClientId, certificate, options);
        var scopes = new[] { "https://graph.microsoft.com/.default" };

        var graphClient = new GraphServiceClient(credential, scopes);

        var org = await graphClient.Organization.GetAsync();
        Console.WriteLine($"Connected to: {org?.Value?.FirstOrDefault()?.DisplayName}");

        var site = await graphClient.Sites["root"].GetAsync();
        var lists = await graphClient.Sites[site.Id].Lists.GetAsync();

        Console.WriteLine("Lists in this site:");
        foreach (var list in lists?.Value ?? [])
        {
            Console.WriteLine($"- {list.DisplayName}");
        }

        var docLib = (lists?.Value ?? []).FirstOrDefault(l => l.DisplayName == "Documents");
        if (docLib != null)
        {
            var content = System.Text.Encoding.UTF8.GetBytes("Hello from Graph SDK!");
            var drive = await graphClient.Sites[site.Id].Lists[docLib.Id].Drive.GetAsync();

            var file = await graphClient.Drives[drive.Id]
                .Items["root:/sample-document.txt:"]
                .Content
                .PutAsync(new MemoryStream(content));

            Console.WriteLine($"Uploaded file: {file?.Name}");
        }
    }
}