David Evans replied to a different blog post of mine and pointed out that the product team has done a very large amount of work to make things work quicker through the CRM SDK and shared the video below. I have heard David Cai tout how he has seen well over 200 rows per second in SSIS though his product. I thought this was AMAZING considering I was used to Scribe and if I got 15 records per second I was pretty dang happy. I could run more than one thread at once though if I could chunk up the job properly.
In the video below I was blown away. Literally you will see using the maximum threads per second against CRM Online which hasn't always been known for being the most performant depending on where in the country you are accessing it from. You will see here shortly after 3:10 in the video where in a matter of seconds the process they run revs up to OVER 650 ROWS PER SECOND! They are running 16 threads, but now I have a new appreciation for just what the SDK is capable of handling if you hit it just the right way.
Check out the video here and just skip to 3:10-ish to see them start running first with one thread and then with 16. WOW!!! http://player.vimeo.com/video/74729381
- I hope you enjoy this video!
-
I don't think that David has told the actual story in the video. First, he must be loading from an Azure environment to CRM online. Second, if you use 16 concurrent threads loading to CRM Online using a batch size of 250, it is very likely that you will hit CRM online throttling in which case your data load will fail. Otherwise, it is very impressive number.
ReplyDeleteI had some clarifications from David Evans on Facebook, his network latency to CRM online server was roughly 23ms, which is impressively low, which makes me particularly jealous of CRM users in UK. However having 16 concurrent threads writing to CRM Online using a batch size (in which case ExecuteMultiple request will be used) is not recommended, so practically you can't really use this kind of combination for production purpose, unless you don't mind your data load process fails. The reason is, there is a throttling with CRM online, you can't have more than 2 concurrent ExecuteMultiple request at the same time.
ReplyDeleteWhy manifest 16 threads when you can use a Parallel.For loop to insure maximum thread output and automatic load balancing?
ReplyDelete