Organization.svc performance issue

[godind]

Hi,

We are seeing 300-400% performance degradation on all D365 CRM Org service calls with the XRM Community Ed compared to Adxstudio 7.x version. We've basically just replace the ADX/XCE part and not changed our customizations (decides adaptations required to migrate - very minimal).

Has any one seen this problem? Could it be a CRM SDK issue, or some other web.config setting that need to be present, added or changed with XCE. Some new IIS or ASP.Net tuning requirements needed for XCE?

We have a IIS VM connected to D365 CRM configured with local cache WebNotifications.axd (what we works the same as in V7 cache.axd, but not sure...).

Thank you

[godind]

Just to add details, we have used the steps describe here to enable WebNotifications read:https://www.engineeredcode.com/blog/automatic-cache-invalidation-for-xrm-portals-community-edition

[gegoodwin]

We have recently upgraded from v7 to Community Edition and are also experiencing a performance decrease. We have not been able to pinpoint a cause of the slow down. Has anyone else experienced this or found a solution to improve performance?

[jliberta]

@gegoodwin Webnotification reduction has helped us improve performance but we are still experiencing CPU spikes that we did not experience before.

[godind]

See #81

[amervitz]

@godind please provide more details on this statement:

We are seeing 300-400% performance degradation on all D365 CRM Org service calls

How have you measured this? Do you have any examples of specific requests that are slower than before?

[godind]

@amervitz The degradation was a result of moving a solution from V7 to XCE (following required migration steps). The orgsvc measurement came from perf telemetry solution and was generalized. WebNotification/Cache Invalidation process behaves and impacts perf differently in XCE/V8.1 so that is probably the main reason. Following a support call some CRM/SQL indexing improvement was made by engineers and it dramatically improved orgsvc performance; the specifics of what was done was not shared.

This supports the WebNotification/Cache Invalidation assumption as the solution has no custom massive Contact entity feature besides the out of the box stuff and the solution's logic was not changed significantly in the migration.

[amervitz]

@godind thank you for the exta information. Unfortunately, without specific information about what requests are slower, it's going to be a challenge to identify and make any changes. This would be an open ended investigation that could prove to be very time-consuming because of the limited information available to work with. If you or others in the community are interested you may feel free to continue with this effort but at this point I'm afraid there's not enough tangible information for me to pursue this further.

[godind]

@amervitz Agreed. I guess the only thing to keep in mind is: the V7-XCE move impacted Orgsvc call. After CRM backend optimization work it improved from a 300-400% degradation down to about 40-50%.

[slautebach]

@godind What indexing improvements did you make to CRM? Can you elaborate and share what entities/attributes you applied for out of the box/portal entities?

[godind]

@slautebach this was for a D365 CRM environment so we did not do ourselves and information was not shares in details. There was quite a few round trips with support to obtain perf info and request tuning. For this specific thread, it's been a while now, but I remember primarily WebRoles and secondly Contacts entities, if memory serves well.

@slautebach are you seeing issue too?

[gegoodwin]

@godind we have definitely experienced performance degradation moving from v7 to XCE. Any information you can provide regarding the indexing solution such as support contacts or ticket number would be helpful to help us fast track a solution with the Dynamics team.

[slautebach]

@Godlind we are having performance issues in general, that have been discussed by @jayrodmcneil in other threads. Just looking for any insights in to see how we can improve our performance. We are running everything on premise, so well have to look at our contacts and web roles to see if there is anything we can do. Thanks.

[godind]

@gegoodwin Sadly I do not have this information. But, if you are fortunate enough to have perf telemetry from before and after, look at total calls/operations that have increased and than average response time for those.

If OrgSvc is worst, it's probably SQL. Show this telemetry to MS support. They are pretty good at fixing those types of issues but, you need to come with very strong facts supporting your theory to get this moving, especially if it involves xCE.

[jliberta]

@amervitz would you happen to know if there is a difference between rendering a web page using the Master Portal solution by configuring the webpage records as "rewrite" in CRM versus using web templates containing liquid code? Would using web templates reduce stress on the front end webservers? Given that a web template doesn't reference an ASPX page in the MasterPortal project does that mean that when a user hits a webpage on the portal that uses a web template then CRM does all the liquid processing and returns the final HTML page to the portal which is then displayed to the user?

Thanks in advance!

[amervitz]

@jliberta web templates result in one of two ASPX files executing. This is the code that chooses the ASPX file to use when using a web template.

xRM-Portals-Community-Edition/Framework/Adxstudio.Xrm/Web/ContentMapCrmSiteMapProvider.cs

Lines 479 to 481 in 0a68951

	var rewriteUrl = template.Type == (int)PageTemplateNode.TemplateType.WebTemplate && template.WebTemplateId != null
	? template.UseWebsiteHeaderAndFooter.GetValueOrDefault(true) ? "~/Pages/WebTemplate.aspx" : "~/Pages/WebTemplateNoMaster.aspx"
	: template.RewriteUrl;

The difference in performance will be determined by what executes as part of each ASPX page.

[jliberta]

@amervitz thank you very much this is helpful!