Monday, June 11, 2012

LINQ to CRM 2011 - Attribute Level Change Tracking Support


I really like LINQ to CRM 2011, it makes development against CRM easier.  However, out of the box LINQ to CRM has one big issue - it only supports entity level change tracking and not attribute level tracking.

This means that whenever you retrieve something via LINQ, mark the entity as "changed" and submit the changes, whether or not you have changed any attributes - it will update ALL of the attributes for that entity in the database - because it does not distinguish which attributes have changed and which haven't before it makes the update.

There are bad some side effects of this:
- The new Auditing feature in CRM 2011 will record all the fields as changed, not just the ones you actually changed.  This not only takes up database space, but makes it harder to read.
- Plugins and Workflows will trigger unnecessarily if you have them filtered on particular fields which you did not actually change.
- You may inadvertently overwrite changes that have occurred since you retrieved the record that you did not mean to.
- If you have field level security on some of those attributes, they may cause an exception even though you never changed it's value.

Luckily, LINQ to CRM allows you to intercept some useful events in it's life-cycle (Attach, SaveChanges, Execute) through the use of a sub-class or partial class, replacing it with your own functionality. Because of this, I've managed to write a class to overcome these issues and commit only the actual attribute changes to CRM rather than ALL of attributes.

So How does this class work?

- Intercepts when an entity is "Attached" or "Detached" from the context.  If the entity has a status of "unchanged" a shallow clone is taken of the entity and stored in an "originals" list; the clone is removed from the "originals" list if it is later detached.  Note: Entities will have a status of "unchanged" if they have been retrieved via LINQ from CRM or manually attached using the Attach() method.

- Intercepts the call to "SaveChanges()" method, inspects each entity that has been flagged as "changed" and compares it against the original version of the entity.  If a change in attributes is detected then a new entity is created containing only the changed attributes and the entity's key; this new entity is put into a new "Deltas" collection.  In addition, any entities that have not changed at all but are marked as "changed" are reset back to "unchanged", so they are not included in the commits that are made against CRM.

- Intercepts when the Execute Method is called to commit the data.  When the data is committed to the database, it uses the "UpdateRequest" class as you normally would using non-LINQ to CRM Entities.  When this is done for entity in the list of entities marked as "changed", the Target property of the UpdateRequest object used is reset to use the delta version of the entity rather than the complete entity itself.  This therefore avoids committing all attributes to the database, and instead only sends those fields that have changed.

Can I use it in a Plugin?

Yes. You should even be able to use it in the online version of CRM.

How do I use it?

- Place it in the same project as your LINQ to CRM context.
- Change the namespace so that it macthes the same namespace as your LINQ to CRM Context.
- Change the class name so that it macthes the same class name as your LINQ to CRM Context.
- If you are doing read-only operations, to remove the change tracking overhead you can disable the functionality with the "EnableAttributeChangeTracking" flag - set it to false as soon as you new up your context.

Any Gotya's?

- Attach(): If you are going to new up an Entity for an update without retrieving the entity from CRM first, make sure you set the key and then Attach() the entity before you change the attributes on that entity.  Otherwise it will not track the changes.

MergeOption: If you change the merge option, it may not work.  AppendOnly (the default) is currently the only supported merge option.

- EntityCollection: Currently, the EntityCollection attribute type is not supported when identifying changes between the original and the current version of an entity.  EntityCollection attributes are always assumed to be "Changed" if included in an attribute, and will always be submitted for update.

Cheers

Matt
                                                                     
                                                                     
                                                                                                                
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using Microsoft.Xrm.Sdk;
using Microsoft.Xrm.Sdk.Messages;

namespace Example.Linq
{
    public partial class CrmContext
    {
        private List<Entity> _originalEntities = new List<Entity>();
        private List<Entity> _deltaEntities = new List<Entity>();

        private bool _enableAttributeChangeTracking = true;

        /// <summary>
        /// Enables or disables Attribute Level Change tracking.
        /// Default is On.
        /// </summary>
        public bool EnableAttributeChangeTracking
        {
            get
            {
                return _enableAttributeChangeTracking;
            }

            set
            {
                if (_originalEntities.Count > 0 && value == false)
                    throw new Exception("You cannot disable attribute change tracking at this time, entities are already being tracked");

                _enableAttributeChangeTracking = value;
            }
        }

        /// <summary>
        /// Overrides the base OnBeginEntityTracking method
        /// When an entity is tracked, this adds the original unmodified version of that entity
        /// to the unchangedEntities list (which contains the original version of every entity).
        /// </summary>
        /// <param name="entity"></param>
        protected override void OnBeginEntityTracking(Entity entity)
        {
            base.OnBeginEntityTracking(entity);

            if (_enableAttributeChangeTracking && entity.EntityState == EntityState.Unchanged)
            {
                var exists = _originalEntities.Where(x => x.Id == entity.Id).FirstOrDefault();
                if (exists == null) _originalEntities.Add(ShallowClone(entity));
            }
        }

        /// <summary>
        /// Overrides the base OnEndEntityTracking method
        /// For any entity that is detached from entity tracking, this also 
        /// removes it from the unchangedEntities list (which contains the original version of every entity)
        /// </summary>
        /// <param name="entity"></param>
        protected override void OnEndEntityTracking(Entity entity)
        {
            base.OnEndEntityTracking(entity);
            if (_enableAttributeChangeTracking)
            {
                var exists1 = _originalEntities.Where(x => x.Id == entity.Id).FirstOrDefault();
                if (exists1 != null) _originalEntities.Remove(exists1);
                var exists2 = _deltaEntities.Where(x => x.Id == entity.Id).FirstOrDefault();
                if (exists2 != null) _deltaEntities.Remove(exists2);
            }
        }

        /// <summary>
        /// Overrides the base OnExecuting Method
        /// For UpdateRequests, replaces the target entity on the update message
        /// with an entity that only contains the key and the changed attributes.
        /// </summary>
        /// <param name="request"></param>
        protected override void OnExecuting(OrganizationRequest request)
        {
            if (_enableAttributeChangeTracking && typeof(UpdateRequest) == request.GetType())
            {
                var updateRequest = (UpdateRequest)request;
                var target = updateRequest.Target;
                var newTarget = _deltaEntities.Where(x => x.Id == target.Id).FirstOrDefault();
                updateRequest.Target = newTarget;
            }
            base.OnExecuting(request);

        }

        /// <summary>
        /// Overrides the base OnSavingChangesMethod
        /// For each entity, determines if an update is required.
        /// If no update is required, it detaches and reattaches to set the entity back to a "unchanged" state.
        /// Also create the entity that will ACTUALLY be subitted to the database (used in the OnExecuting method).
        /// </summary>
        /// <param name="options"></param>
        protected override void OnSavingChanges(Microsoft.Xrm.Sdk.Client.SaveChangesOptions options)
        {

            // Clear the list of entities to be sumbitted
            _deltaEntities.Clear();

            // Mark any entities as unchanged that
            // are only sending the key
            var updated = this.GetAttachedEntities().Where(x => x.EntityState == EntityState.Changed).ToList();

            foreach (var target in updated)
            {
                // Ignore updates where nothing has been updated
                var unchanged = _originalEntities.Where(x => x.Id == target.Id).FirstOrDefault();
                if (unchanged != null)
                {
                    var cloneOfTarget = ShallowClone(target);
                    RemoveUnchangedFields(cloneOfTarget, unchanged);
                    _deltaEntities.Add(cloneOfTarget);

                    // Test to see if it's only the key left... if so ignore the update otherwise you will get a blank audit record
                    if (cloneOfTarget.Attributes.Count == 1 && cloneOfTarget.Attributes.First().Value != null)
                    {
                        if (cloneOfTarget.Attributes.First().Value is Guid || cloneOfTarget.Attributes.First().Value is Guid?)
                        {
                            if (cloneOfTarget.Id == (Guid?)cloneOfTarget.Attributes.First().Value)
                            {
                                this.Detach(target);
                                _deltaEntities.Remove(cloneOfTarget);
                                target.EntityState = EntityState.Unchanged;
                                this.Attach(target);

                            }
                        }
                    }
                }
            }

            base.OnSavingChanges(options);
        }

        protected override void OnSaveChanges(SaveChangesResultCollection results)
        {
            _deltaEntities.Clear();
            base.OnSaveChanges(results);
        }

        /// <summary>
        /// Overrides the base OnExecute Method
        /// For Update Requests, if an error was thrown ignores it if Target is null (i.e. no need to update)
        /// </summary>
        /// <param name="request"></param>
        /// <param name="exception"></param>
        protected override void OnExecute(OrganizationRequest request, Exception exception)
        {
            ///
            if (_enableAttributeChangeTracking && typeof(UpdateRequest) == request.GetType())
            {
                var updateRequest = (UpdateRequest)request;

                if (updateRequest.Target != null)
                {
                    base.OnExecute(request, exception);
                }
            }
            else
            {
                base.OnExecute(request, exception);
            }
        }


        /// <summary>
        /// This method loops through comparing the changed entity with the unchanged entity
        /// and removed the unchanged fields from the changed entity.  Keys are kept.
        /// </summary>
        /// <param name="changed"></param>
        /// <param name="unchanged"></param>
        public void RemoveUnchangedFields(Entity changed, Entity unchanged)
        {

            // Lookp through the changed fields, if there is one missing from the unchanged list, add it as null so at least it will be compared.
            foreach (var changedAttribute in changed.Attributes)
            {
                var key = changedAttribute.Key;

                if (!unchanged.Contains(key))
                    unchanged.Attributes.Add(new KeyValuePair<string, object>(key, null));
            }

            // Loop through each attribute and compare properties
            foreach (var unchangedAttribute in unchanged.Attributes)
            {
                var key = unchangedAttribute.Key;

                var originalAttributeVal = unchanged.Attributes[key];
                var changedAttributeVal = changed.Attributes[key];

                // Fix any original value issues
                if (originalAttributeVal != null)
                {
                    // Fix any zero length strings
                    if (originalAttributeVal.GetType() == typeof(string) && string.IsNullOrWhiteSpace((string)originalAttributeVal))
                    {
                        originalAttributeVal = null;
                    }
                    else
                    {
                        // Fix any dates that aren't of a specified kind
                        if (originalAttributeVal.GetType() == typeof(DateTime) || originalAttributeVal.GetType() == typeof(DateTime?))
                        {
                            var date = (DateTime?)originalAttributeVal;
                            if (date.Value.Kind == DateTimeKind.Unspecified)
                                originalAttributeVal = date.Value.ToUniversalTime();
                        }
                    }
                }

                // Fix any changed value issues
                if (changedAttributeVal != null)
                {
                    // Fix any zero length strings
                    if (changedAttributeVal.GetType() == typeof(string) && string.IsNullOrWhiteSpace((string)changedAttributeVal))
                    {
                        changedAttributeVal = null;
                    }
                    else
                    {
                        // Fix any dates that aren't of a specified kind
                        if (changedAttributeVal.GetType() == typeof(DateTime) || changedAttributeVal.GetType() == typeof(DateTime?))
                        {
                            var date = (DateTime?)changedAttributeVal;
                            if (date.Value.Kind == DateTimeKind.Unspecified)
                                changedAttributeVal = date.Value.ToUniversalTime();
                        }
                    }
                }

                // Compare the values
                if (originalAttributeVal == null && changedAttributeVal == null)
                {
                    changed.Attributes.Remove(key);
                }
                else if (originalAttributeVal != null && changedAttributeVal == null)
                {
                    // Do nothing, keep the attribute        
                }
                else if (changedAttributeVal is EntityCollection)
                {
                    // Do nothing, Leave it in... (complicated to compare).
                }
                else if (changedAttributeVal.Equals(originalAttributeVal))
                {
                    if (unchangedAttribute.Value.GetType() == typeof(Guid) || unchangedAttribute.Value.GetType() == typeof(Guid?))
                    {
                        // Avoid removing the key
                        if (!((Guid?)unchangedAttribute.Value == unchanged.Id))
                        {
                            changed.Attributes.Remove(key);
                        }
                    }
                    else
                    {
                        changed.Attributes.Remove(key);
                    }
                }
            }
        }

        /// <summary>
        /// Take a shallow copy of an entity
        /// </summary>
        /// <param name="entity"></param>
        /// <returns></returns>
        private static Entity ShallowClone(Entity entity)
        {
            var clone = new Entity(entity.LogicalName);
            clone.EntityState = entity.EntityState;
            clone.Id = entity.Id;

            foreach (var attribute in entity.Attributes)
            {
                if (attribute.Value == null)
                {
                    clone.Attributes.Add(attribute.Key, null);
                }
                else if (attribute.Value is EntityReference)
                {
                    EntityReference entityReference = attribute.Value as EntityReference;
                    clone.Attributes.Add(attribute.Key, new EntityReference(entityReference.LogicalName, entityReference.Id));
                }
                else if (attribute.Value is Money)
                {
                    Money money = attribute.Value as Money;
                    clone.Attributes.Add(attribute.Key, new Money(money.Value));
                }
                else if (attribute.Value is OptionSetValue)
                {
                    OptionSetValue option = attribute.Value as OptionSetValue;
                    clone.Attributes.Add(attribute.Key, new OptionSetValue(option.Value));
                }
                else if (attribute.Value is EntityCollection)
                {
                    // Don't copy EntityCollection values, just re-reference it.
                    EntityCollection entityCollection = attribute.Value as EntityCollection;
                    clone.Attributes.Add(attribute.Key, entityCollection);
                }
                else // value types
                {
                    clone.Attributes.Add(attribute.Key, attribute.Value);
                }
            }

            return clone;
        }

    }
}

15/09/2012 Update: If you are using the entities generated by the CRM 2011 Developer Toolkit rather than the CrmSvcUtil.exe, it doesn't generate a context class for some reason - so, in order to use the partial class above, you'll just need to create a simple class for the Context similar to the code below and use the CrmContext.CreateQuery<Entity>() syntax instead of the CrmContext.EntitySet when querying.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.Xrm.Sdk.Client;

namespace Example.Linq
{
    public partial class CrmContext : OrganizationServiceContext
    {        
        /// <summary>
        /// Constructor.
        /// </summary>
        public CrmContext(Microsoft.Xrm.Sdk.IOrganizationService service) : 
                base(service)
        {
        }
        
    }
}

28 comments:

Anonymous said...

Hi,
Is it ok to reuse your code?
Thanks

Matthew Hunter said...

Yep, you sure can!

snataw said...

Hi Matt,

Thanks for sharing the code, it works like a charm, great work!

Anonymous said...

Hi Matt,

Thanks for the code. Very well done and much appreciated.

Anonymous said...

Hi Matt,

Thanks for the code. Well done and much appreciated.

Anonymous said...

Hi Matt,

Thanks for the code. Well done and much appreciated.

Anonymous said...

Hi Matt,

Thanks for the code. Well done and much appreciated.

Anonymous said...

Hi Matt,

Thanks for the code. Well done and much appreciated.

seguros said...

On OnEndEntityTracking(Entity entity) overrided method last line isn't it suppose to be if (exists2 != null) _deltaEntities.Remove(exists2); instead of cchecking for exists1?

seguros said...

On OnExecuting overrided method shouldn't you be calling base.OnExecuting(updateRequest); instead of base.OnExecuting(request);?

Matthew Hunter said...

"updateRequest" is just a casted version of request - so it's fine.

Matthew Hunter said...

As for the exists1 & 2 - yep my bad :) bug! I've fixed it in the above code.

seguros said...

The code works great BUT I had to change my context merge option to MergeOption.NoTracking, and now this doesn't work. Any work around?

Matthew Hunter said...

Sorry, haven't used the Merge Option yet - it only works with the default mode. I'll have a look when I next get a chance.

waterbox said...

The attach after detatch is causing exception to be thrown as another type (call it BAR) is not in the null or unchanged state.

Any ideas why this is happening or what the solultion is? Objects are being and attached to the object that is having the issue using:

ServiceContext.AddRelatedObject(foo, relationship, BAR)

Matthew Hunter said...

Hi, I'm not sure what the problem is. Could you clarify if you think the issue is with or the Attribute Level Change tracking OR is it just an issue in general.

I found this:

http://social.microsoft.com/Forums/en-US/8b7e6aba-471b-43f1-9d8a-ca985e6a0a0e/the-product-entity-must-be-in-the-default-null-or-unchanged-state-when-using-xrm

Where someone had a similar issue, turned out to be a plugin.

I would have expected the AddRelatedObject() Call to work with it, it sounds like it just attaches BAR to the context and then references it from FOO.

Check the following:

1. FOO is already attached to the context
2. relationship is the Name of the attribute/property on the object FOO representing the relationship. (i.e. The "One" end of the "One to Many relationship)
3. BAR is NOT already attached to the context (if it is, you should be using AddLink()).

Alternatively, tell me what happens when you call AddObject() and AddLink() seperately OR if instead of calling AddLink() you just add a relationship directly via ToEntityReference() which is what I usually do.

Matthew Hunter said...
This comment has been removed by the author.
seguros said...

Having problem updating an atribute to null: GetAttachedEntities() doesn't show null attribute and entity doesn't get updated.

Any idea how to fix?

Thanks

Matthew Hunter said...

Could it be that you have attached the entity at the point after you had already started modifying the entity?
(see notes on Attach() method above in blog post.), that's the only thing I can think of. If not, please provide some more details so we can get to the bottom of this.

Andrew Fleury said...

Hello,
I'm trying to get something to work but keep receiving errors.

In my scenario, I fetch an existing lead, create a new record who has a lookup to this Lead and use the AddLink to associate them. but also need to call UpdateObject on the lead because some values have changed. I've tried every single order imaginable but I receive the error: "The 'ol_websiterequest' entity must be in the default (null) or unchanged state."

So the order is
Get Lead, create new child record, update property of lead, call AddObject(childRecord), AddLink, UpdateObject(lead) and finally SaveChanges

Matthew Hunter said...

Hi Andrew,

When I built this, i didn't take the AddLink into consideration because I never use it.

When I related records I simply set the entity reference field, and don't use AddLink at all.

So in your scenario I would:

Grab the lead using linq
Create a newRecord (in memory)
Assign the newRecord a key (optional)
Set the new records Lookup field to lead.ToEntityReference() value
Use the linq context's AddObject(newRecord)
Call SaveChanges();

With AddLink you have to fill in the parameters, I think the above is much simpler - no need to identify the relationship name and all that.


Cheers

Matthew

Andrew Fleury said...

Hi Matthew,

Thanks for the fast reply. Your solution would work only when the Lead is existing. I'd still need to use AddLink in the case of a new lead (if I'm not mistaken)

In my scenario, sometimes the Lead is new, sometimes it's existing so it was easier for me to use the AddLinnk in both cases instead of calling AddLink when the Lead is new, but set the lookup field when the Lead is existing..

Is it feasible to fix this in the source code? Would it be possible to explain to me why it gives an error so it can give me a starting path to maybe tweak the code you've provided

Matthew Hunter said...

If the lead is not existing give it a key (leadid=Guid.NewGuid) then .AddObject(newLead), then follow instructions.

You may need to use new EntityReference instead of ToEntityReference but it will work as the Linq context will insert the lead before inserting the other entity referencing it.

Matthew Hunter said...

I will look @ updating the source code but won't be until later date as I'm extremely busy atm. Will post here if/when I update but don't wait for me.

Matthew Hunter said...
This comment has been removed by the author.
Unknown said...

Thank you, very useful stuff, works nicely with CRM 2016 On-Prem!

Julian said...

A lifesaver!

We have a complex piece of user provisioning logic that sits between CRM and several external services, and hit a nasty bug this week with a loop firing hundreds of emails to members of staff.

Root cause was CRM plugins firing because of the "whole entity" update effect. We were already sub-classing and injecting the CRM Context so adding your code was very fast, and stopped the bug in its tracks.

Thanks for taking the time to share your code.

Matthew Hunter said...

You are welcome!