LINQ to CRM 2011 - Attribute Level Change Tracking Support

I really like LINQ to CRM 2011, it makes development against CRM easier.  However, out of the box LINQ to CRM has one big issue - it only supports entity level change tracking and not attribute level tracking.

This means that whenever you retrieve something via LINQ, mark the entity as "changed" and submit the changes, whether or not you have changed any attributes - it will update ALL of the attributes for that entity in the database - because it does not distinguish which attributes have changed and which haven't before it makes the update.

There are bad some side effects of this:
- The new Auditing feature in CRM 2011 will record all the fields as changed, not just the ones you actually changed.  This not only takes up database space, but makes it harder to read.
- Plugins and Workflows will trigger unnecessarily if you have them filtered on particular fields which you did not actually change.
- You may inadvertently overwrite changes that have occurred since you retrieved the record that you did not mean to.
- If you have field level security on some of those attributes, they may cause an exception even though you never changed it's value.

Luckily, LINQ to CRM allows you to intercept some useful events in it's life-cycle (Attach, SaveChanges, Execute) through the use of a sub-class or partial class, replacing it with your own functionality. Because of this, I've managed to write a class to overcome these issues and commit only the actual attribute changes to CRM rather than ALL of attributes.

So How does this class work?

- Intercepts when an entity is "Attached" or "Detached" from the context.  If the entity has a status of "unchanged" a shallow clone is taken of the entity and stored in an "originals" list; the clone is removed from the "originals" list if it is later detached.  Note: Entities will have a status of "unchanged" if they have been retrieved via LINQ from CRM or manually attached using the Attach() method.

- Intercepts the call to "SaveChanges()" method, inspects each entity that has been flagged as "changed" and compares it against the original version of the entity.  If a change in attributes is detected then a new entity is created containing only the changed attributes and the entity's key; this new entity is put into a new "Deltas" collection.  In addition, any entities that have not changed at all but are marked as "changed" are reset back to "unchanged", so they are not included in the commits that are made against CRM.

- Intercepts when the Execute Method is called to commit the data.  When the data is committed to the database, it uses the "UpdateRequest" class as you normally would using non-LINQ to CRM Entities.  When this is done for entity in the list of entities marked as "changed", the Target property of the UpdateRequest object used is reset to use the delta version of the entity rather than the complete entity itself.  This therefore avoids committing all attributes to the database, and instead only sends those fields that have changed.

Can I use it in a Plugin?

Yes. You should even be able to use it in the online version of CRM.

How do I use it?

- Place it in the same project as your LINQ to CRM context.
- Change the namespace so that it macthes the same namespace as your LINQ to CRM Context.
- Change the class name so that it macthes the same class name as your LINQ to CRM Context.
- If you are doing read-only operations, to remove the change tracking overhead you can disable the functionality with the "EnableAttributeChangeTracking" flag - set it to false as soon as you new up your context.

Any Gotya's?

- Attach(): If you are going to new up an Entity for an update without retrieving the entity from CRM first, make sure you set the key and then Attach() the entity before you change the attributes on that entity.  Otherwise it will not track the changes.

- MergeOption: If you change the merge option, it may not work.  AppendOnly (the default) is currently the only supported merge option.

- EntityCollection: Currently, the EntityCollection attribute type is not supported when identifying changes between the original and the current version of an entity.  EntityCollection attributes are always assumed to be "Changed" if included in an attribute, and will always be submitted for update.

Cheers

Matt

                                                                     
                                                                     
                                                                                                                
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using Microsoft.Xrm.Sdk;
using Microsoft.Xrm.Sdk.Messages;

namespace Example.Linq
{
    public partial class CrmContext
    {
        private List<Entity> _originalEntities = new List<Entity>();
        private List<Entity> _deltaEntities = new List<Entity>();

        private bool _enableAttributeChangeTracking = true;

        /// <summary>
        /// Enables or disables Attribute Level Change tracking.
        /// Default is On.
        /// </summary>
        public bool EnableAttributeChangeTracking
        {
            get
            {
                return _enableAttributeChangeTracking;
            }

            set
            {
                if (_originalEntities.Count > 0 && value == false)
                    throw new Exception("You cannot disable attribute change tracking at this time, entities are already being tracked");

                _enableAttributeChangeTracking = value;
            }
        }

        /// <summary>
        /// Overrides the base OnBeginEntityTracking method
        /// When an entity is tracked, this adds the original unmodified version of that entity
        /// to the unchangedEntities list (which contains the original version of every entity).
        /// </summary>
        /// <param name="entity"></param>
        protected override void OnBeginEntityTracking(Entity entity)
        {
            base.OnBeginEntityTracking(entity);

            if (_enableAttributeChangeTracking && entity.EntityState == EntityState.Unchanged)
            {
                var exists = _originalEntities.Where(x => x.Id == entity.Id).FirstOrDefault();
                if (exists == null) _originalEntities.Add(ShallowClone(entity));
            }
        }

        /// <summary>
        /// Overrides the base OnEndEntityTracking method
        /// For any entity that is detached from entity tracking, this also 
        /// removes it from the unchangedEntities list (which contains the original version of every entity)
        /// </summary>
        /// <param name="entity"></param>
        protected override void OnEndEntityTracking(Entity entity)
        {
            base.OnEndEntityTracking(entity);
            if (_enableAttributeChangeTracking)
            {
                var exists1 = _originalEntities.Where(x => x.Id == entity.Id).FirstOrDefault();
                if (exists1 != null) _originalEntities.Remove(exists1);
                var exists2 = _deltaEntities.Where(x => x.Id == entity.Id).FirstOrDefault();
                if (exists2 != null) _deltaEntities.Remove(exists2);
            }
        }

        /// <summary>
        /// Overrides the base OnExecuting Method
        /// For UpdateRequests, replaces the target entity on the update message
        /// with an entity that only contains the key and the changed attributes.
        /// </summary>
        /// <param name="request"></param>
        protected override void OnExecuting(OrganizationRequest request)
        {
            if (_enableAttributeChangeTracking && typeof(UpdateRequest) == request.GetType())
            {
                var updateRequest = (UpdateRequest)request;
                var target = updateRequest.Target;
                var newTarget = _deltaEntities.Where(x => x.Id == target.Id).FirstOrDefault();
                updateRequest.Target = newTarget;
            }
            base.OnExecuting(request);

        }

        /// <summary>
        /// Overrides the base OnSavingChangesMethod
        /// For each entity, determines if an update is required.
        /// If no update is required, it detaches and reattaches to set the entity back to a "unchanged" state.
        /// Also create the entity that will ACTUALLY be subitted to the database (used in the OnExecuting method).
        /// </summary>
        /// <param name="options"></param>
        protected override void OnSavingChanges(Microsoft.Xrm.Sdk.Client.SaveChangesOptions options)
        {

            // Clear the list of entities to be sumbitted
            _deltaEntities.Clear();

            // Mark any entities as unchanged that
            // are only sending the key
            var updated = this.GetAttachedEntities().Where(x => x.EntityState == EntityState.Changed).ToList();

            foreach (var target in updated)
            {
                // Ignore updates where nothing has been updated
                var unchanged = _originalEntities.Where(x => x.Id == target.Id).FirstOrDefault();
                if (unchanged != null)
                {
                    var cloneOfTarget = ShallowClone(target);
                    RemoveUnchangedFields(cloneOfTarget, unchanged);
                    _deltaEntities.Add(cloneOfTarget);

                    // Test to see if it's only the key left... if so ignore the update otherwise you will get a blank audit record
                    if (cloneOfTarget.Attributes.Count == 1 && cloneOfTarget.Attributes.First().Value != null)
                    {
                        if (cloneOfTarget.Attributes.First().Value is Guid || cloneOfTarget.Attributes.First().Value is Guid?)
                        {
                            if (cloneOfTarget.Id == (Guid?)cloneOfTarget.Attributes.First().Value)
                            {
                                this.Detach(target);
                                _deltaEntities.Remove(cloneOfTarget);
                                target.EntityState = EntityState.Unchanged;
                                this.Attach(target);

                            }
                        }
                    }
                }
            }

            base.OnSavingChanges(options);
        }

        protected override void OnSaveChanges(SaveChangesResultCollection results)
        {
            _deltaEntities.Clear();
            base.OnSaveChanges(results);
        }

        /// <summary>
        /// Overrides the base OnExecute Method
        /// For Update Requests, if an error was thrown ignores it if Target is null (i.e. no need to update)
        /// </summary>
        /// <param name="request"></param>
        /// <param name="exception"></param>
        protected override void OnExecute(OrganizationRequest request, Exception exception)
        {
            ///
            if (_enableAttributeChangeTracking && typeof(UpdateRequest) == request.GetType())
            {
                var updateRequest = (UpdateRequest)request;

                if (updateRequest.Target != null)
                {
                    base.OnExecute(request, exception);
                }
            }
            else
            {
                base.OnExecute(request, exception);
            }
        }


        /// <summary>
        /// This method loops through comparing the changed entity with the unchanged entity
        /// and removed the unchanged fields from the changed entity.  Keys are kept.
        /// </summary>
        /// <param name="changed"></param>
        /// <param name="unchanged"></param>
        public void RemoveUnchangedFields(Entity changed, Entity unchanged)
        {

            // Lookp through the changed fields, if there is one missing from the unchanged list, add it as null so at least it will be compared.
            foreach (var changedAttribute in changed.Attributes)
            {
                var key = changedAttribute.Key;

                if (!unchanged.Contains(key))
                    unchanged.Attributes.Add(new KeyValuePair<string, object>(key, null));
            }

            // Loop through each attribute and compare properties
            foreach (var unchangedAttribute in unchanged.Attributes)
            {
                var key = unchangedAttribute.Key;

                var originalAttributeVal = unchanged.Attributes[key];
                var changedAttributeVal = changed.Attributes[key];

                // Fix any original value issues
                if (originalAttributeVal != null)
                {
                    // Fix any zero length strings
                    if (originalAttributeVal.GetType() == typeof(string) && string.IsNullOrWhiteSpace((string)originalAttributeVal))
                    {
                        originalAttributeVal = null;
                    }
                    else
                    {
                        // Fix any dates that aren't of a specified kind
                        if (originalAttributeVal.GetType() == typeof(DateTime) || originalAttributeVal.GetType() == typeof(DateTime?))
                        {
                            var date = (DateTime?)originalAttributeVal;
                            if (date.Value.Kind == DateTimeKind.Unspecified)
                                originalAttributeVal = date.Value.ToUniversalTime();
                        }
                    }
                }

                // Fix any changed value issues
                if (changedAttributeVal != null)
                {
                    // Fix any zero length strings
                    if (changedAttributeVal.GetType() == typeof(string) && string.IsNullOrWhiteSpace((string)changedAttributeVal))
                    {
                        changedAttributeVal = null;
                    }
                    else
                    {
                        // Fix any dates that aren't of a specified kind
                        if (changedAttributeVal.GetType() == typeof(DateTime) || changedAttributeVal.GetType() == typeof(DateTime?))
                        {
                            var date = (DateTime?)changedAttributeVal;
                            if (date.Value.Kind == DateTimeKind.Unspecified)
                                changedAttributeVal = date.Value.ToUniversalTime();
                        }
                    }
                }

                // Compare the values
                if (originalAttributeVal == null && changedAttributeVal == null)
                {
                    changed.Attributes.Remove(key);
                }
                else if (originalAttributeVal != null && changedAttributeVal == null)
                {
                    // Do nothing, keep the attribute        
                }
                else if (changedAttributeVal is EntityCollection)
                {
                    // Do nothing, Leave it in... (complicated to compare).
                }
                else if (changedAttributeVal.Equals(originalAttributeVal))
                {
                    if (unchangedAttribute.Value.GetType() == typeof(Guid) || unchangedAttribute.Value.GetType() == typeof(Guid?))
                    {
                        // Avoid removing the key
                        if (!((Guid?)unchangedAttribute.Value == unchanged.Id))
                        {
                            changed.Attributes.Remove(key);
                        }
                    }
                    else
                    {
                        changed.Attributes.Remove(key);
                    }
                }
            }
        }

        /// <summary>
        /// Take a shallow copy of an entity
        /// </summary>
        /// <param name="entity"></param>
        /// <returns></returns>
        private static Entity ShallowClone(Entity entity)
        {
            var clone = new Entity(entity.LogicalName);
            clone.EntityState = entity.EntityState;
            clone.Id = entity.Id;

            foreach (var attribute in entity.Attributes)
            {
                if (attribute.Value == null)
                {
                    clone.Attributes.Add(attribute.Key, null);
                }
                else if (attribute.Value is EntityReference)
                {
                    EntityReference entityReference = attribute.Value as EntityReference;
                    clone.Attributes.Add(attribute.Key, new EntityReference(entityReference.LogicalName, entityReference.Id));
                }
                else if (attribute.Value is Money)
                {
                    Money money = attribute.Value as Money;
                    clone.Attributes.Add(attribute.Key, new Money(money.Value));
                }
                else if (attribute.Value is OptionSetValue)
                {
                    OptionSetValue option = attribute.Value as OptionSetValue;
                    clone.Attributes.Add(attribute.Key, new OptionSetValue(option.Value));
                }
                else if (attribute.Value is EntityCollection)
                {
                    // Don't copy EntityCollection values, just re-reference it.
                    EntityCollection entityCollection = attribute.Value as EntityCollection;
                    clone.Attributes.Add(attribute.Key, entityCollection);
                }
                else // value types
                {
                    clone.Attributes.Add(attribute.Key, attribute.Value);
                }
            }

            return clone;
        }

    }
}

15/09/2012 Update: If you are using the entities generated by the CRM 2011 Developer Toolkit rather than the CrmSvcUtil.exe, it doesn't generate a context class for some reason - so, in order to use the partial class above, you'll just need to create a simple class for the Context similar to the code below and use the CrmContext.CreateQuery<Entity>() syntax instead of the CrmContext.EntitySet when querying.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.Xrm.Sdk.Client;

namespace Example.Linq
{
    public partial class CrmContext : OrganizationServiceContext
    {        
        /// <summary>
        /// Constructor.
        /// </summary>
        public CrmContext(Microsoft.Xrm.Sdk.IOrganizationService service) : 
                base(service)
        {
        }
        
    }
}