During my eight years as a Microsoft Dynamics CRM / 365 developer, I have felt a strong pain every time it was time to package and distribute a customer or product implementation.

Over the years our tools have evolved to now support a complete automated process; from source repository via compilation, updating dev environment, exporting solutions and configuration, collecting the artifacts and compose deployment packages to be installed manually or by VSTS release management.

This is the first article of three telling the tale of our own DevOps for Microsoft Dynamics 365, and the technology behind it.

Part I – Background and how our DevOps tools evolved before we knew about it
Part II – Automation of the build and deploy process using custom VSTS Build Tasks
Part III – Demo of complete build and release definitions taking you from A to Z   My hope is that these articles will inspire you to take your delivery process to the next stage by implementing automation through CI and what is usually called DevOps. If you are already there, these articles should provide an alternative solution, which may or may not suit your needs today, but might be worth considering.

It started with the data

Back in the days of CRM 4.0 we started delivering systems that were based more and more on generic configurable functionality. The benefit of having an automated way of delivering and moving configuration data between CRM environments was becoming increasingly obvious. This was long before utilities like the Configuration Migration Tool, so we started to draw the blueprints to develop the functionality we needed. Basically, what we needed was to grab a bunch of data from a source environment to persist it in a file, and later push it into another CRM organization.

Both the export and import had to support configuration beyond the “Export to Excel” functionality available. When exporting you need to be able to select which records to get and which attributes to include from each entity. And during import it must be possible to define how existing records should be identified if the GUID was not used for matching, and how to behave if an existing match exists. Being able to respect relationships in the data became a requirement, to only include child records for entity B that relate to exported records of entity A. Moving native N:N-relationship records was also a requirement. Obviously a GUID mapping mechanism must be built into the tool, to compensate for matching records not having the same GUIDs in source and targets. The data file should be “readable”, to be able to manipulate data between export and import, or to generate the entire data file manually or through an arbitrary external system.

The Shuffle is born

So I began developing the Cinteros Shuffle. The name might not give you the feeling of a surgically accurate export/import tool, but more like you are shuffling a deck of cards throwing data around by chance. The background is that the Swedish word for what you do with a shovel is “skyffla”, the same word as the English “shuffle”. And that is what this tool does – it acts as a shovel digging up data from one place, and putting it into another place.

Since the first 0.1 version of the Cinteros Shuffle, this has been the theme song that best describes me and my colleagues at Cinteros (now Innofactor). In the beginning Cinteros Shuffle was strictly a console application, where you either passed parameters to specify the Shuffle Definition file (configuration for the export or import to perform) and CRM connection information etc, or you simply entered it when prompted. Today the Shuffle functionality has been abstracted to a separate library that can be consumed by the console application, by the Shuffle Runner tool for XrmToolBox, or even by CRM plugins where the Shuffle can actually be used in specific integration scenarios.

Configuring the Shuffle

We chose the easy but powerful way to configure the Shuffle – an XML file that we call the Shuffle Definition. In the Shuffle Definition you specify blocks of entities to export and import, filter for which records to export, which attributes to include etc. And for the import you specify which attribute or attributes that is the key for the target records, to be able to configure more advanced matching than just using the primary key GUID. For each block you can also specify a relation to one or more other blocks, to limit the records being exported by parent record conditions. To be able to implement proper behavior for export and import of native many to many relationship records, for each block you can specify the type of the entity to shuffle, declaring if it is a normal entity or an native intersect entity. Import blocks could declare if existing records should be overwritten, untouched or even deleted. The untouched option might seem strange, but is really useful if you “know” matching records exist, and just want to identify them to perform GUID mapping for import of child records to work. There are a bunch of other settings to work with, and ways to use them to get the data moved just the way you want, but I will not go into more details at this point. Sample Shuffle Definition that shuffles Treeviews, their Nodes and Node Relations:

<ShuffleDefinition>
  <Blocks>
    <DataBlock Name="Treeview" Entity="cint_treeview" Type="Entity">
      <Export ActiveOnly="true">
        <Filter Attribute="cint_identifier" Operator="Like" Type="string" Value="MMS_%" />
        <Attributes>
          <Attribute Name="cint_*" />
        </Attributes>
      </Export>
      <Import CreateWithId="false" Save="CreateUpdate">
        <Match PreRetrieveAll="true">
          <Attribute Name="cint_identifier" />
        </Match>
      </Import>
    </DataBlock>
    <DataBlock Name="TreeviewNode" Entity="cint_treeviewnode" Type="Entity">
      <Export ActiveOnly="true">
        <Attributes>
          <Attribute Name="cint_*" />
        </Attributes>
      </Export>
      <Import CreateWithId="false" Save="CreateUpdate">
        <Match PreRetrieveAll="true">
          <Attribute Name="cint_treeview_id" />
          <Attribute Name="cint_name" />
        </Match>
      </Import>
      <Relation Block="Treeview" Attribute="cint_treeview_id" />
    </DataBlock>
    <DataBlock Name="TreeviewNodeRelation" Entity="cint_treeviewnoderelation" Type="Entity">
      <Export ActiveOnly="true">
        <Attributes>
          <Attribute Name="cint_*" />
        </Attributes>
      </Export>
      <Import CreateWithId="false" Save="CreateUpdate">
        <Match PreRetrieveAll="true">
          <Attribute Name="cint_parent_id" />
          <Attribute Name="cint_child_id" />
        </Match>
      </Import>
      <Relation Block="TreeviewNode" Attribute="cint_parent_id" />
      <Relation Block="TreeviewNode" Attribute="cint_child_id" />
    </DataBlock>
  </Blocks>
</ShuffleDefinition>

The Shuffle Definition is simply an XML file, and just like the SiteMap for CRM can be managed using the Sitemap Editor in XrmToolBox, it was a major improvement when we created the XrmToolBox plugin Shuffle Builder to compose your Shuffle Definitions. This has made the Shuffle available to any BA consultant, even though they might usually get intimidated by the mention of “XML”.

The Shuffle data file

Originally the result of the Shuffle export was simply an EntityCollection that was serialized using the DataContractSerializer. This format was not very efficient and did not quite fulfil the requirement that the data files should be readable as this produces a “bloated“ XML format.
Instead we implemented our own serialization style producing XML files with just the information necessary to be able to import the data efficiently. A few different types of serialization have been introduced, each suitable for different needs. Below is an extract from an exported data file using the most compact of our serialization styles.

<ShuffleData Type="Explicit" ExportTime="2017-04-20T10:27:40">
  <Block Name="Treeview" Count="7">
    <Entities>
      <cint_treeview id="83678d67-3cba-e511-80e0-000d3ab1faf7">
        <cint_description type="String">Visar en trädvy över (filtrerade) Ämnesområden och Frågeställningar</cint_description>
        <cint_name type="String">Ämnesområden</cint_name>
        <cint_identifier type="String">MMS_MUA_IncidentSteps</cint_identifier>
        <cint_duplicate_limit type="OptionSetValue">478880000</cint_duplicate_limit>
      </cint_treeview>
      <cint_treeview id="84678d67-3cba-e511-80e0-000d3ab1faf7">
        <cint_description type="String">Visar trädvy över (filtrerade) KB-artiklar.</cint_description>
        <cint_name type="String">KB-artiklar</cint_name>
        <cint_identifier type="String">MMS_MUA_IncidentKB</cint_identifier>
        <cint_duplicate_limit type="OptionSetValue">478880000</cint_duplicate_limit>
      </cint_treeview>
      ...

To make the Shuffle even more accessible to manual editing or production of Shuffle data files from other systems, one of the serialization methods use simple CSV files, with just some extra tags to separate the different data blocks.

<<<Treeview>>>
Entity;Id;cint_description;cint_name;cint_identifier;cint_duplicate_limit
cint_treeview;83678d67-3cba-e511-80e0-000d3ab1faf7;Visar en trädvy över (filtrerade) Ämnesområden och Frågeställningar;Ämnesområden;MMS_MUA_IncidentSteps;478880000
cint_treeview;84678d67-3cba-e511-80e0-000d3ab1faf7;Visar trädvy över (filtrerade) KB-artiklar.;KB-artiklar;MMS_MUA_IncidentKB;478880000
<<<TreeviewNode>>>
Entity;Id;cint_link_mode;cint_chk_relation_allow_many;cint_treeview_id;cint_duplicate_limit;cint_nodeid;cint_loadonexpand;cint_name;cint_sortorder;cint_fetchxml;cint_click_to_open;cint_nodetext;cint_isleaf;cint_isroot;cint_immediate_children;cint_autoexpand;cint_nodeicon;cint_description;cint_checkbox;cint_requirechildren;cint_chk_entity2_id;cint_chk_relation_to_entity1;cint_chk_relation_to_entity2;cint_chk_relation_entity;cint_chk_on_changed_function;cint_chk_relation_type;cint_chk_entity1;cint_chk_entity2;cint_chk_is_checked;cint_chk_entity1_id
cint_treeviewnode;8a678d67-3cba-e511-80e0-000d3ab1faf7;478880000;False;cint_treeview:83678d67-3cba-e511-80e0-000d3ab1faf7;478880000;{cint_mms_mua_subject_areaid};False;Ämnesområden;10;"<fetch mapping=""logical"" count=""-1"" version=""1.0"">,</fetch>";False;{cint_name};False;True;False;True;cint_mua_/images/cint_mms_mua_subject_area.16.png;Visar Ämnesområden filtrerade på Ärendetyp och Ämnesomreådes sökord (om specificerat);False;True;<null>;<null>;<null>;<null>;<null>;<null>;<null>;<null>;<null>;<null>
cint_treeviewnode;8b678d67-3cba-e511-80e0-000d3ab1faf7;478880000;False;cint_treeview:83678d67-3cba-e511-80e0-000d3ab1faf7;478880000;{cint_mms_mua_questionid};False;Frågeställningar;20;"<fetch mapping=""logical"" distinct=""true"" version=""1.0"">,</fetch>";False;{cint_name};True;False;False;True;<null>;Visar Frågeställningar filtrerade på Ärendetyp och Ämnesomreådes sökord (om specificerat);True;False;{rootid};cint_mms_mua_question_id;cint_incident_id;cint_mms_mua_incident_question;incidentQuestionTreeChanged;478880000;cint_mms_mua_question;incident;{iq_questionid};{cint_mms_mua_questionid}

Regardless of serialization method used, the Shuffle works the same way with GUID mappings, matching during import etc.

The introduction of solutions

With CRM 2011 the concept of solutions was introduced. Pretty soon we realized that to be able to perform our deployments in a semi-automated way, we could expand the Shuffle to be able to export and import solutions too.
The Shuffle Definition schema was expanded to be able to define if the blocks were DataBlocks or SolutionBlocks. For solution blocks, a similar but slightly different set of sub nodes were defined. Instead of specifying the name of an entity, you specify the name of a solution. Instead of matching attributes and import behavior, you specify overwrite behavior based on versions of solution existing in target environment and solution to import.
And then a few more options to declare if customizations should be published before export and after import, if plugin steps should be activated and so on.

<ShuffleDefinition>
  <Blocks>
    <SolutionBlock Name="MMS_MUA_WEB">
      <Export Type="Both" SetVersion="{ShuffleVar:version}" PublishBeforeExport="true" TargetVersion="{ShuffleVar:target}" />
      <Import Type="Managed" OverwriteCustomizations="true" ActivateServersideCode="false" PublishAll="false" OverwriteNewerVersion="true" OverwriteSameVersion="false">
        <PreRequisites>
          <Solution Name="CinterosUtils" Comparer="ge" Version="15.*" />
        </PreRequisites>
      </Import>
    </SolutionBlock>
    <SolutionBlock Name="MMS_MUA_MODEL">
      <Export Type="Both" SetVersion="{ShuffleVar:version}" PublishBeforeExport="false" TargetVersion="{ShuffleVar:target}" />
      <Import Type="Managed" OverwriteCustomizations="true" ActivateServersideCode="false" PublishAll="false" OverwriteNewerVersion="true" OverwriteSameVersion="false">
        <PreRequisites>
          <Solution Name="MMS_MUA_WEB" Comparer="any" />
        </PreRequisites>
      </Import>
    </SolutionBlock>
  </Blocks>
</ShuffleDefinition>

The ShuffleDefinition.xsd schema file can be investigated by the curious 🙂

Even though a few changes have been made since the first versions, where the introduction of asynchronous import of solutions provided us with improvements for feedback during long running imports, the base of the Cinteros Shuffle is still the same today. The foundation it was built upon has proven to be stable and easy to extend, and the Shuffle is still a vital part of our build and deployment process.

Composing packages

As our deliveries usually consist of several solutions and configuration data sets, we saw the need for tools to deploy multiple solutions and data files as a “package”. Having the Shuffle that automated each individual part, we composed deployment packages in the form of a couple of Shuffle Definitions with the associated solution and data files, and then a classic old “bat file” (or PowerShell, indeed) to make the necessary calls to our Shuffle.exe console application.

BAT files are not enough

That solution helped us with our own immediate need. But many of our customers had hosting partners that did not always want to grant any other vendor access their production environments. They requested routines and packaging that they could maintain without having any detailed knowledge about Dynamics CRM. Delivering a zip file that was to be unpacked and then running a bat file in a DOS window, being prompted for connection details and login credentials was no longer an option.

Solution: The Cinteros CRM Deployer

Instead we used the core Shuffle functionality, and wrapped everything in another layer specifying deployment packages. Yes, we created our own Package Deployer. Why? Well, simply because the one from Microsoft didn’t exist. We built a quite nice UI, with progress bars and possibility to select which parts of the package to deploy, detailed logging, etc.

Installation and packaging

The Cinteros CRM Deployer is delivered as a MSI package for easy installation. The Package Definition it uses is also a simple XML file that is saved with a specific file suffix “.cdpkg” which is associated with the CRM Deployer during installation. The application also has a feature to pack the Package Definition, all required Shuffle Definitions, and all associated solution and data files to a zip archive, which will get the suffix “.cdzip”, also associated with the CRM Deployer.

Connecting to CRM

To handle connections to target CRM environments, we used a spinoff from XrmToolBox – the MscrmTools.Connection package on NuGet. Using this package, you handle your connections just the same way that is done in XrmToolBox. Well indeed, except being a spinoff, XrmToolBox itself now uses the same package. The development of the CRM Deployer has actually led to a few new features in the connection package, such as being able to specify path to the connection file.

Deliverables

This way, we can deliver a single CDZIP archive that is executed by just double-clicking it. The CRM Deployer automatically unpacks it, reads the Package Definition, and can then deploy selected components to target CRM environment. The delivered package does not contain anything but the actual contents of the system to be deployed. The CRM Deployer is only installed once, and can then handle all packages without distributing more executables to the customers or hosting partners.

What’s next?

In the next article I will describe how we identified what we were missing to be able to automate the build and release process, and how we solved the last obstacles by creating our own extensions for VSTS with custom build and release tasks.
Continue to Part II – Automation of the build and deploy process using custom VSTS Build Tasks