This is the second article of three telling the tale of our own DevOps for Microsoft Dynamics 365, and the technology behind it.

Part I – Background and how our DevOps tools evolved before we knew about it
Part II – Automation of the build and deploy process using custom VSTS Build Tasks
Part III – Demo of complete build and release definitions taking you from A to Z In the previous article I described the background of our struggles with moving configuration data and later scripted export and import of solutions and the CRM Deployer tool. This article takes these features to the next level by packing those and some other missing features into custom VSTS Build Tasks.

Automating the Build Process

We now had the tools we needed to automate the central parts of the build/deploy process. But it still involved lots of manual or script based steps. To describe it simply, the following steps were required to produce a full deploy of a Customer Solution (CS) that has a prerequisite in one of our Product Solutions (PS), assuming we wanted the latest available code and customizations for both PS and CS.

1. Build PS
   1. Define new PS version number
   2. Set version number for assemblies
   3. Compile assemblies
   4. Use Plugin Registration to update PS DEV assemblies
   5. Minify webresources (scripted post build event)
   6. Use Web Resource Utility to update webresources in PS DEV
      (scripted post build event using modified version from SDK, that allows command line execution w/o user interaction)
   7. Run Shuffle scripts that export PS solutions and data from PS DEV
2. Build CS
   1. Run scripts that import PS to CS DEV
   2. Define new CS version number
   3. Set version number for assemblies
   4. Compile assemblies
   5. Use Plugin Registration to update CS DEV assemblies
   6. Minify webresources
   7. Use Web Resource Utility to update webresources in CS DEV
   8. Run Shuffle scripts that export CS solutions and data from CS DEV
3. Create package (by running a script)
   1. Collect all required definitions, solutions and data files from PS and CS exports
   2. Execute CRM Deployer with cdpkg file and a flag to create cdzip archive

The missing pieces

Some of the steps above would be possible to encapsulate a bit more with scripting or tailored tools, and some of them possible to perform using Microsoft’s Developer Toolkit, Jason Lattimer’s CRM Developer Extensions or Wael Hamze’s CI FrameWork. But with the legacy of our Shuffle and later the CRM Deployer, that both have well proven technology and still save us literally hundreds of hours every month, we decided to add the few missing pieces in our puzzle ourselves. Obvious missing pieces are the manual update of the assemblies and the dying technology of the Web Resource Utility, that started failing badly as we configured some of our dev environments for IFD. The Web Resource Utility also required connection and packages files defining target CRM (including credentials) and which files to upload, which was still a time consuming and error prone manual work.

Continuous Integration with VSTS

For quite some time we have been using VSTS CI Builds for verifying commits and code quality using the SonarQube integration with VSTS. But the artifacts of these builds were not used in any way, as we could not instruct VSTS to update a CRM environment with an assembly compiled on a build agent. This is one of the parts we were missing from the CI FrameWork and other tools we have looked at. Another missing feature was to minify and update webresources from the VSTS agent to CRM before exporting the solutions. After some googling and big thanks to the very nicely structured and open sourced CI Framwork by Wael we were able to find enough information to get us started with the custom VSTS Tasks described below.

VSTS Build Task: Update Assembly

The next step towards our automated build and deploy process was to create a custom VSTS Build Task that could use an assembly compiled by the build agent, extract the assembly info details, and match it with an existing plugin assembly registered in a CRM environment to update it’s contents. I knew a tool in XrmToolBox that does exactly this – the Plugin Auto Deployer. This tool basically “listens” for changes on a specific assembly on disk, and when it is updated it will update the corresponding assembly in CRM. Enjoying the fruits of open source, I looked at the code (which was actually created by a colleague of mine), tweaked it a bit to make the code a lot easier by using our DevUtils (supporting C# libraries for development against the CRM SDK) and inherited the cmdlet abstract class to make the function accessible from PowerShell. Then I had to go studying documentation and examples on creating VSTS Build Tasks. It seems like a powerful framework, but just as our earliest versions of the Shuffle, it sure could use some tools to make the process easier. But finally I got all the JSON manifests, folder structures and TFX-CLI commands working, and we had our first custom VSTS Build Task!

VSTS Build Task: Update WebResources

Now that our first Build Task was in place, the door was wide open to create more tasks. Next up was the Update WebResources task. This one was a bit more complex, as I somehow had to define which files to update. I decided to implement a wildcard pattern similar to that used in VSTS to e.g. specify which solutions to build. I added the possibility to exclude files too by adding an exclamation sign at the beginning of each pattern. This is a sample pattern specifying files to include and exclude from the webresources root folder in the project:

  scripts*.js
  images*.*
  contents*.html
  styles*.css
  !***.maxi.js
  !scriptstest.js

Other information needed is the (local) root folder for these files, and the prefix the files should have in CRM.
So how do you update webresources in CRM? I know someone who knows… as this is just what Tanguy Touzard’s Web Resource Manager does. So I had a look at his code, found how it identifies the webresource in CRM from the relative local path and the prefix, and realized that it is just a simple UpdateRequest on the webresource entity, updating the content attribute, that does all the magic.
Now we had overcome the two major parts we were missing from other tools that we considered adopting to, without really having to change the core of our delivery process.

VSTS Build Task: Minify JavaScripts

To minify our javascripts I turned to our web guys, who know painfully much more than I do about these things. We ended up with a tiny gulpfile that uses gulp-strip-debug and gulp-uglify. This was wrapped in a PowerShell file that installs the gulp features we needed, and then calls the gulpfile to minify all files matching a certain pattern in specified folder.
In our case, we traditionally write our code in “maxi files” that might be called contact.maxi.js which are saved as contact.js in their minified form.

VSTS Build Tasks: Do the Shuffle

Now all we needed to complete the full automated process was to Shuffle from VSTS Build Tasks.
As all Shuffle logic was extracted to separate libraries several years ago, it was quite easy to create the two build tasks Shuffle Export and Shuffle Import. These tasks are basically happy with just a path to the Shuffle Definition file and a connection string.

What’s next?

In the last article I will demonstrate a complete build process with VSTS Build using OOB tasks and our custom tasks. After that a release definition is created that picks up artifacts from several project builds to compose deployment packages for distribution and deploy directly from VSTS to target CRM environments.
Continue to Part III – Demo of complete build and release definitions taking you from A to Z