Objects ignored by Code Upgrade Tool

When testing my new rule for code upgrade tool, I found that certain objects are skipped and rules are not checked for them. That makes the tool significantly less useful, because you can’t be sure that it found everything.

This blog post explains why it happens, but unfortunately doesn’t provide any reasonable workaround, except making the problem more visible. You should be simply aware of it until Microsoft provides a fix. (Please let me know if there already is a fix that I missed.)

I created a rule to check for a certain code pattern and I added the pattern to a few objects for testing. Although the rule seemed to work, some objects were not included (SalesLine table, for instance). Soon I discovered that it happened due to an exception thrown when checking rules for these objects. You can see the exception in the debugger, but it’s not displayed anywhere in the UI, unless you make the following one-lin change in SysUpgradeRuleRunner.processUtilElement():

if(Microsoft.Dynamics.AX.Framework.Tools.CodeUpgradeTool.Parser.Severity::Fatal == xppParserSeverity)
{
    error = diagnosticItem.ToString();
    error(error); // new code to display the error
    continue;
}

If you use the Code Upgrade Tool, I would recommend to add this modification to your application. It will at least let you know that an object was skipped.

This is the error message I got:

An error happened while executing PipelineTypeResolverPipelineEntryCould not load file or assembly ‘Microsoft.Dynamics.Retail.StoreConnect.TransAutomClient, Version=0.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35′ or one of its dependencies. The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)

What’s going on? First let me explain what happens when you run the Code Upgrade Tool:

  1. It finds all code in the current layer
  2. Each application object (such as a class) is loaded for analysis. Note that the whole object is loaded, not just the code in the current layer.
  3. In some cases, especially when a .NET type is used in code, AX loads all assemblies listed under AOT > References (so it can later look for types in these assemblies). It’s normally done just once and cached for subsequent calls.

The problem is that one of these references can’t ever be loaded. It’s the reference called TransAutomClient_x64, which refers to assembly Microsoft.Dynamics.Retail.StoreConnect.TransAutomClient for processor architecture AMD64. Dynamics AX client is a 32-bit process and it can’t load this assembly, therefore such as attempt must fail. I believe that Code Upgrade Tool would work without any problem if this reference was removed from AOT (but I can’t prove it, because it requires deleting a SYS-layer object).

AX tries to load .NET assemblies in several cases. For instance, if a .NET type is used in variable declaration or as method return type. It also happens when a static method is called on a .NET type, or when AX suspects it might be such a call. I noticed that even calls to table map methods (such as inventItemPrice.InventPriceMap::pcsPrice()) triggers loading of assemblies, because the syntax is exactly the same as for .NET calls (compare with System.Environment::GetLogicalDrives(), for example).

What can be done about it? I don’t think we can do anything by ourselves. What Microsoft should do is simply skipping assemblies that can’t be loaded. The process might fail later, if some code actually uses types defined in that assembly, but that’s inevitable. What happens now is much worse – it fails even if the library isn’t used by any code at all. There is still a potential issue that the tool loads assemblies even if no .NET types are needed (because of table maps calls, as above), but it wouldn’t really cause any harm if assembly loading behaved reasonably.

I simultaneously logged this issue on Connect (link). I really hope that Microsoft will address it soon, because it significantly affects usefulness of this great tool.

Custom rules for Code Upgrade Tool

The Code Upgrade Tool allows you to detect patterns in X++ code and even to automatically change code. It’s especially useful for code upgrade, because you can easily upgrade legacy code. For example, CompanyInfo.CurrencyInfo field was replaced by CompanyInfo::standardCurrency() method in AX 2012. To upgrade your code, you would have to find and fix all occurrences, which would be tedious. Fortunately the Code Upgrade Tools knows how to detect this pattern and how to fix the code for you. And this is still an easy case – you would at least find the problem anyway, because your original code wouldn’t compile. In some other cases, code is still compilable in the new version, but it’s wrong for some reason and you need something more clever to identify it. Also note that the Code Upgrade Tool doesn’t try to fix all the issues – in some cases, it only reports them and it’s up to you to decide how the code should be upgraded.

The tool was first introduced as a separate download for AX 2012 RTM and integrated to the standard installation in version R2. The original name was Code Upgrade Tool, it’s called Code Upgrade Service on InformationSource and it doesn’t seem to have any separate name since integration to AX. I keep calling it Code Upgrade Tool and I recommend the same to you, because that’s how related assemblies and namespaces are named anyway.

I’m using AX 2012 R3 when writing this article, but it all works the same in AX 2012 R2, as far as I know.

To use the tool, first go to the developer workspace and open Tools > Code upgrade > Configure rules for code upgrade. If you don’t see any rules, click Manage code upgrade rules and then Browse. Choose Microsoft.Dynamics.AX.Framework.Tools.CodeUpgradeTool.Rules.dll file from Dynamics AX client bin folder (typically c:\Program Files (x86)\Microsoft Dynamics AX\60\Client\Bin\) and press Import rules. This imports rules prepared by Microsoft for code upgrade to AX 2012. You can also configure which rules should be used.

RuleConfig

When running code conflict detection (Tools > Code upgrade > Detect code upgrade conflicts), you can detect issues by ticking Create code upgrade patterns project checkbox and run automatic fixes by Auto-resolve code patterns.

DetectConflicts

The cool thing is that you can define your own rules and nothing prevents you from using them for other things than just code upgrade. You can use them for additional static analysis of your code, such as looking for dangerous code patters, and I’m sure it could be used for many different things if enough creative people played with it.

I’m going to show you how to write a simple rule. We found that a developer of our current codebase sometimes updated fields in InventDim table (such as InventLocationId) directly, i.e. not by creating a new InventDimId (or finding existing one). That’s very wrong and we want to be 100%-sure that not a single update like this remains in the application. Calls to update() method can be easily found by cross-references and I hoped to find references to doUpdate() in cross-reference tables as well, but unfortunately they’re not there. Therefore I decided to write a simple rule for Code Upgrade Tool to detect all such calls.

You can find some documentation for writing custom rules (Code Upgrade Tool User Guide [AX 2012]) and even code for rules prepared by Microsoft (downloadable from InformationSource), which will give you plenty of useful examples. The problem is that all this is for the beta version released for AX 2012 RTM and it won’t work with later versions. But let’s take a look anyway, because it’s still a good starting point.

Log on to InformationSource, switch to Services and download the Code Upgrade Service. Extract the archive and ignore everything except CodeUpgradeTool.Rules.zip and CodeUpgradeTool.Rules.Partner.zip. The first file contains code for standard rules implemented by Microsoft (don’t forget – it’s how it looked in the beta version) and CodeUpgradeTool.Rules.Partner.zip contains a starting project for new rules. Unpack CodeUpgradeTool.Rules.Partner.zip, remove write protection from files and open the Visual Studio project inside. By the way, I used Visual Studio 2013 to build the following example.

The project contains two classes: TestRuleSweeper (for rules that only looks for patterns) and TestRuleMutator (for rules that can actually fix identified patterns). I doubt I could write a completely generic fix for wrong updates of InventDim, therefore I’ll create a sweeper (and delete the mutator). This is TestRuleSweeper.cs:

namespace Microsoft.Dynamics.AX.Tools.CodeUpgradeTool.Rules.Partner.PartnerRules
{
    using System;
    using System.Collections.Generic;
    using System.Globalization;
    using System.Linq;
    using Microsoft.Dynamics.AX.Framework.Xlnt.XppParser;
    using Microsoft.Dynamics.AX.Framework.Xlnt.XppParser.Pass2;
 
    [Author("")] 
    [Description("PartnerRuleSweeperTest")]
    [AutomaticallyFixable("No")]
    [Category("Test")]
    [Version("2009 -> 2012")]
    public class TestRuleSweeper : PipelineableSweeper
    {
        public TestRuleSweeper(IDiagnosticSink diagnostics, IMetadataProvider metadataProvider, IDictionary<string, string> parameters)
          : base(diagnostics, metadataProvider, parameters)
        {         
        }
 
        ////TODO: override the method from ASTSweeper that fits the rule
    }
}

We can go and override an appropriate “visit” method, such as VisitQualifiedCall():

protected override object VisitQualifiedCall(object payload, QualifiedCall qualifiedCall)
{
    return base.VisitQualifiedCall(payload, qualifiedCall);
}

Let’s not continue with this implementation, because it wouldn’t work anyway. Instead, we’ll update the project to work with the new version.

First of all, remove these assemblies from project references:

  • Microsoft.Dynamics.AX.Framework.Xlnt.XppParser
  • Microsoft.Dynamics.AX.Framework.Xlnt.XppParser.Pass2

and add a few new ones:

  • Microsoft.Dynamics.AX.Framework.Tools.CodeUpgradeTool.Parser
  • Microsoft.Dynamics.AX.Framework.Tools.CodeUpgradeTool.Parser.Pass2
  • Microsoft.Dynamics.AX.Framework.Tools.CodeUpgradeTool.Rules

You’ll find them in AX client bin folder (such as c:\Program Files (x86)\Microsoft Dynamics AX\60\Client\Bin\).

References

Now a few things stop working, because they’ve changed in newer versions. We have to update them.

Firstly, remove old using statements and use new namespaces instead:

using Microsoft.Dynamics.AX.Framework.Tools.CodeUpgradeTool.Parser;
using Microsoft.Dynamics.AX.Framework.Tools.CodeUpgradeTool.Parser.Pass2;
using Microsoft.Dynamics.AX.Framework.Tools.CodeUpgradeTool.Rules.ObjectModel;

Then throw away old attributes, such as [Description], and replace them with new ones, such as [RuleDescription] (you’ll find an example below).

The next thing you’ll notice is a compilation error related to the base class, PipelineableSweeper. Ignore the message about generic arguments; the fact is that even sweepers now extends the PipelineableMutator class. Change the class accordingly.

This is how the TestRuleSweeper template should look like these days:

namespace Microsoft.Dynamics.AX.Framework.Tools.CodeUpgradeTool.Rules.Partner.PartnerRules
{
    using System.Collections.Generic;
    using Microsoft.Dynamics.AX.Framework.Tools.CodeUpgradeTool.Parser;
    using Microsoft.Dynamics.AX.Framework.Tools.CodeUpgradeTool.Parser.Pass2;
    using Microsoft.Dynamics.AX.Framework.Tools.CodeUpgradeTool.Rules.ObjectModel;
 
    [RuleCategory("")]
    [RuleTargetVersion("")]
    [RuleAuthor("")]
    [RuleDescription("")]
    [RuleType(AutomaticallyFixable.No)]
    public class TestRuleSweeper : PipelineableMutator
    {
        public TestRuleSweeper(IDiagnosticSink diagnostics, IMetadataProvider metadataProvider, IDictionary<string, string> parameters)
          : base(diagnostics, metadataProvider, parameters)
        {       
        }
    }
}

If you copy VisitQualifiedCall() from our previous implementation and compile the solution, you’ll get another error saying that VisitQualifiedCall() must returns Evaluation. Simply change the return type from object to Evaluation.

The remaining thing is to actually implement VisitQualifiedCall(). It’s also time to rename the class and fill values to the attributes. You may also want to change the namespace and the assembly name.

This is my complete class:

namespace Microsoft.Dynamics.AX.Framework.Tools.CodeUpgradeTool.Rules.Goshoom
{
    using System.Collections.Generic;
    using Microsoft.Dynamics.AX.Framework.Tools.CodeUpgradeTool.Parser;
    using Microsoft.Dynamics.AX.Framework.Tools.CodeUpgradeTool.Parser.Pass2;
    using Microsoft.Dynamics.AX.Framework.Tools.CodeUpgradeTool.Rules.ObjectModel;
 
    [RuleCategory("Inventory")]
    [RuleTargetVersion("2012")]
    [RuleAuthor("Martin Dráb")]
    [RuleDescription("InventDim.update() / doUpdate() shoukd not be updated by directly. InventDim::findOrCreate() should be used instead.")]
    [RuleType(AutomaticallyFixable.No)]
    public class InventDimUpdateSweeper : PipelineableMutator
    {
        public InventDimUpdateSweeper(IDiagnosticSink diagnostics, IMetadataProvider metadataProvider, IDictionary<string, string> parameters)
          : base(diagnostics, metadataProvider, parameters)
        {       
        }
 
        protected override Evaluation VisitQualifiedCall(object payload, QualifiedCall qualifiedCall)
        {
            Guard.ValidateIsNotNull(qualifiedCall, @"qualifiedCall");
 
            base.VisitQualifiedCall(payload, qualifiedCall);
            var qualifierType = qualifiedCall.Qualifier.GetExpressionType();
 
            if ((qualifiedCall.MethodName.Compare("doUpdate") || qualifiedCall.MethodName.Compare("update"))
                && (qualifierType != null && qualifierType.ToString().Compare("InventDim")))
            {
                this.AddDiagnosticWarning(1, qualifiedCall.Position, "Do not update InventDim table directly. Use InventDim::findOrCreate() instead.");
            }
 
            return qualifiedCall;
        }
    }
}

I don’t want to spend much time exploring the implementation of VisitQualifiedCall(), just notice how easy is to check whether the type of variable is InventDim. It’s because the compiler understands the code and makes this information available to Code Upgrade Tool as well.

It’s time to deploy our new rule. Build the solution, right-click the project and choose Open Folder in File Explorer. Open Debug\bin or Release\bin (depending on which configuration you used), find the DLL file with your rule and copy it to AX client bin folder.

Assembly

Now you can load the rule in exactly the same way as how you loaded Microsoft rules (return to the top of this post if you’re not sure how). Metadata such as author and description are copied from attributes of the class:

And you’re done! Now simply run conflict detection with Create code upgrade patterns project ticked let AX to do the rest for you.

UpgradeProject

One disadvantage of my implementation is that you won’t see actual place in code where to find the pattern. Microsoft rules place TODO comments to code to make it obvious and they have a nice utility class for it, unfortunately it’s an internal class and therefore it would take more effort to achieve the same thing.

Although this post was mainly about infrastructure and not about all the fancy rules you could write, I hope you see the potential by yourself. It may be really useful, whether you want to automatically change code during upgrade, find some patterns in your application or maybe something completely different.

Unfortunately I also have one bad news. When testing my new rule, I found it’s ignored in quite a few cases. You can find my analysis and some suggestions in the follow-up blog post: Objects ignored by Code Upgrade Tool.

Curah!

Have you already heard about Curah!? It’s a web site where you can group, comment and share links to useful resources about a technical issues. Curations may be useful for their authors (so they can find related resources on a single place), but they will be especially helpful for those who are new to a particular topic.

If you want to know more, look at Curah! FAQ and, for something more AX-oriented, AX Content: Curating Microsoft Dynamics AX Resources.

It’s also helpful to see some examples. This is from Microsoft: Microsoft Dynamics AX: Companion apps. And this is my humble contribution so far: Microsoft Dynamics AX 2012: AIF and Web Services.

Delete order line through AIF

The following code sample shows how to delete a line of an existing sales order through AIF from outside Dynamics AX 2012 . It uses a partial update, which means that we don’t have to send the whole document back to AX. As the documentation says, we have to include just the fields to change (none in my case) and “any fields required by the document” – that’s why I included fields such as PurchOrderFormNum. Note that you may have different fields set as mandatory.

Also notice how action properties are specified – we’re deleting the line, which means updating the order.

static void Main(string[] args)
{
    using (SalesOrderServiceClient client = new SalesOrderServiceClient())
    {
        EntityKey[] entityKeyList = EntityKeyForSalesId("SO00001");
 
        // Get the order to modify
        var order = client.read(new CallContext(), entityKeyList);
 
        // For demo, always the last order line is deleted
        var lastLine = order.SalesTable[0].SalesLine.Last();
 
        var salesLine = new AxdEntity_SalesLine()
        {
            RecId               = lastLine.RecId,
            RecIdSpecified      = true,
            SalesUnit           = lastLine.SalesUnit,
            action              = AxdEnum_AxdEntityAction.delete,
            actionSpecified     = true
        };
 
        var salesTable = new AxdEntity_SalesTable()
        {
            _DocumentHash           = order.SalesTable[0]._DocumentHash,
            PurchOrderFormNum       = order.SalesTable[0].PurchOrderFormNum,
            ReceiptDateRequested    = order.SalesTable[0].ReceiptDateRequested,
            action                  = AxdEnum_AxdEntityAction.update,
            actionSpecified         = true,
            SalesLine               = new[] { salesLine }
        };
 
        AxdSalesOrder newOrder = new AxdSalesOrder()
        {
            SalesTable = new[] { salesTable }
        };
 
        // Update the order
        client.update(new CallContext(), entityKeyList, newOrder);
    }            
}
 
// Helper method
private static EntityKey[] EntityKeyForSalesId(string salesId)
{
    KeyField field = new KeyField()
    {
        Field = "SalesId",
        Value = salesId
    };
 
    EntityKey key = new EntityKey()
    {
        KeyData = new[] { field }
    };
 
    return new[] { key };
}

Locking in Team Foundation Version Control

If you use Team Foundation Version Control, you have several ways to control whether multiple developers can work on same objects simultaneously. You normally want that, because developers often need to change a common file (a class, a configuration file and so on) and waiting for each other would cause unnecessary delays. But it’s not always the case – maybe you want to lock an individual file before making a sensitive change, some files can’t be easily merged so it’s not wise creating concurrent versions, maybe you’re developing in a shared AX instance and so on.

Before we actually look at locking, it’s important to understand the difference between local and server workspaces, because each of them work with locks differently. A TFS workspace defines which files are downloaded from TFS, tracks edited files and so on. Beginning with TFS 2012, you can choose between local and server workspaces; older versions of TFS support server workspaces only.

Note that you can easily switch between workspace types – simply open workspace properties in Visual Studio, switch to the “advanced” display and set Location to either Local or Server.

WorkspaceProperties

If you use a server workspace, a lot of state information is maintained on server, therefore you need a connection to your TFS to perform actions such as checkout or add. Local workspaces can track such actions by themselves, therefore you may be disconnected from TFS (if you have all files you need already in the workspace). If you want to check in changes to version control, you obviously need connection to server regardless of the type of your workspace.

This clearly leads to a difference in when you can apply locks – because local workspaces don’t have to communicate with server on checkout, you can’t expect them to honor locks. Therefore if you need exclusive check-out locks, you have to use server workspaces. Both type of workspaces can use check-in locks, which means that developers can do local changes but they can’t check them in until the lock is released.

Look at Understand lock types if you need more information about lock types and their use.

One way of taking a lock is as a part of check out. Simply choose a lock type in the Check Out dialog:

Checkout

The picture above is from a server workspace. If I use a local workspace, Check Out lock isn’t available.

LockTypeLocalWorkspace

You can be offline if you don’t need any lock, but you obviously must have a connection to TFS if you want it to lock the file and prevent others from uploading a new version.

Another option is taking a lock manually.

MenuLock

If you want TFS to take a Check Out lock automatically for certain file types (especially because you can’t merge them), you can configure it in team project collection settings. One option to get to the right form is Team > Team Project Collection Settings > Source Control…

TPCSourceControlMenu
TPCSourceControlSettings

Here you can edit or add file types and configure whether multiple checkouts should be allowed:

EditFileType

And finally, you can disable multiple checkouts for the whole team project (regardless of file type). Open source control settings for an individual team project (Team > Team Project Settings > Source Control…) and uncheck Enable multiple check-out.

TPSourceControlSettings

People sometimes think that it’s all or nothing (everything locked/unlocked), but you clearly have more options available. And the impact of workspace types can be very confusing for those who don’t understand workspaces.

All pictures are from Visual Studio 2013 connected to Visual Studio Online.