Save storage with data deduplication

This blog post isn’t directly about software development, because I’m going to look at a certain feature of Windows Server, but my primary motivation is still dealing with development and testing.

When you want to set up AX 7 (Microsoft Dynamics 365 for Finance and Operations, Enterprise Edition :)) environments for development, testing or demonstration purposes, you can either deploy them to Azure or download a virtual disk and create a virtual machine locally or in a custom data center. Many partners (especially VARs) prefer hosting VMs by themselves, because they’ve already invested to data centers and they’re used to hosting virtual machines with previous versions of AX.

They may need a large number of VMs for AX 7 (for many projects and many developers), but it’s not a problem for computation resources, because just a fraction of these VMs are actually running. For example, a developer may have VMs for five projects but only one of them is used for development at any time. The problem people complain about is the storage requirement, because the virtual disks are pretty big (a single fresh VHD takes approximately 70 GB and it grows when used).

Most data in these virtual disks is the same, therefore if there was a feature to somehow store the same bits just once, it would drastically reduce the amount of data in physical storage. Fortunately it’s exactly what Windows offers.

Windows Server 2012 introduced the Data Deduplication feature, which detects identical blocks of data (not whole files, which would be useless for virtual disks) and stores them only once, keeping a sort of a link from the other places. It’s completely transparent to you – you work with all files in the same way as before. Windows Server 2012 R2 then added extra support for virtual disks in VDI (Virtual Desktop Infrastructure) scenarios, therefore this is the minimum version you want to use. (Windows Server 2016 supports Data Deduplication too, of course, and have added a few improvements.)

According to Microsoft measurements, the usual space saving in these scenarios is 80-95% and I would indeed expect high numbers with AX VMs, because they’re almost the same. This means that any additional VM will need (after deduplication) just a few extra gigabytes of storage and having many VMs won’t be a problem anymore.

I haven’t try it by myself, though, because I don’t need it at home and clients have specialized infrastructure guys dealing with these things. If you have practical experience, you can share it in comments below this post.

If you want to learn more about data deduplication in Windows Server, the latest documentation is here.

Catching exceptions in AX 7

In this blog post, I briefly recapitulate how to throw and catch exceptions in X++ and introduce a new way of handling CLR exceptions in AX 7.

X++ exceptions

When you want to throw an exception in X++, you typically do it by something like this:

throw error("It's broken!");

It’s a functional equivalent of adding a message to infolog and throwing Exception::Error.

infolog.add(Exception::Error, "It's broken");
throw Exception::Error; // number 3

As you see, an exception in X++ is (or used to be, as I’ll discuss later) just a number, because enums are backed by integer values. It doesn’t come with any extra information, not even the message. Adding messages to infolog is a separate process; you can add error messages to infolog without throwing any exceptions and throwing exceptions without adding anything to infolog.

When you want to catch an exception, you’ll use a catch clause with the right value of Exception enum (usually Exception::Error) or you’ll handle all exceptions (by a catch clause without any type).

The following statement will work for all errors, but you don’t get any information about what error it was:

catch (Exception::Error) {}

Just knowing that there is an error doesn’t allow you to react differently to different errors, therefore you’ll rarely see code in AX trying to recover from a particular error.

CLR exceptions (in general)

It’s very different in Common Language Runtime (CLR). Exceptions there are objects (instances of classes extending System.Exception) and they contain a lot of information that can help you to identify what happened. For example, you can look at the type of exception (e.g. ArgumentNullException or FileNotFoundException) and react accordingly. The error message is a part of the object too, not just lying somewhere in infolog. The exception also comes with the stack trace, so you can easily see where it was thrown from.

Because the classes form a hierarchy, you can also handle exceptions in a hierarchic way. This is what you can write in C#:

catch (FileNotFoundException ex) {}
catch (Exception ex) {}

The system will try catch clauses one by one, going from the top down, and will use the first compatible clause it finds. If a FileNotFoundException is thrown, the first block will be used, but all other exceptions will go to the latter one, because they’re not FileNotFoundException but they’re all compatible with System.Exception, which its their common base class.

Exception::CLRError

Sometimes you have to handle CLR exception in X++, because you can use .NET libraries from X++ and such libraries can throw their usual (CLR) exceptions. The traditional approach is catching Exception::CLRError, getting the exception object from CLRInterop::getLastException() and extracting information from it. It’s further complicated by the fact that the actual error is usually wrapped in TargetInvocationException.

This is how you can handle FileNotFoundException in X++:

try
{
    System.IO.File::ReadAllText('c:\\nothing.here');
}
catch (Exception::CLRError)
{
    System.Exception wrapperEx = CLRInterop::getLastException() as System.Reflection.TargetInvocationException;
    if (wrapperEx)
    {
        if (wrapperEx.InnerException is System.IO.FileNotFoundException)
        {
            warning(wrapperEx.InnerException.Message);
        }      
    }
}

It works, but it’s cumbersome and hard to understand.

Catch with object reference

Fortunately AX 7 offers a new, much better option. Let me start by refactoring the previous example:

System.IO.FileNotFoundException fileNotFoundEx;
 
try
{
    System.IO.File::ReadAllText('c:\\nothing.here');
}
catch (fileNotFoundEx)
{            
    warning(fileNotFoundEx.Message);
}

This is obviously much shorter and easier to follow. In catch, I don’t have to use only values of the Exception enum anymore, I can also use exception objects. The system checks the type, selects the right catch clause and passes the exception object there. It’s almost the same as in C#, except of the fact that you can’t define the exception type directly in the catch condition; you have to declare a variable in an outer scope.

As in C#, you may have several catch clauses for different types of exceptions and benefit from the hierarchical nature of exception types. For example, the following snippet handles two types of exceptions in a special way and all remaining exceptions go to the last catch clause.

System.Exception                generalEx;
System.FieldAccessException     accessEx;
System.IO.FileNotFoundException fileNotFoundEx;
 
try
{}
catch (accessEx)
{
    warning("Field access");
}
catch (fileNotFoundEx)
{
    warning("File not found");
}
catch (generalEx)
{	
    warning(ex.Message);
}

ErrorException

All right, so X++ errors can be caught with catch (Exception::Error) and CLR exceptions by their particular type, but isn’t all code now executed by CLR? Does it mean that X++ exceptions use a different mechanism than CLR exceptions, or they’re just normal CLR exceptions under the hood?

You can find the answer in debugger, which reveals that “native” AX exceptions are indeed implemented as CLR exceptions:

The object (as you can see in debugger) has all usual properties, such as Message, Source, StackTrace and so on. Note that you can see the same behavior in AX 2012 if you debug CIL generated from X++.

Now what if you want to access the properties when handling an exception, e.g. to include the stack trace in a log? If you catch Exception::Error, you won’t get any details, but what if you try to catch ErrorException in the same way as any other CLR exception? If you think it should be possible, you’re right – the following piece of code successfully catches an X++ exception and shows how you can access its properties.

Microsoft.Dynamics.Ax.Xpp.ErrorException xppEx;
 
try
{
    throw error("It's broken!");
}
catch (xppEx)
{
    this.log(xppEx.StackTrace);
}

The problem that all X++ errors have the same exception type (ErrorException) is still there, therefore handling different errors in different ways is still hard, but you can now easily find which message belongs to the exception (without digging into infolog), where it was thrown from and so on.

By the way, I also wondered what would happen if I tried to catch both Exception::Error and ErrorException, because they’re internally the same thing.

Microsoft.Dynamics.Ax.Xpp.ErrorException xppEx;
 
try  
{
    throw Exception::Error;
} 
catch (Exception::Error)
{
    info("Exception::Error");
}
catch (xppEx)
{
    info("ErrorException");
}

Such code compiles without any problem and the resulting CIL will actually contains two catch clauses for the same exception type (ErrorException). It means that the top one always wins, regardless whether it’s catch (Exception::Error) or catch (xppEx). Using both for the same try statement makes little sense, but at least we now know that nothing catastrophic happens if somebody does it by mistake.

Conclusion

The native implementation of exceptions in X++ is very limited in comparison to CLR exceptions. The inability to distinguish between different kinds of errors makes meaningful recovery from errors virtually impossible and the lack of associated information (such as the message) makes even mere logging quite cumbersome. This doesn’t apply to CLR exceptions and their handling in X++ is even easier than before, thanks to the new ability of catch clauses. The fact that we can use this approach for X++ exception as well means that we can easily work around one of the limitations and access properties such as message and stack trace even for them.

It seems that we’re just a small step from being able to work with exceptions in X++ in the same way as in .NET. If we could throw exception objects (e.g. throw new MissingRecordException()), we would be able catch this particular type of exception and implement more robust logic for recovery from errors.

It would also help if X++ was extended to support declaration of the exception type directly in catch (e.g. catch (MissingRecordException ex)).

Flow and Logic Apps

I see a lot of confusion regarding Microsoft Flow and Logic Apps, so let me very briefly explain what they are and when you should use them.

First of all, you should realize that they’re very closely related. For example, this is the designer for Flow:

And this is the designer for Logic Apps:

They’re almost identical, because Microsoft Flow is built on top of Logic Apps. Many capabilities are not just similar; they’re physically the same.

Both Logic Apps and Flow are cloud-based integration services. They allow access to data from various system (my example above uses Azure Service Bus and Dynamics 365 for Operations), to run various actions, define workflows with conditions, branches, loops and things like that. They use the same connectors, the same actions and the same graphical designer.

So what’s the difference?

Microsoft Flow is intended for self-service integration. It empowers end users to set up various integration by themselves, without having to request developers to do it for them. For example, I may want to get some data (such as expected receipts) by e-mail every morning. It would help me, but asking my implementation partner to do it for me would be an overkill. I can use Microsoft Flow to set it by myself.

Azure Logic Apps, on the other hand, is a development platform which can be used for company-wide and mission-critical integration scenarios. Although connectors and things like that are the same as in Flow, Logic Apps offers Code view in addition to the graphical designer, which may be more efficient in some cases and more importantly, it offers some additional advanced capabilities. Logic Apps can also be developed in Visual Studio, stored in version control as any other code, built and deployed by VSTS, they offer more features around security, monitoring, automation and so on.

It seems that people hear more about Flow than Logic Apps and some of them try to use Flow for all their integration needs, but it isn’t a good idea. Flow isn’t intended for this purpose and you would soon miss many tools needed for development, deployment and maintenance of your solution. Flow is really a simplified version for personal needs.

By the way, people often ask about how to integrate with on-premises systems. Both Logic Apps and Flows offer gateways (link for Logic Apps, link for Flow) that you can use to access on-premises files, databases, SharePoint and thing like that.

IoC containers for extensibility

Dynamics 365 for Operations (AX 7) tries to get rid of the ugly concept of “customizations”, when developers directly change somebody else’s code, which has a plenty of negative consequences. Most importantly, installing a new version isn’t easy, because it requires dealing with conflicts in code.

There are several ways how to make a class extensible without requiring changes in the class itself. In certain cases, you can make a specialization of a base class, add additional behavior or modify the current behavior by overriding methods. It’s easy to do and understand, but there is a catch: you have to control the place where the class is instantiated, to provide your class instead of the original one. For example, if you have code like this,

IComponent c = new ComponentBase();

it will always create an instance of ComponentBase and nothing else. You need something smarter.

The solution is dependency injection – things like the construction of specific types are taken out from the class using them. The logic is only aware of an interface or base class and doesn’t care about what exact type it is, therefore you can start using another type and it will all work.

For example, you can provide the dependency in a constructor:

class MainLogic
{
    IComponent component;
 
    public void new(IComponent _c)
    {
        component = _c;
    }
}

The class doesn’t refer to ComponentBase or any other specific type; it will work with any class implementing IComponent interface and it’s up to the calling code to provide a suitable type.

If you don’t do anything else, you might end up with the same problem as before. There may be a piece of code constructing ComponentBase directly and passing it to the constructor, which you would have to replace by changing the code:

new MainLogic(new BaseClass());

You need to stop constructing such instances by yourself – leave it to a special piece of logic that knows which types to use: an Inversion of Control (IoC) container.

In Dynamics 365 for Operations, you can do something similar with plugins, but it’s quite cumbersome. Let me demonstrate how easy it can be to use an IoC container, namely Autofac in C#. I think it should be inspiration for X++.

First of all, let’s make a base class with some default implementation:

public class ComponentBase
{
    public virtual int GetValue()
    {
        return 42;
    }
}

and another class (overriding the method) that we want to use instead of the base class.

class ComponentCustom : ComponentBase
{
    public override int GetValue()
    {
        return 1337;
    }
}

Using an interface would be better, but because most code in X++ doesn’t utilize interfaces, this is closer to reality.

Then let’s have a class calling a method of the component and using the value (in this case simply showing it in console).

class MainLogic
{
    public ComponentBase Component { get; set; }
 
    internal void DoStuff()
    {
        Console.WriteLine($"The value is {Component.GetValue()}");
    }
}

In this case, I’m not getting the component in constructor – I’m using a property instead, which the calling code (such as an IoC container) will have to set.

Before actually calling the logic, we have to set up our IoC container. Here is the code for Autofac:

var builder = new ContainerBuilder();
builder.RegisterType<ComponentCustom>().As<ComponentBase>();
builder.RegisterType<MainLogic>().PropertiesAutowired();
IContainer container = builder.Build();

The most interesting is the second line, where we instruct the container to use ComponentCustom in place of ComponentBase.

Then we can use container’s Resolve method to get an instance of MainLogic and execute DoStuff():

MainLogic mainLogic = container.Resolve<MainLogic>();
mainLogic.DoStuff();

The result is, as expected, is the value from ComponentCustom .


 
IoC containers allow you to get rid of instantiating components by yourself, therefore there is nothing like new ComponentBase() that you would have to change by overlayering; you just need a way to call RegisterType() of the builder. AX could offer an event for this purpose, where each extension would register its logic. Or there could be attributes similar to those used for declarative event handler subscriptions. IoC containers even allow setting type mapping in configuration files, therefore you can easily change type to use without touching code at all.

It would still require changes to the application, but it would usually mean merely replacing a constructor call with Resolve(). The current alternative, plugins, requires much more work and therefore it’s unlikely to be widely adopted for this purpose (to be fair, it’s not intended for this).

Because X++ doesn’t support properties, the implementation of my example would be more involved in current X++, but that’s just an implementation detail. And it may be extra motivation for introducing properties to X++.

Although I focused on extensiblity in this blog post, the loose coupling you can achieve with dependency injection has many other benefits, such as simplifying isolation for unit testing.

Note that IoC containers can do much more than what I’ve demonstrated here and there are many different solutions to choose from.

Just for the sake of completeness, X++ plugins are based on Managed Extensibility Framework.

Filter for enum fields in OData service

It took me a while to figure out how to filter AX (Operations) enum fields in the URL for OData service (I was using Get records in Flow). For example, let’s say I wanted to get sales orders with status = open order.

After a few failed attempts, I actually used classes generated by OData Client Code Generator, constructed a LINQ query and ran it.

context.SalesOrderHeaders.Where(h => h.SalesOrderStatus == SalesStatus.Backorder).ToArray();

Then I intercepted the URL in an event handler for SendingRequest2 and finally got the answer. The filter must be set in this way:

SalesOrderStatus eq Microsoft.Dynamics.DataEntities.SalesStatus'Backorder'

No wonder I didn’t guess it.