Recurring Integrations Scheduler: File already exists

I’ve recently run into a problem with Recurring Integrations Scheduler and I think I won’t be the only one, therefore I’m going to explain it here.

The Recurring Integrations Scheduler (RIS) supports both the old recurring integrations API (enqueue/dequeue) and the new package API, which is the recommended approach. With the package API, you can either import whole packages (which you must prepare by yourself in some way) or you can give RIS actual data files (XML, CSV…) and let it build packages for you. That’s the scenario I’m talking about below.

If you want RIS to create packages, you must give it a package template. Go to the Data Management workspace in AX 7, create an import job, configure it as you like and then download the package. I’m using the Customer groups entity as an example.

You’ll get a file with a GUID as the name, such as {99DD43E4-936E-4402-80E6-1477013A5275}.zip. Feel free to rename it; I’ll call it CustGroupImportTemplate.zip.

This is its content:

You can see one data file for the entity (Customer groups.xml in my case) plus some metadata defining the import project.

It’s not exactly what’s needed, as we’ll see later, but let’s continue for now with this package.

Go to RIS and configure an import job. Don’t forget to set Package template to the package file.

Make sure the job is running, create an input file in the right format and put it into the input folder.

Unfortunately the import fails and you’ll find these files in the Processing errors folder:

The status file merely says that StatusCode = BadRequest and events logs for RIS (Applications and Services Logs > Recurring Integrations Scheduler) don’t reveal anything else either. More useful event logs are under Applications and Services Logs > Microsoft > Dynamics > AX-OData Service, where you can find something like this:

System.InvalidOperationException: Exception occurred while executing action ImportFromPackage on Entity DataManagementDefinitionGroup: The file ‘C:\windows\TEMP\005DE7CC-C7E8-410D-8690-7A63C30224BF_FF5EE0BE-205C-48BC-8987-96C007CE74ED\Customer groups.xml’ already exists.

A unique folder name is generated for each import, therefore the problem isn’t related to existing files in the Temp folder.

To make it short, the problem is in the package file itself, which you unfortunately never see, because it’s automatically deleted on failure. If you captured it and looked inside, you would find this:

There are literally two files with the same name and therefore the attempt to unpack the archive fails on the second instance.

How does this happen?

One file is coming from the package template. The other is the input file, which RIS renamed to match the entity name and added to the archive. Having files with same names is supported by ZIP, but the library used by RIS for unpacking isn’t able to handle it.

To get it working correctly, you must remove the data file (Customer groups.xml) from the package used as a template. A better solution, though, would be modifying RIS to replace the file instead of attaching another file with the same name.

Recurring Integrations Scheduler

You may have heard about QuartzAX, an application for file-based integration with AX 7 (Dynamics 365 for Finance and Operations, Enterprise Edition). Microsoft announced at the last Technical conference that it would be released in a few days, but it didn’t happen and it looked like it wouldn’t ever be made available. But it has changed today.

You can find its source code and documentation on GitHub under the new name: Recurring Integrations Scheduler.

Open API for JSON-based custom services in AX 7

If you’re familiar with SOAP web services, you likely know that they use Web Services Description Language (WSDL) to document what operations a service provide, what parameters they accept, what they return and so on. This information can be used to generate proxy classes, which you can use to communicate with remote systems simply by calling class methods and all implementation details are handled for you under the hood.

While SOAP-based services offer many configuration options (e.g. you can switch from TCP to HTTP just by changing a config) and great tooling, sometimes it’s beneficial to use a less sophisticated approach and expose and call services directly over HTTP (that’s why AX 7 added JSON-based endpoints for custom services). Almost everything can communicate over HTTP, it doesn’t require any extra libraries, it’s very flexible and so on. There are some patterns how to design such services (especially REST), but you can do whatever you want. But it means that you have to give up features like code generation of proxy classes. Or not?

While HTTP web services don’t have to use any descriptions similar to WSDL, it doesn’t mean they can’t. One of your options is Open API (also known as Swagger), which allows you to describe services in JSON or YAML. There are many tools for working with these descriptions, such as an editor, code generators for many languages, a generator of documentation, you can generate Open API descriptions for your WebAPI projects, there is a support for Open API in Azure Logic Apps, in Azure API Management and so on.

Let me show you an example of such a description for a custom service in AX 7 (Dynamics 365 for Finance and Operations, Enterprise Edition). You don’t have to examine it in detail, just notice that it describes available operations, what URL you must use to reach it, what parameters it expects, what it returns and so on, and it also contains additional textual annotations.

{
  "swagger": "2.0",
  "info": {
    "title": "User management",
    "description": "Services for user management offered by Microsoft Dynamics 365 for Finance and Operations, Enterprise Edition.",
    "version": "1.0.0"
  },
  "host": "YourAX7Instance.cloudax.dynamics.com",
  "basePath": "/api/services",
  "schemes": [
    "https"
  ],
  "paths": {
    "/SysSecurityServices/SysUserManagement/getRolesForUser": {
      "post": {
        "summary": "Gets security roles",
        "description": "Returns security roles currently assigned to the given user in Microsoft Dynamics 365 for Finance and Operations, Enterprise Edition.",
        "operationId": "getRolesForUser",
        "produces": [
          "application/json"
        ],
        "parameters": [
          {
            "name": "body",
            "in": "body",
            "description": "User ID as defined in UserInfo table.",
            "required": true,
            "schema": {
              "$ref": "#/definitions/GetRolesRequest"
            }
          }
        ],
        "responses": {
          "200": {
            "description": "Roles assigned to user",
            "schema": {
              "type": "array",
              "items": {
                "type": "string"
              },
              "example": [
                "-SYSADMIN-",
                "SYSTEMUSER"
              ]
            }
          },
          "500": {
            "description": "Processing inside Dynamics 365 for Finance and Operations failed."
          }
        }
      }
    }
  },
  "definitions": {
    "GetRolesRequest": {
      "type": "object",
      "required": [
        "axUserId"
      ],
      "properties": {
        "axUserId": {
          "type": "string",
          "example": "admin"
        }
      }
    }
  }
}

The first thing we can do with it is to show the same information in a more readable way. Open the online editor and paste the sample code there. On the right side, you’ll see it rendered as a nice HTML document:

Service documentation generated by editor.swagger.io

The editor can also use YAML (and will offer you to convert JSON to YAML), which is a more succinct format, therefore you may prefer it for manually edits. I intentionally used JSON, because we’ll need it in the next step.

Code generation

Now let’s generate code for calling the service, so we don’t have to do it all by hand and deal with all implementation details of communication over HTTP, with (de)serialization of values and so on.

Before you start, download and configure AX service samples from GitHub and verify that they work. Then create a new console application in ServiceSamples solution, where we’ll call a custom service through generated client classes. My code below assumes that the project is called JsonConsoleWithSwagger; you’ll have to adjust a few things if you use a different name.

Client classes can be generated by several different tools, therefore the following process is just an example; feel free to use other options in your projects.

Download and install NSwagStudio (if you use Chocolatey, as I do, all you need is cinst nswagstudio). Run NSwagStudio, paste the JSON on Swagger Specification tab, tick CSharp Client and change a few properties at CSharp Client tab:

  • Namespace: JsonConsoleWithSwagger (or something else, if you named the project in Visual Studio differently)
  • Class Name: UserManagementClient
  • Use and expose the base URL: No
  • Class style: Poco

You could use different parameters, such as different Class style, but this will suffice in our case.

Then press Generate Outputs, which will generate corresponding C# code. Copy it to clipboard, create a new class in your Visual Studio project and replace the file content with the generated code.

We need to add just two things – the actual URL of your AX instance and an authentication header. We don’t want to change the generated code itself, because we may need to regenerate it later and we would lose our changes. Fortunately, it’s not needed – the class is partial, therefore we can create another part of the same class in a separate file and keep generated code and custom code cleanly separated.

Create a new file in your project and paste the following code there:

using AuthenticationUtility;
using System.Text;
using System.Net.Http;
 
namespace JsonConsoleWithSwagger
{
    public partial class UserManagementClient
    {
        partial void PrepareRequest(HttpClient client, HttpRequestMessage request, StringBuilder urlBuilder)
        {
            PrependAxInstanceUrl(urlBuilder);
            client.DefaultRequestHeaders.Add(OAuthHelper.OAuthHeader, OAuthHelper.GetAuthenticationHeader());
        }
 
        private void PrependAxInstanceUrl(StringBuilder urlBuilder)
        {
            string service = urlBuilder.ToString();
            urlBuilder.Clear();
            urlBuilder.Append(ClientConfiguration.Default.UriString);
            urlBuilder.Append("api/services/");
            urlBuilder.Append(service);
        }
    }
}

PrependAxInstanceUrl() takes the address of your AX from configuration and puts it at the beginning of the request URL.

Then the code sets the authentication header with the help of OAuthHelper from AuthenticationUtility project, therefore we must add a reference to it:

The last step is adding code to Main() method of Program class to actually call the service. We create a request object (please provide an existing user ID there), create an instance of the UserManagementClient class generated from our Open API document and call the operation. It’s asynchronous, as recommended, but we don’t really need that here, therefore the code immediately asks for the result and waits for it. Then we iterate roles received from AX and puts them into console.

GetRolesRequest request = new GetRolesRequest()
{
    AxUserId = "user7281"
};
 
var roles = new UserManagementClient().GetRolesForUserAsync(request).Result;
 
foreach (string role in roles)
{
    Console.WriteLine(role);
}

That was easy – we didn’t have to bother about what exact parameters the services expects (and we would get a compile error if we did it wrong), we didn’t have to serialize objects to JSON or anything like that. The generator was able to create all the code for us, it just needed to know how the service looks like.

In this case, I wrote the Open API document by hand, which obviously took some time. A much better approach would be generating it from metadata of custom services in AX, and while I don’t have such a solution in the moment, it’s definitely doable. Some information is already in place (e.g. getting the list of operation is easy), services and service groups already have properties for description (although they’re currently empty in most cases) and things like parameter description can be included in XML documentation. It still doesn’t cover everything, but additional information can be easily provided in attributes. It’s exactly what Swashbuckle does, e.g. with SwaggerResponseAttribute and RequiredAttribute.

I think it’s something that Microsoft should use for its custom services, to provide documentation and to make custom services much easier to consume. Open API / Swagger is a natural choice for this purpose, because Microsoft is a founding member of Open API Initiative and already support it in several products. Maybe Microsoft could somehow utilize Swashbuckle inside the implementation of custom services, instead of building something new from scratch.

But even if Microsoft isn’t interested, I believe it would still be useful for many people, therefore the community could build a tool to extract information about custom services and generate Open API documents. It wouldn’t necessarily have to support all features (or not from the beginning); people can easily add textual annotations, examples and so on in Swagger Editor, if needed. But being able to automatically generate JSON-based client classes for any custom service would be really handy.

Discovery of JSON-based custom services in AX 7

If you download AX integration samples from GitHub, you’ll see (in JsonConsoleApplication project) that you can call JSON-based custom services by code like this:

var request = HttpWebRequest.Create(ClientConfiguration.Default.UriString + "api/services/UserSessionService/AifUserSessionService/GetUserSessionInfo");
request.Headers[OAuthHelper.OAuthHeader] = OAuthHelper.GetAuthenticationHeader();
request.Method = "POST";
request.GetResponse();

It will call the operation and return its return value as JSON:

{
    "$id":"1",
    "AOSLocaleName":"en-US",
    "AXLanguage":"EN-US",
    "Company":"DAT",
    "CompanyTimeZone":58,
    "CurrencyInfo": {
        "$id":"2",
        "CurrencyCode":"USD",
        "Description":"US Dollar",
        "ExchangeRate":100.0,
        "ISOCurrencyCode":"USD",
        "Prefix":"","Suffix":""
    },
    "IsSysAdmin":true,
    "UserId":"wintermute",
    "UserPreferredCalendar":0,
    "UserPreferredTimeZone":18
}

This is what happens when you use the POST method of HTTP; if you switch to GET, you’ll actually get some information about the service.

Therefore if you merely change the value of request.Method:

var request = HttpWebRequest.Create(ClientConfiguration.Default.UriString + "api/services/UserSessionService/AifUserSessionService/GetUserSessionInfo");
request.Headers[OAuthHelper.OAuthHeader] = OAuthHelper.GetAuthenticationHeader();
request.Method = "GET";
request.GetResponse();

you’ll get a very different response:

{
    "Parameters":[],
    "Return": {
        "Name":"return",
        "Type":"AifUserSessionInfo"
    }
}

You can see that the operation doesn’t expect any parameters and it returns a single object of AifUserSessionInfo type. This gives you some limited information about how to use this service.

You can also use GET requests to discover existing services and their operations.

/api/services gives you a list of all service groups:

{"ServiceGroups":[{"Name":"AxClient"},,{"Name":"UserSessionService"}]}

/api/services/UserSessionService provides a list of services in the service group:

{"Services":[{"Name":"AifUserSessionService"}]}

/api/services/UserSessionService/AifUserSessionService shows all operations of the individual service:

{
    "Operations": [
        {"Name":"ApplyTimeZone"},
        {"Name":"GetAccessRights"},
        {"Name":"GetPartitionKey"},
        {"Name":"GetPartitionKeysForUser"},
        {"Name":"GetUserSessionInfo"},
        {"Name":"IsSinglePartitionSystem"},
        {"Name":"RemoveTimeZone"}
    ]
}

Don’t forget than opening an URL in browser makes a GET request, therefore you don’t have to write any code to get this kind of information about custom services.

But maybe you would like something a bit more sophisticated, which is the topic of the next post: Open API for JSON-based custom services in AX 7.

Integration with AX 7 in cloud

I’m getting a lot of questions along the lines of “how on earth can we integrate our on-premises systems with AX 7 running in cloud?”, therefore let me share some of my thoughts.

The first important point to realize is that “to integrate” means many different things, so there is no single solution for it. On the contrary – you know have more options than before, because you can easily utilize many existing cloud services, which weren’t that useful when your ERP wasn’t running in cloud. Although I mention quite a few options below, the list isn’t comprehensive by any means.

You have to take many things into account when designing your solution: which system triggers the integration (AX, external systems), what’s the trigger (schedule, user activity…), whether you need a response in synchronous manner or you can utilize queues (which is better for load balancing), what are requirements for performance, reliability, monitoring and so on and so on. I’m not going to discuss these things in detail, but you mustn’t forget about them.

Below I look at some integration options based on which system is triggering integration. There are quite a few links, if you want to learn more.

From your servers / workstations

The most straightforward approach is sending requests from your own machines to AX.

It may be a from a desktop application (the Excel add-in calling OData services exposed by AX 7 is a good example) or a server process (e.g. a service monitoring a network folder and pushing files to AX).

If you want to call AX directly, you can use its services (especially OData and custom services), including pushing files through web services (e.g. data management API), but it’s often not the best strategy. For example, you may want to send messages even if AX is down for maintenance and therefore you merely put them to some storage (Service Bus queue, blob storage, file storage…) where AX will find them when it’s back online.

Or maybe you have some additional logic to run, such as a workflow with conditions, message transformations, aggregation with other data and so on. You can utilize things like Logic Apps or Azure Functions for this purpose. Alternatively, you can run your logic on-premises before sending messages to cloud; then you can use BizTalk, for instance.

Logic Apps can also read data from storage and pushing it to AX, so you don’t have to implement it by yourself.

From AX

Running a piece of code in AX (usually in a batch) is a natural choice for many AX developers, but some aspects are a bit different in cloud.

In past, people often put files to import (to take this direction as an example) to a network folder, where AX read them on schedule. Now they ask how to put files to production VMs, which simply isn’t possible. You also can’t expect Azure applications to read files directly from your local hard disk. If you want to use this approach, you must store your files elsewhere, such as in an Azure storage in your subscription, on FTP and so on. For example, you can put files to OneDrive or a file storage mapped as a drive and let AX to read them from there via an API (although you may want to leave it to a Logic App, as demonstrated in File based integration using Logic Apps).

You can also call web services from X++, which includes even web services offered by your on-premises systems (exposed through Azure AD Application Proxy or Azure Relay, for example).

You can also call middle-tier services and execute any other code you like.

From a middle-tier service

There is a third option, when integration is orchestrated by another service in the middle. For example, you may have a logic app that runs on schedule, reads data from your on-premises Oracle database and uses the connector for AX 7 to create records in AX. Then you don’t have to develop any on-premises application pushing data to cloud, you merely install a gateway).

Some people are afraid that there is no way how to integrate on-premises and cloud systems, but that’s obviously not the case. You have many options, you just must choose the right one for your particular goal. It’s different than if everything runs in your own server room and it’s likely confusing if you’re not familiar with Azure (with new types of storage, app authorization and so on), but that’s what always happen when you start working with different architecture. You’ll also often combine several services, rather than having a single universal solution for integration.