How to detect anomalies in Business Central data using AL Http datatypes

Anomaly Detector is an Azure Cognitive Service that detects anomalies in time-series data. It consists of simple REST APIs that can be consumed by any application. This post explains how to setup Azure Anomaly Detector Service, and how to consume Anomaly Detector REST APIs in Business Central using AL Http datatypes.


Why and Where to find Anomalies

Purpose of Anomaly Detector is to identify unknown threats in time-series data. Anomalies can be detected in any time-series data.


The following are the potential data sources in Business Central for anomaly detection:

  • Bank Ledger (Payments / Receipts)

  • Cash / GL (Payments / Receipts)

  • Sales (Invoices / Cr. Memos)

  • Purchase (Invoices / Cr. Memos)

  • Row Material Consumption (Item Ledger)

  • Production Output (Item Ledger)

Any data that has Time (Date / Time / DateTime) and Value (Integer / Decimal) columns is a valid data source for anomaly detection.


How to Setup Cognitive Service

  • Select Cognitive Services and click the Create button in Cognitive Services page.

  • Search for "Anomaly Detector" and click the Create button in "Anomaly Detector" card.

  • Update the project details and click the "Review + Create" button.

  • After creating your "Anomaly Detector" service, open the service and click the Overview link in the left panel.

You can check Endpoint URL (masked) in the above screen.

  • Click the "Keys and Endpoint" link in the left panel or the "Click here to manage keys" link in the previous screen to get the API key.

You can take the API key (KEY 1) and the Endpoint URL from the above screen.


AL Code to detect anomalies using REST APIs


The following code sends date wise bank transactions to anomaly detector service and retrieves anomalies in those transactions.


Constants / Labels : SubscriptionKeyLbl is masked for security reasons. EndpointLbl is the value of "KEY 1" from "Keys and Endpoint" screen. Ideally these values should be kept in a Setup table.

var
        RequestUrlLbl: Label '%1/anomalydetector/v1.0/timeseries/entire/detect', Locked = true, Comment = '%1 = Endpoint';
        EndpointLbl: Label 'https://anomalydetector.cognitiveservices.azure.com', Locked = true;
        SubscriptionKeyLbl: Label 'XXXX95d2XXXX1ebaXXXX4fdbXXXX898', Locked = true;

FindAnomalies method takes Bank Account No., From Date and To Date parameters and returns Anomalies in the data.

procedure FindAnomalies(BankAccountNo: Code[20]; FromDate: Date; ToDate: Date; AnomalyData: Dictionary of [Date, Decimal]): Boolean
var
	TimeSeriesData: Dictionary of [Date, Decimal];
begin
	ToTimeSeriesData(BankAccountNo, FromDate, ToDate, TimeSeriesData);
	exit(CheckAnomaly(TimeSeriesData, AnomalyData));
end;

ToTimeSeriesData method takes data from "Bank Account Ledger Entry" and stores it in the TimeSeriesData variable.

Note: Query object can be used to achieve the same for better performance.

local procedure ToTimeSeriesData(BankAccountNo: Code[20]; FromDate: Date; ToDate: Date; TimeSeriesData: Dictionary of [Date, Decimal])
var
	BankAccountLedgerEntry: Record "Bank Account Ledger Entry";
	PrevValue: Decimal;
begin
	BankAccountLedgerEntry.Reset();
	BankAccountLedgerEntry.SetRange("Bank Account No.", BankAccountNo);
	BankAccountLedgerEntry.SetRange("Posting Date", FromDate, ToDate);
	if BankAccountLedgerEntry.FindSet() then
		repeat
			if TimeSeriesData.Get(BankAccountLedgerEntry."Posting Date", PrevValue) then
				TimeSeriesData.Set(BankAccountLedgerEntry."Posting Date", PrevValue + BankAccountLedgerEntry."Amount (LCY)")
			else
				TimeSeriesData.Add(BankAccountLedgerEntry."Posting Date", BankAccountLedgerEntry."Amount (LCY)");
		until BankAccountLedgerEntry.Next() = 0;
end;

TimeSeriesDataToJson method convert TimeSeriesData from Dictionary to JSON object.

local procedure TimeSeriesDataToJson(TimeSeriesData: Dictionary of [Date, Decimal]): JsonObject
var
	JTimeSeriesData: JsonObject;
	JSeriesItem: JsonObject;
	JSeriesItems: JsonArray;
	PostingDate: Date;
	Value: Decimal;
begin
	foreach PostingDate in TimeSeriesData.Keys do begin
		TimeSeriesData.Get(PostingDate, Value);
		JSeriesItem.Add('timestamp', Format(PostingDate, 0, 9));
		JSeriesItem.Add('value', Format(Value, 0, 9));

		JSeriesItems.Add(JSeriesItem);
	end;

	JTimeSeriesData.Add('series', JSeriesItems);
	JTimeSeriesData.Add('maxAnomalyRatio', 0.25);
	JTimeSeriesData.Add('sensitivity', 95);
	JTimeSeriesData.Add('granularity', 'daily');
	exit(JTimeSeriesData);
end;

CheckAnomaly method takes TimeSeries data and returns anomalies in that data.

local procedure CheckAnomaly(TimeSeriesData: Dictionary of [Date, Decimal]; AnomalyData: Dictionary of [Date, Decimal]): Boolean
var
	JTimeSeriesData: JsonObject;
	JResponse: JsonObject;
begin
	JTimeSeriesData := TimeSeriesDataToJson(TimeSeriesData);
	if not GetAPIResponse(JTimeSeriesData, JResponse) then
		exit;

	exit(AnomalyValuesToDataSet(JResponse, TimeSeriesData, AnomalyData));
end;

GetAPIResponse method sends TimeSeriesData to REST API and returns the JSON response from the API.

local procedure GetAPIResponse(JTimeSeriesData: JsonObject; JResponse: JsonObject): Boolean
var
	HttpClient: HttpClient;
	RequestHeaders: HttpHeaders;
	ContentHeaders: HttpHeaders;
	ReqHttpContent: HttpContent;
	ResHttpResponseMessage: HttpResponseMessage;
	ContentTypeValues: array[1024] of Text;
	JsonText: Text;
	Url: Text;
begin
	RequestHeaders := HttpClient.DefaultRequestHeaders();
	RequestHeaders.Add('Ocp-Apim-Subscription-Key', SubscriptionKeyLbl);

	JTimeSeriesData.WriteTo(JsonText);
	ReqHttpContent.WriteFrom(JsonText);
	ReqHttpContent.GetHeaders(ContentHeaders);
	if ContentHeaders.GetValues('Content-Type', ContentTypeValues) then
		ContentHeaders.Remove('Content-Type');
	ContentHeaders.Add('Content-Type', 'application/json');

	Url := StrSubstNo(RequestUrlLbl, EndpointLbl);
	if not HttpClient.Post(Url, ReqHttpContent, ResHttpResponseMessage) then
		exit;

	if not ResHttpResponseMessage.IsSuccessStatusCode() then
		exit;

	ResHttpResponseMessage.Content.ReadAs(JsonText);
	JResponse.ReadFrom(JsonText);
	exit(true);
end;

AnomalyValuesToDataSet method reads the API response, populates the anomaly data in a Dictionary variable and returns it.

local procedure AnomalyValuesToDataSet(JResponse: JsonObject; TimeSeriesData: Dictionary of [Date, Decimal]; AnomalyData: Dictionary of [Date, Decimal]): Boolean
var
	JAnomalyValues: JsonArray;
	JToken: JsonToken;
	Index: Integer;
	PostingDate: Date;
	Amount: Decimal;
begin
	if not JResponse.Get('isAnomaly', JToken) then
		exit(false);

	JAnomalyValues := JToken.AsArray();
	foreach JToken in JAnomalyValues do begin
		Index += 1;

		if JToken.AsValue().AsBoolean() then begin
			TimeSeriesData.Keys.Get(Index, PostingDate);
			TimeSeriesData.Get(PostingDate, Amount);
			AnomalyData.Add(PostingDate, Amount);
		end;
	end;

	exit(true);
end;

Conclusion

Azure Cognitive Service is the one which provides artificial intelligence to business applications like Business Central. Obviously, it is more convenient to have a system that can detect potential threats automatically, instead of analyzing data manually. Azure Cognitive Service fulfills that need.


Source code can be downloaded from GitHub.


Happy Coding!!!


#MsDyn365 #MsDyn365BC #BusinessCentral #DynamicsNAV #AzureCognitiveService #AzureAnomalyDetector

Recent Posts

See All

© 2020 by msnJournals.com

  • Facebook Social Icon
  • LinkedIn Social Icon
  • Twitter Social Icon
  • GitHub-Mark-120px-plus