Do you know how to enable Azure Data Factory alerts in case of failure? This is one of the great capabilities to consider when developing solutions.
In this blog post, you will learn how to:
- Create and configure alerts for Azure Data Factory
- Monitor alerts
- Opt-out an action group
You can create the alerts by using Azure monitor alerts. You can access them from your Azure Data Factory or Azure Alerts.
New Alert Rule
First, click “New alert rule” to get started.
When creating alerts, you need to configure four different options:
- Scope – the service(s) that you are monitoring.
- Condition – the rule that needs to be met to trigger the alert.
- Action group – the action to execute after the alert is triggered.
- Alert rule details – the name and description of the alert.
Scope
If you have entered the alerts option from your Azure Data Factory, the information will be filled in automatically.
However, if you have entered the alerts from Azure Monitor Alerts, you have to select the services that you want to monitor.
Condition
When configuring the condition, all the metrics available for Azure Data Factory are going to be displayed. The number of metrics has increased significantly during the past few years, as well as analytics for information in the log.
In this case, to enable the alerts in case of failure, just filter by failed and select “Failed pipeline runs metrics.” This will cover the majority of failures as data flows and SSIS packages run within pipelines.
When selecting the metric, you need to configure how to aggregate the information and the frequency of evaluation. Then, click done.
In the following picture, I have selected the alert to evaluate failures every 15 minutes (for the previous 15 minutes.)
Additionally, you can filter by specific pipeline names of error type if required. This is really useful when you require more control over the criticality of your data movement.
Filtering by pipeline name
Filtering by error type
Finally, after saving the condition, you will see that it is available in the configuration page.
Action Group
Action groups are independent objects from alerts and you can reuse them.
An action group can be defined by using the following option:
Or in this case, directly from the alert.
For the action group, you must configure a few options.
- Basics – basic details of the action group
- Notifications – delivery notification method (email, SMS, call, etc.)
- Actions – an additional action to trigger (for example Azure functions or Webhooks)
- Tags
- Review and create
Basics
Let’s define some basic information for the action group.
Notifications
In this case, I am the only member of this action group and will configure the email and SMS delivery options. Each delivery method has different costs.
Actions
For this tutorial, I won’t define any actions, but the option is available to you.
Tags, review and create
Once you are ready, create the action group.
Once you are back on the alerts configuration page, you will see the action group. It’s also possible to define multiple action groups.
Alert rule details
Finally, the last step is to define a name, a description, and the severity level for your alert.
There are different severity levels defined by Microsoft in Azure:
- 0 = Critical
- 1 = Error
- 2 = Warning
- 3 = Informational
- 4 = Verbose
Testing the alert
To test the alerts, I have a simple pipeline with a lookup activity that tries to divide by zero.
After triggering the alert, I received the following notifications:
SMS
Monitoring the alerts
In case you want to know which alerts have been triggered, you can do this by going to the alerts option.
Did someone accidentally add you to an action group?
When you’re added to an action group, you will receive a notification. You can disable it if you don’t want to receive any alerts.
SMS
An option for disabling the alerts is available in the alert configuration.
Summary
In this blog post, we reviewed how to easily enable alerts for Azure Data Factory failures using the different metrics available within Azure Monitor.
Final Thoughts
Complementing your Azure Data Factory solutions with good monitoring capabilities is a practice that you need to take into account for new or mature environments.
Having alerts helps you respond to critical and non-critical issues in production environments.
What’s Next?
In upcoming blog posts, we’ll continue to explore some of the features within Azure Services.
As always, please leave any comments or questions below.
If you haven’t already, you can follow me on Twitter for blog updates, virtual presentations, and more!
Check out these other blog posts:
