Tech Talk Corner Sign up with your email address to be the first to know about new publications

Monitor Azure Data Factory with Annotations (Tags)

Did you know that you can monitor Azure Data Factory pipelines by using annotations? Azure Data Factory annotations help you easily filter different Azure Data Factory objects based on a tag. You can define tags so you can see their performance or find errors faster. 

Using annotations is a best practice for developing Azure Data Factory solutions. They complement  Azure Data Factory User Properties that are more dynamic. 

Azure Data Factory annotations are static and the values will not change during runtime.

Post Contents: 

  • Why should I use Azure Data Factory annotations? 
  • What are Azure Data Factory annotations?
  • Create Azure Data Factory Annotations 
  • Monitor Azure Data Factory Pipelines with Annotations 

Why should I use Azure Data Factory annotations? 

To begin, the easier it is to find the error, the faster you can fix it. Monitoring Azure Data Factory with annotations is a tool that enhances monitoring experience. 

Now, imagine that it’s 3 am and you have received an SMS or a call because one of your company’s Azure Data Factory mission-critical data movements has failed. You need to find out what failed and what to do to fix it (I have been in this position; I know how it feels). 

On top of that, imagine that your environment has hundreds of pipelines running on different schedules at the same time. 

You might waste valuable time just to find the error. 

What are Azure Data Factory annotations? 

Azure Data Factory annotations are tags that you can add to your Azure Data Factory components to identify them.  A tag allows you to classify or group different objects in order to easily monitor them after an execution. You can create multiple Azure Data Factory annotations. 

In addition, you can add annotations to most Azure Data Factory components. 

Pipeline example 

Pipeline example using Azure Data Factory annotations

Dataset example 

Dataset example using Azure Data Factory annotations

Create Azure Data Factory Annotations 

First, let’s add Azure Data Factory annotations to a component.  

In any dataset, click the Properties option and add a few annotations. 

Click properties and add annotations

Then, do the same for the Linked Service. 

Add annotations for Linked services

Now, save all the changes. 

Let’s test them! 

Monitor Azure Data Factory Pipelines with Annotations 

Now, let’s monitor the pipelines by using Azure Data Factory annotations. Execute any pipeline where you have added annotations. 

Go to the Monitor Hub and click annotations and add filter. 

Add annotation filter in Monitor Hub

Azure Data Factory will link different annotations from different pipeline components. 

You can also use the search box to find items related to your annotations. 

Also, it’s possible to enhance the Azure Data Factory governance by using Azure Log Analytics

Use Azure Log Analytics

Summary 

In summary, Azure Data Factory annotations are one more tool to minimise the impact on your sleeping patterns when you are the one looking after the error of mission-critical Azure Data Factory data movements. 

What’s Next?  

In upcoming blog posts, we’ll continue to explore some of the features within Azure Data Services.    

As always, please leave any comments or questions below.  

If you haven’t already, you can follow me on Twitter for blog updates, virtual presentations, and more!  

Check out these other blog posts: 

comment [ 0 ]
share
No tags 0

No Comments Yet.

Do you want to leave a comment?

Your email address will not be published. Required fields are marked *