Azure Data Factory (ADF) : A Complete Guide with Real-Time Examples

 

Working with Azure Data Factory (ADF): A Complete Guide with Real-Time Examples

Introduction

In today's data-driven world, organizations collect data from multiple sources such as SQL databases, ERP systems, CRM applications, APIs, IoT devices, cloud storage, and third-party applications. However, collecting data is only the first step. The real challenge is moving, transforming, validating, and loading that data into a centralized repository for reporting, analytics, and machine learning.

This is where Azure Data Factory (ADF) comes into the picture.

Azure Data Factory is Microsoft's cloud-based data integration service that enables you to create, schedule, automate, and monitor data pipelines without writing large amounts of custom code.

In this article, we'll explore Azure Data Factory in detail, understand its architecture, components, and work through real-world enterprise examples.

---

What is Azure Data Factory?

Azure Data Factory (ADF) is a cloud ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) service provided by Microsoft Azure.

It allows organizations to:

• Move data between systems

• Transform data

• Schedule workflows

• Automate data processing

• Integrate on-premises and cloud data

• Build enterprise-grade data pipelines

Think of Azure Data Factory as the orchestrator of your data ecosystem.

Instead of manually copying files every day, ADF automates everything.

---

Why Do We Need Azure Data Factory?

Consider a retail company.

Every day data is generated from:

• Sales Database

• Inventory Database

• SAP ERP

• CRM

• Excel Files

• CSV Files

• APIs

• Azure Blob Storage

• Amazon S3

• Oracle Database

Management wants a dashboard every morning.

Without ADF:

• Developers manually write scripts

• Jobs fail frequently

• No monitoring

• Difficult maintenance

With ADF:

• Automated pipelines

• Error handling

• Monitoring

• Alerts

• Scheduling

• Scalability

---

Azure Data Factory Architecture

                SQL Server
                     |
                Oracle DB
                     |
                REST APIs
                     |
             Azure Blob Storage
                     |
             Amazon S3 Storage
                     |
          ------------------------
          | Azure Data Factory |
          ------------------------
             /      |       \
      Copy Data  Transform  Monitor
           |         |         |
        Azure SQL  Databricks  Logs
           |
     Power BI Reports

ADF acts as the central orchestrator.

---

Core Components of Azure Data Factory

1. Pipeline

A Pipeline is a logical grouping of activities.

Example:

Read CSV

Validate Data

Transform Data

Load into Azure SQL

Send Email

All these activities together form one pipeline.

---

2. Activity

Activities are individual tasks.

Examples:

• Copy Activity

• Lookup Activity

• Stored Procedure Activity

• Data Flow Activity

• Execute Pipeline

• Web Activity

• Azure Function Activity

Think of activities as individual functions inside a program.

---

3. Linked Service

Linked Service stores connection information.

Example:

Azure SQL Connection

Server:
company.database.windows.net

Database:
SalesDB

Username

Password

ADF uses Linked Services to connect to data sources.

Supported sources include:

• Azure SQL

• SQL Server

• Oracle

• SAP

• Blob Storage

• ADLS

• Snowflake

• Amazon S3

• REST APIs

• MySQL

• PostgreSQL

• MongoDB

---

4. Dataset

Dataset represents the data you want to use.

Example:

Sales.csv

Customer Table

Blob Folder

Excel File

Linked Service = Connection

Dataset = Actual Data

---

5. Integration Runtime (IR)

Integration Runtime is the compute engine that moves data.

Three types:

Azure Integration Runtime

For cloud-to-cloud data movement.

Example:

Azure Blob → Azure SQL

---

Self-hosted Integration Runtime

Used when data is inside your company network.

Example:

On-Prem SQL Server → Azure SQL

Install Self-hosted IR on a local server.

---

Azure SSIS Integration Runtime

Used for running existing SSIS packages in Azure.

---

ETL vs ELT

ETL

Extract

↓

Transform

↓

Load

Transformation happens before loading.

---

ELT

Extract

↓

Load

↓

Transform

Transformation happens after loading.

Azure Data Factory supports both.

---

Real-Time Example 1

Daily Sales Data Migration

Company:

ABC Retail

Requirement:

Every night:

• Read Sales data

• Validate

• Remove duplicates

• Load into Data Warehouse

Pipeline:

Trigger

↓

Copy Sales Data

↓

Data Flow

↓

Remove Duplicates

↓

Load Azure SQL

↓

Success Email

Result:

Automated daily process.

---

Real-Time Example 2

Banking Industry

Every night:

• Customer Accounts

• Transactions

• Loan Data

• Credit Card Data

Need to move into Data Warehouse.

Pipeline:

Oracle

↓

ADF Copy

↓

Azure Data Lake

↓

Databricks

↓

Azure Synapse

↓

Power BI

---

Real-Time Example 3

Healthcare

Hospitals receive:

• Patient Records

• Laboratory Results

• Pharmacy Data

• Insurance Data

ADF:

SQL Server

↓

Validate

↓

Encrypt Sensitive Columns

↓

Store in Data Lake

↓

Analytics

---

Real-Time Example 4

E-Commerce

Every hour:

• Orders

• Payments

• Delivery Status

Pipeline:

REST API

↓

ADF

↓

Blob Storage

↓

Azure SQL

↓

Dashboard

---

Copy Activity

Most commonly used activity.

Example:

Source:

SQL Server

Destination:

Azure Blob Storage

ADF copies data without writing code.

---

Data Flow

Data Flow performs transformations visually.

Operations include:

• Filter

• Sort

• Join

• Aggregate

• Derived Columns

• Lookup

• Union

• Pivot

• Unpivot

• Conditional Split

• Window Functions

No coding required.

---

Parameters in Pipeline

Instead of hardcoding values:

Customer2025.csv

Use:

@pipeline().parameters.FileName

Now the same pipeline works for:

Customer2024.csv

Customer2025.csv

Customer2026.csv

---

Variables

Variables store temporary values.

Example:

CurrentDate

FileCount

Status

RowCount

---

Control Flow Activities

ADF provides many workflow activities:

• If Condition

• Switch

• Until

• ForEach

• Wait

• Execute Pipeline

• Fail

• Validation

Example:

If File Exists

↓

Copy

Else

Send Email

---

Triggers

Pipelines can execute automatically.

Types:

Schedule Trigger

Run every day.

Daily 2 AM

---

Tumbling Window

Useful for time-series processing.

---

Event Trigger

Automatically executes when a new file arrives.

Example:

Customer.csv uploaded

↓

Pipeline Starts

---

Monitoring

ADF provides monitoring dashboards showing:

• Running Pipelines

• Failed Pipelines

• Duration

• Success Rate

• Retry Count

• Error Messages

Alerts can be integrated with Azure Monitor.

---

Security

ADF supports enterprise-grade security:

• Azure Key Vault integration

• Managed Identity

• RBAC

• Private Endpoints

• Encryption at Rest

• Encryption in Transit

Avoid storing passwords directly in Linked Services. Use Azure Key Vault whenever possible.

---

Best Practices

• Use parameterized pipelines.

• Separate development, testing, and production environments.

• Store secrets in Azure Key Vault.

• Implement retry policies for transient failures.

• Enable monitoring and alerts.

• Use naming conventions for pipelines and datasets.

• Optimize parallelism for large data loads.

• Use Incremental Loading instead of Full Loading whenever possible.

---

Real Enterprise Scenario

A multinational retail company receives:

• 500 CSV files daily

• SAP data every hour

• Oracle data every night

• CRM data through REST APIs

• SQL Server transactions every 30 minutes

ADF orchestrates the process:

Blob Storage
        │
Oracle DB
        │
SQL Server
        │
REST API
        │
SAP
        │
   Azure Data Factory
        │
    Validation
        │
 Azure Data Lake
        │
 Azure Databricks
        │
 Azure Synapse Analytics
        │
     Power BI

Business users start their day with up-to-date dashboards, while data engineers benefit from automated, monitored, and scalable pipelines.

---

Advantages of Azure Data Factory

• Fully managed cloud service

• No infrastructure maintenance

• Visual pipeline designer

• Supports 100+ connectors

• Highly scalable

• Enterprise-grade security

• Built-in scheduling and monitoring

• Easy integration with Azure services

• Supports hybrid cloud environments

---

Common Interview Questions

What is Azure Data Factory?

Azure Data Factory is a cloud-based data integration service that automates data movement and transformation across cloud and on-premises systems.

What is a Pipeline?

A pipeline is a logical container that groups one or more activities into a complete workflow.

What is the difference between Linked Service and Dataset?

A Linked Service defines the connection to a data source, while a Dataset represents the specific data (such as a table, file, or folder) within that source.

What is Integration Runtime?

Integration Runtime is the compute infrastructure used by Azure Data Factory to move and transform data. It is available as Azure Integration Runtime, Self-hosted Integration Runtime, and Azure SSIS Integration Runtime.

What is the difference between ETL and ELT?

ETL transforms data before loading it into the destination. ELT loads raw data first and performs transformations afterward, typically using the compute capabilities of the target system.

---

Conclusion

Azure Data Factory is one of the most powerful cloud-native data integration services available today. It enables organizations to build secure, scalable, and automated data pipelines that connect virtually any data source to modern analytics platforms. Whether you're migrating on-premises databases, orchestrating enterprise ETL workflows, integrating SaaS applications, or feeding business intelligence dashboards, ADF simplifies the entire process with minimal code and rich monitoring capabilities.

For modern data engineering teams working in Azure, Azure Data Factory is a foundational service that plays a key role in building reliable and enterprise-ready data platforms.

Azure Data Factory Connectors

How to Remove Duplicate Values from an Array in C#

How to Remove Duplicate Values from an Array in C# – 6 Best Approaches with Examples

Removing duplicate values from an array is one of the most common tasks in C# programming. Whether you're processing user input, reading data from a database, or preparing data for reports, eliminating duplicate values improves data quality and application performance.

In this article, we'll explore multiple ways to remove duplicate values from an array in C#, ranging from beginner-friendly approaches to optimized solutions suitable for production applications and technical interviews.

---

Problem Statement

Suppose we have the following array:

int[] iArray = new int[] { 1, 1, 2, 2, 3, 3, 4, 5, 6, 6 };

Expected Output

1, 2, 3, 4, 5, 6

---

Method 1: Using LINQ Distinct() (Recommended)

The easiest and most readable approach is to use LINQ's Distinct() method.

using System;
using System.Linq;

class Program
{
    static void Main()
    {
        int[] iArray = { 1, 1, 2, 2, 3, 3, 4, 5, 6, 6 };

        int[] uniqueArray = iArray.Distinct().ToArray();

        Console.WriteLine(string.Join(", ", uniqueArray));
    }
}

Output

1, 2, 3, 4, 5, 6

Advantages

• Very easy to write

• Highly readable

• Excellent for production code

• Preserves the first occurrence order

Disadvantages

• Requires LINQ

• Slightly more memory usage than in-place algorithms

Time Complexity: O(n)

Space Complexity: O(n)

---

Method 2: Using HashSet (Fastest)

A HashSet<T> stores only unique values. Any duplicate values are automatically ignored.

using System;
using System.Collections.Generic;

class Program
{
    static void Main()
    {
        int[] iArray = { 1, 1, 2, 2, 3, 3, 4, 5, 6, 6 };

        HashSet<int> uniqueNumbers = new HashSet<int>(iArray);

        foreach (int item in uniqueNumbers)
        {
            Console.Write(item + " ");
        }
    }
}

Output

1 2 3 4 5 6

Advantages

• Extremely fast

• Automatically removes duplicates

• Excellent for large collections

Disadvantages

• Uses additional memory

• Order is not guaranteed in all scenarios

Time Complexity: O(n)

Space Complexity: O(n)

---

Method 3: Using List Without LINQ

This approach is useful for beginners or interview situations where LINQ is not allowed.

using System;
using System.Collections.Generic;

class Program
{
    static void Main()
    {
        int[] iArray = { 1, 1, 2, 2, 3, 3, 4, 5, 6, 6 };

        List<int> result = new List<int>();

        foreach (int item in iArray)
        {
            if (!result.Contains(item))
            {
                result.Add(item);
            }
        }

        Console.WriteLine(string.Join(", ", result));
    }
}

Advantages

• Easy to understand

• No LINQ dependency

Disadvantages

• Contains() performs a linear search

• Not suitable for large datasets

Time Complexity: O(n²)

---

Method 4: Using Dictionary

A dictionary can also be used to keep track of values that have already been processed.

using System;
using System.Collections.Generic;

class Program
{
    static void Main()
    {
        int[] iArray = { 1, 1, 2, 2, 3, 3, 4, 5, 6, 6 };

        Dictionary<int, bool> dictionary = new Dictionary<int, bool>();

        foreach (int item in iArray)
        {
            if (!dictionary.ContainsKey(item))
            {
                dictionary.Add(item, true);
            }
        }

        foreach (var item in dictionary.Keys)
        {
            Console.Write(item + " ");
        }
    }
}

Advantages

• Fast lookup

• Good alternative to HashSet

Disadvantages

• Stores unnecessary values (bool)

• More verbose than HashSet

Time Complexity: O(n)

---

Method 5: Using Nested Loops (Without Collections)

This method is frequently asked during coding interviews because it demonstrates your understanding of the logic without relying on built-in collection classes.

using System;

class Program
{
    static void Main()
    {
        int[] iArray = { 1, 1, 2, 2, 3, 3, 4, 5, 6, 6 };

        for (int i = 0; i < iArray.Length; i++)
        {
            bool duplicate = false;

            for (int j = 0; j < i; j++)
            {
                if (iArray[i] == iArray[j])
                {
                    duplicate = true;
                    break;
                }
            }

            if (!duplicate)
            {
                Console.Write(iArray[i] + " ");
            }
        }
    }
}

Advantages

• No LINQ

• No collections

• Demonstrates algorithmic thinking

Disadvantages

• Slow for large datasets

Time Complexity: O(n²)

Space Complexity: O(1)

---

Method 6: Remove Consecutive Duplicates from a Sorted Array

If the array is already sorted, you only need to compare each element with the previous one.

using System;
using System.Collections.Generic;

class Program
{
    static void Main()
    {
        int[] iArray = { 1, 1, 2, 2, 3, 3, 4, 5, 6, 6 };

        List<int> result = new List<int>();

        if (iArray.Length > 0)
        {
            result.Add(iArray[0]);

            for (int i = 1; i < iArray.Length; i++)
            {
                if (iArray[i] != iArray[i - 1])
                {
                    result.Add(iArray[i]);
                }
            }
        }

        Console.WriteLine(string.Join(", ", result));
    }
}

Advantages

• Very fast for sorted arrays

• Simple implementation

Disadvantages

• Works correctly only when duplicates are adjacent

Time Complexity: O(n)

---

Performance Comparison

Method	Time Complexity	Space Complexity	Best Use Case
LINQ Distinct()	O(n)	O(n)	General-purpose applications
HashSet	O(n)	O(n)	Large datasets and high performance
List + Contains()	O(n²)	O(n)	Learning and small datasets
Dictionary	O(n)	O(n)	Alternative to HashSet
Nested Loops	O(n²)	O(1)	Coding interviews
Sorted Array Comparison	O(n)	O(n)	Already sorted arrays

---

Real-World Use Cases

Removing duplicates is useful in many real-world applications, including:

• Removing duplicate customer IDs

• Eliminating repeated product codes

• Cleaning imported CSV or Excel data

• Processing unique email addresses

• Generating distinct reports

• Removing duplicate API response records

• Preparing analytics data

• Data migration and ETL processes

---

Interview Questions

1. What is the easiest way to remove duplicates from an array in C#?

Use Distinct() from LINQ.

---

2. Which collection automatically removes duplicates?

HashSet<T>.

---

3. What is the complexity of Distinct()?

• Time Complexity: O(n)

• Space Complexity: O(n)

---

4. Can we remove duplicates without using LINQ?

Yes. We can use:

• HashSet

• Dictionary

• List

• Nested loops

---

5. Which approach is preferred in production?

For most applications:

• Distinct() for readability

• HashSet<T> for maximum performance

---

6. Which approach is commonly asked in coding interviews?

The nested-loop solution is a popular interview question because it tests your understanding of algorithms without relying on built-in methods.

---

Best Practices

• Use Distinct() for clean, readable, and maintainable code.

• Use HashSet<T> when performance is critical.

• Avoid List.Contains() for large datasets because it results in quadratic time complexity.

• Understand the manual nested-loop solution for coding interviews.

• Choose the approach that best matches your application's performance and readability requirements.

---

Conclusion

Removing duplicate values from an array is a common programming task in C#. The language offers several approaches, each with its own strengths.

• Use Distinct() for simplicity and maintainability.

• Use HashSet<T> for the best performance with large collections.

• Learn the nested-loop solution to strengthen your problem-solving skills for interviews.

• Select the right approach based on your application's size, performance requirements, and coding standards.

Mastering these techniques will help you write cleaner, more efficient C# code and prepare you for technical interviews with confidence.

If you'd like, I can also create:

• an SEO-optimized version (targeting keywords like "Remove Duplicates from Array in C#"),

• a professional feature image/thumbnail,

• and schema-ready FAQ markup to improve your blog's Google search visibility.