How Azure Data Factory Works Behind the Scenes.

Posted on by

Imagine you’re working as a data engineer in a company where customer orders, website activity, payment transactions, and inventory updates are generated every minute. Every night, millions of records need to be copied from different databases into a central data lake so analysts can prepare reports the next morning. You create an Azure Data Factory pipeline, click Trigger Now, and after a while, the pipeline shows a green “Succeeded” status. For many beginners, that’s where the story ends. The data has moved, the pipeline succeeded, and the dashboard looks healthy. But if someone asks, “What actually happened after you clicked Trigger?”, the answer is often unclear.

Did Azure Data Factory copy every record itself? Where did the data actually travel? What is Integration Runtime doing? Why does the same pipeline finish in ten minutes one day but take thirty minutes the next? And when a company copies hundreds of gigabytes—or even terabytes—of data every day, how does Azure Data Factory manage such large workloads without becoming a bottleneck?These are not just interview questions. They are questions every data engineer eventually faces while working on real projects.

This article is written for freshers, aspiring data engineers, ETL developers, and professionals who have started using Azure Data Factory but want to understand how it works behind the scenes, not just how to drag activities onto a canvas. By the end of this article, you’ll have a clear mental model of pipeline execution, Integration Runtime, Data Integration Units (DIUs), Copy Activity, monitoring, and the common mistakes that affect both performance and cost.

Before We Begin: Think Beyond the Pipeline Designer

Most Azure Data Factory tutorials focus on building pipelines. You create a Linked Service, define a Dataset, drag a Copy Activity, connect a source and destination, and run the pipeline. While this teaches you how to use the product, it doesn’t explain why the pipeline behaves the way it does. This is similar to learning how to drive a car without understanding what happens when you press the accelerator. You may know how to reach your destination, but when the engine starts making unusual noises or the fuel efficiency suddenly drops, you won’t know where to look. Azure Data Factory works in much the same way. Building a pipeline is relatively straightforward. Designing efficient, reliable, and cost-effective pipelines requires understanding what happens underneath. Once you understand the architecture, many confusing concepts—such as Integration Runtime, DIUs, parallel copy, retry behavior, and pipeline monitoring—begin to make sense.

One lesson I’ve learned from working on enterprise data platforms is that performance problems rarely originate where people first suspect. When a pipeline slows down, many teams immediately blame Azure Data Factory. In reality, ADF is often exposing existing problems such as slow databases, poor network connectivity, badly written SQL queries, or inefficient file layouts. Understanding the architecture helps you diagnose the real cause instead of guessing.

The Biggest Misconception About Azure Data Factory

One of the most common misconceptions among beginners is that Azure Data Factory is a data processing engine. Since it is responsible for moving data between systems, it’s easy to imagine that every file, row, and byte passes through Azure Data Factory itself. In reality, that’s not how the platform is designed.

Azure Data Factory is primarily an orchestration service. Its main responsibility is to coordinate, schedule, monitor, and manage data movement. It decides what should happen, when it should happen, and in what sequence different activities should execute. The heavy lifting is performed elsewhere.

A useful analogy is an airport control tower.

Every day, hundreds of aircraft take off and land at busy airports around the world. The control tower manages runway assignments, monitors aircraft positions, grants takeoff permissions, coordinates arrivals, and ensures flights don’t interfere with one another. Yet the control tower never flies the aircraft.

Azure Data Factory plays a very similar role. It coordinates the movement of data but doesn’t necessarily transport or process the data itself. Instead, it delegates the execution to another component specifically designed for that purpose.

That component is called the Integration Runtime, and it is arguably the single most important concept in Azure Data Factory.

Understanding this distinction immediately answers one of the most frequently asked interview questions:

Does data actually flow through Azure Data Factory?

In most scenarios, no. Azure Data Factory orchestrates the workflow, while the Integration Runtime performs the actual movement of data between the source and destination. This architecture allows ADF to scale efficiently without becoming the bottleneck, even when organizations move hundreds of gigabytes or terabytes of data every day.

What Actually Happens When You Click “Trigger Now”?

Let’s walk through a simple example. Suppose your organization has an on-premises SQL Server containing sales transactions, and every night those transactions must be copied into Azure Data Lake Storage for reporting and analytics. From the Azure portal, you manually trigger a pipeline. To you, it looks like a single click. Behind the scenes, however, Azure Data Factory begins a sequence of coordinated operations.

First, it reads the pipeline definition stored within your ADF instance. It identifies the activities that need to run, checks their execution order, and validates the dependencies between them. Next, it examines the Linked Services to determine where the source and destination systems are located and how they should be authenticated. It then loads the associated Datasets, which describe the structure and location of the data that will be copied. Only after these validations succeed does Azure Data Factory decide which Integration Runtime should execute the work. If the source and destination are Azure services, it may choose an Azure Integration Runtime. If the source resides inside a private corporate network, it may instead use a Self-Hosted Integration Runtime installed within the organization’s infrastructure. Notice something important here: at this stage, no business data has moved yet. Azure Data Factory is preparing the execution plan, validating configurations, selecting the appropriate execution engine, and coordinating the workflow. The actual data transfer hasn’t even begun.

This separation between orchestration and execution is one of the reasons Azure Data Factory scales so effectively in enterprise environments. The orchestration layer remains lightweight while specialized runtime components handle the demanding task of reading, transferring, and writing large volumes of data.

Integration Runtime: The Real Engine Behind Every Pipeline

If Azure Data Factory is the brain that decides what needs to happen, then Integration Runtime (IR) is the muscle that actually performs the work. Almost every important operation inside Azure Data Factory eventually depends on an Integration Runtime. Without it, a pipeline may exist on your screen, but it cannot read data, write data, connect to external systems, or execute integration activities.

Think of a construction company building a new office. The project manager creates the schedule, assigns tasks, tracks deadlines, and coordinates different teams. However, the manager doesn’t pour concrete, install electrical wiring, or paint the walls. Those tasks are performed by engineers, electricians, and construction workers. Azure Data Factory behaves in exactly the same way. It coordinates the workflow, while the Integration Runtime carries out the physical work. This separation of responsibilities makes Azure Data Factory highly scalable because Microsoft can improve orchestration independently from execution.

integration runtime

In interviews, candidates are often asked, “What is Integration Runtime?” Many answer by saying, “It’s a compute engine.” While technically correct, that answer barely scratches the surface. A better answer would be:

Integration Runtime is the execution infrastructure that Azure Data Factory uses to connect to data sources, authenticate requests, move data, execute integration activities, and communicate execution status back to Azure Data Factory.

Once you understand this definition, many ADF concepts start fitting together naturally.

The Three Types of Integration Runtime

Azure Data Factory provides three different Integration Runtime options because not every organization stores its data in the same place.

1. Azure Integration Runtime

This is the default runtime created automatically when you build an Azure Data Factory instance. Suppose your source is Azure SQL Database and your destination is Azure Data Lake Storage. Both systems already exist inside Microsoft’s cloud, so Azure can securely connect to them without any additional infrastructure from your side. When the pipeline starts, Microsoft provisions the necessary resources, establishes secure connections, copies the data, reports execution statistics, and automatically releases the resources when the work is complete. As developers, we never need to install servers or worry about operating system patches because Microsoft manages everything.

For most cloud-to-cloud integrations, Azure Integration Runtime is the preferred choice because it offers simplicity, scalability, and minimal administrative effort.

2. Self-Hosted Integration Runtime

Now consider a different scenario. A manufacturing company stores production data inside an SQL Server located in its own data center. That database sits behind a corporate firewall and is not exposed to the internet for obvious security reasons. If Azure Data Factory attempted to connect directly, the connection would fail because Azure has no route into the organization’s private network. This is where Self-Hosted Integration Runtime (SHIR) becomes essential. Instead of Azure entering the corporate network, the organization installs a lightweight Integration Runtime service on one of its own Windows servers. That server establishes an outbound encrypted connection to Azure, which is much safer than opening inbound firewall ports. When a pipeline executes, Azure Data Factory sends instructions to the Self-Hosted Integration Runtime. The runtime connects to the local SQL Server, reads the required data, securely transfers it to Azure, and reports the execution status back to Azure Data Factory. You can think of Self-Hosted Integration Runtime as a trusted courier who already has access to the building. Azure doesn’t need to force its way inside—it simply hands the courier a list of instructions.

One common mistake I see among beginners is assuming Self-Hosted Integration Runtime stores data permanently. It doesn’t. It simply acts as a secure execution bridge between your private environment and Azure.

3. Azure-SSIS Integration Runtime

Many enterprises have spent years developing SQL Server Integration Services (SSIS) packages. Rewriting hundreds of packages into Azure Data Factory would require significant time and effort. Azure-SSIS Integration Runtime solves this migration challenge by allowing organizations to execute existing SSIS packages inside Azure without redesigning every workflow from scratch.

Although modern cloud-native projects often rely on Copy Activity, Data Flows, or Databricks, Azure-SSIS Integration Runtime continues to play an important role for organizations modernizing legacy ETL systems.

What Are Data Integration Units (DIUs)?

Now let’s talk about one of the most frequently asked Azure Data Factory interview topics that many introductory articles completely ignore.

Data Integration Units (DIUs).

Whenever a Copy Activity executes using Azure Integration Runtime, Azure Data Factory allocates compute resources behind the scenes. Those compute resources are measured using Data Integration Units, commonly known as DIUs. A simple way to think about DIUs is to imagine a team unloading a truck full of packages. If only two workers are unloading boxes, the truck takes longer to empty. If ten workers unload the same truck simultaneously, the work finishes much faster. DIUs work in a similar way. They represent the processing power Azure allocates for a Copy Activity. In many scenarios, increasing DIUs allows Azure Data Factory to read data faster, transform it more efficiently during transfer, and write it to the destination more quickly. However, this is where experience becomes important. Many beginners believe increasing DIUs automatically makes every pipeline faster. That isn’t always true. Imagine you’re filling a bucket using a narrow garden hose. Replacing the person holding the bucket won’t increase the water flow because the hose itself is the limitation.

Similarly, if your SQL Server can only deliver data at 50 MB per second, allocating additional DIUs won’t magically increase the database’s throughput. The bottleneck exists at the source, not inside Azure Data Factory.

This is an important production lesson:

Increasing compute cannot solve every performance problem. First identify where the bottleneck actually exists.

From a cost perspective, DIUs also matter because allocating more compute resources generally increases the execution cost of your pipeline. Optimizing DIUs is therefore not just about performance—it’s also about spending cloud resources wisely.

How Copy Activity Actually Moves Your Data

Copy Activity is probably the most frequently used activity in Azure Data Factory, but most tutorials explain it in only one sentence: “It copies data from source to destination.” While true, it hides a lot of interesting engineering happening behind the scenes. Imagine a retail company that wants to copy yesterday’s sales transactions from SQL Server into Azure Data Lake every night. The moment the Copy Activity starts, Azure Data Factory first retrieves the connection information from the Linked Services. Instead of hardcoding usernames and passwords inside the pipeline, authentication details are often stored securely using Azure Key Vault or managed identities. This improves security and avoids exposing sensitive credentials inside pipeline definitions. Once authentication succeeds, the selected Integration Runtime establishes connections with both the source and destination systems. It begins reading data from the source, often using multiple parallel readers depending on the source type and configuration. Rather than moving one record at a time, Azure Data Factory attempts to maximize throughput by transferring data efficiently in batches.

As the data flows, the Integration Runtime monitors throughput, network performance, retry attempts, and any transient failures. If a temporary network interruption occurs, Azure Data Factory can automatically retry the operation according to the configured retry policy instead of immediately marking the pipeline as failed. Finally, once all records have been written successfully, execution statistics such as rows copied, data volume transferred, throughput, execution duration, and DIU utilization are sent back to Azure Data Factory, where they become visible in the monitoring dashboard. From the user’s perspective, all they see is a green “Succeeded” icon. Behind that single icon, however, Azure Data Factory has coordinated authentication, connection management, resource allocation, parallel processing, retry logic, monitoring, and detailed logging.

That is why understanding Copy Activity is far more valuable than simply memorizing its configuration options.

How Azure Data Factory Moves 100 GB or Even 1 TB of Data

One question that surprises many beginners is:

“If Azure Data Factory is only orchestrating the pipeline, how does it move hundreds of gigabytes—or even terabytes—of data without becoming painfully slow?”

The answer lies in parallelism. Imagine you need to move 10,000 boxes from one warehouse to another. If a single worker carries one box at a time, the work might take an entire day. Hiring a stronger worker won’t necessarily solve the problem because one person can only carry so much. Now imagine twenty workers moving boxes simultaneously. The total amount of work hasn’t changed, but the time required drops dramatically because the work is distributed. Azure Data Factory follows a very similar principle. Instead of reading one row after another, Copy Activity can read multiple chunks of data simultaneously, provided the source system supports parallel reads. Likewise, the destination can receive multiple streams of data concurrently. The Integration Runtime coordinates these parallel operations, allowing very large datasets to be transferred efficiently.

For example, suppose you need to copy a 100 GB sales table from Azure SQL Database to Azure Data Lake Storage. If the source database allows multiple parallel readers and the destination supports concurrent writes, Azure Data Factory can split the workload across several threads. Instead of waiting for one long sequential transfer, multiple portions of the dataset are copied at the same time.

In many real-world scenarios, increasing parallelism can reduce execution time significantly. However, this doesn’t mean every pipeline should simply use the maximum possible number of parallel threads. More parallelism also means more pressure on the source database, increased network utilization, and higher compute consumption. A busy production SQL Server that already serves thousands of application users might become slower if dozens of parallel readers suddenly start scanning large tables.

Experienced engineers therefore treat parallelism as a balancing act rather than a race. The objective isn’t to consume the maximum available resources—it is to finish the workload efficiently without negatively affecting other systems.

When More DIUs Will Not Make Your Pipeline Faster

One of the most common misconceptions in Azure Data Factory is that increasing Data Integration Units automatically improves performance. This assumption sounds reasonable because additional compute resources generally help in many cloud services. Unfortunately, data integration doesn’t always work that way. Imagine you’re downloading a large file using an internet connection that delivers only 50 Mbps. Replacing your laptop with a much faster computer won’t suddenly increase your download speed because the internet connection—not the computer—is the limiting factor. Exactly the same principle applies to Azure Data Factory. Suppose your source system is an on-premises SQL Server connected through a slow VPN. Even if you double the DIUs, the SQL Server can still send data only as fast as the network allows. The bottleneck remains outside Azure Data Factory. Similarly, poorly written SQL queries can become another hidden bottleneck. If your Copy Activity executes a query that scans millions of unnecessary rows or performs expensive joins before returning results, Azure Data Factory must patiently wait for the database to finish. Increasing DIUs won’t improve the query itself.

This is one of the biggest differences between beginners and experienced data engineers.

Beginners often ask:

“How many DIUs should I allocate?”

Experienced engineers ask:

“Where is the bottleneck?”

That single mindset shift can save both time and cloud costs.

Before increasing DIUs, always investigate:

  • Is the source database responding slowly?
  • Is the network bandwidth limited?
  • Is the destination storage throttling writes?
  • Is the SQL query optimized?
  • Is the Integration Runtime under heavy load?

Only after identifying the real bottleneck should you consider allocating additional compute resources.

Why Do Azure Data Factory Pipelines Become Slow?

Every data engineer eventually encounters a pipeline that suddenly takes twice as long to complete. In many cases, Azure Data Factory itself is blamed, even though the underlying cause lies elsewhere. One of the most common reasons is the source system. If the database is busy serving application users during business hours, analytical queries executed by Azure Data Factory may receive fewer resources, increasing overall pipeline duration. Network latency is another frequent culprit. Organizations often copy data between different Azure regions or from on-premises environments connected through VPNs. Even with excellent pipeline design, slow network links can significantly reduce throughput. Another overlooked factor is the Self-Hosted Integration Runtime server itself. Because this runtime executes on infrastructure managed by the customer, insufficient CPU, memory, or disk performance can become a limiting factor. Installing Self-Hosted Integration Runtime on an underpowered virtual machine is a surprisingly common production mistake. Pipeline design also plays an important role. Many beginners copy unnecessary columns simply because it’s easier to use **SELECT ***. While convenient during development, this increases network traffic, storage usage, and execution time. Reading only the columns actually required by downstream systems almost always results in better performance. Finally, inefficient source queries frequently become the hidden bottleneck. Copy Activity can only transfer data after the database returns it. If the database spends fifteen minutes generating the result set, Azure Data Factory has no choice but to wait.

One valuable lesson I’ve learned over the years is this:

Azure Data Factory usually exposes performance problems that already exist. It rarely creates them.

Understanding this distinction prevents countless hours of unnecessary troubleshooting.

Monitoring Pipelines Like a Production Engineer

Creating a pipeline is only the beginning. Production environments require continuous monitoring to ensure pipelines remain healthy, reliable, and efficient. Fortunately, Azure Data Factory provides a dedicated Monitor section that offers detailed execution information for every triggered pipeline.

To access it:

Azure Portal → Azure Data Factory → Monitor → Pipeline Runs

From here, you can inspect every pipeline execution and drill down into individual activities. Instead of simply checking whether a pipeline succeeded or failed, experienced engineers pay attention to much richer information. For example, they examine how long each activity required, how much data was transferred, the achieved throughput, retry attempts, execution duration, and any warning or error messages returned by the source or destination systems. Imagine a pipeline that normally completes in twelve minutes but suddenly begins taking thirty minutes every day. Without monitoring, identifying the root cause becomes guesswork. By reviewing execution history, you may discover that the Copy Activity duration remained unchanged while a SQL query executed before it suddenly doubled in execution time. The issue isn’t Azure Data Factory at all—it’s the database. This ability to isolate performance bottlenecks is one reason experienced data engineers spend almost as much time inside the Monitor page as they do inside the pipeline designer.

Monitoring should therefore become part of your daily routine rather than something you check only after failures occur.

Common Mistakes Beginners Make in Azure Data Factory

Most Azure Data Factory issues are not caused by complex architecture. They usually stem from a handful of recurring mistakes that appear in project after project. The first mistake is treating Azure Data Factory as a data processing engine instead of an orchestration platform. This misunderstanding leads to unrealistic expectations about performance and scalability. Another common mistake is increasing DIUs without first identifying the actual bottleneck. More compute cannot compensate for slow databases, poor SQL queries, or limited network bandwidth. Many beginners also overlook monitoring. They celebrate successful pipeline executions without ever checking throughput, execution duration, retry counts, or activity-level statistics. As pipelines grow larger, this habit makes troubleshooting significantly more difficult. Finally, developers often design pipelines that work perfectly with a few thousand records during testing but struggle once millions of rows enter production. Designing for production scale from the beginning almost always produces more reliable solutions. If you remember only one lesson from this article, let it be this:

Azure Data Factory is excellent at orchestrating data movement, but successful pipelines depend just as much on good database design, healthy infrastructure, efficient networking, and thoughtful pipeline architecture as they do on Azure Data Factory itself.

When people first start learning Azure Data Factory, they naturally focus on building pipelines because that’s the most visible part of the platform. However, as projects grow larger, success depends far less on dragging activities onto a canvas and far more on understanding how the platform behaves behind the scenes. The biggest lesson from this article isn’t how to configure a Copy Activity or when to choose a particular Integration Runtime. It’s understanding that Azure Data Factory is an orchestrator, not a data processing engine. Once that mental model clicks, many seemingly unrelated concepts—from DIUs and monitoring to pipeline performance and troubleshooting—begin to fit together naturally. If you’re serious about becoming a Data Engineer, don’t stop at creating pipelines that “work.” Spend time understanding why they work, how they scale, and what happens when they don’t. That’s the difference between someone who knows the Azure portal and someone who can confidently design production-grade data integration solutions.

As a practical next step, create a simple Azure Data Factory pipeline that copies a CSV file from Azure Blob Storage to Azure Data Lake Storage. After running it, open the Monitor section in the Azure portal and inspect the execution details. Look at the activity duration, throughput, Integration Runtime used, and execution metrics. Seeing these concepts in action will reinforce everything you’ve learned in this article far better than simply reading documentation.

Total Views 14