What Is Hadoop? Architecture, Components & Uses

The useful way to read Apache Hadoop today is as a layered data platform, not a single engine. Storage, resource management, and batch processing can evolve independently. An existing Hadoop environment can therefore keep HDFS and YARN while another engine performs the computation. That separation connects Hadoop's architecture to both its current role and its limits.

This guide defines Hadoop, follows a dataset through the system, and examines each layer. It then explains where Hadoop remains useful and why teams may pair its storage and resource-management layers with engines such as Spark.

What is Hadoop?

Apache Hadoop is a distributed data platform designed around two related problems: storing large files reliably across multiple machines and moving computation close to that data. Its original processing model, MapReduce, breaks a batch job into independent tasks that can run in parallel. Its file system, HDFS, breaks files into blocks and stores those blocks across DataNodes. YARN provides a shared resource-management layer so multiple applications and processing frameworks can use the same cluster.

The design assumes that individual disks, servers, and tasks will fail. HDFS maintains redundant block copies and creates replacements when replicas become unavailable. MapReduce can rerun failed tasks. YARN monitors application processes and containers. Fault recovery is therefore part of the software architecture rather than a property expected from every machine.

Hadoop is not a relational database, a data format, or a single query engine. HDFS stores bytes without imposing tables or schemas. Tools above Hadoop supply those abstractions. Hive can describe HDFS files as tables, HBase can provide distributed key-value access, and Spark can process HDFS data through YARN. These tools do not require MapReduce as their compute engine.

Common Hadoop use cases

Batch data transformation. Distributed engines can clean, join, aggregate, and rewrite large HDFS datasets into analytical data.

Log and event analysis. HDFS can retain logs, clickstreams, and telemetry for historical scans that do not require immediate results.

On-premises data lakes. HDFS can hold raw and curated data when infrastructure or residency requirements favor self-managed clusters.

Multi-team analytics. Platform teams can use YARN to allocate shared cluster capacity to MapReduce, Spark, and other engines.

These uses share a workload shape: large sequential reads, parallel computation, and tolerance for batch-oriented latency. Hadoop is less suitable for frequent small transactions or latency-sensitive interactive requests.

How does Hadoop work?

Consider a batch job that calculates daily request counts by service from a large collection of logs. The path from input files to output illustrates how Hadoop's storage, resource management, and processing layers cooperate.

1. The files enter HDFS. A client asks the NameNode where to place each file block, then sends the data directly to a pipeline of DataNodes. HDFS stores replicas according to the file's configured replication factor.
2. The client submits an application. For a MapReduce job, the client sends the job configuration, code, and input and output paths to YARN. The ResourceManager accepts the application and allocates a first container for its ApplicationMaster.
3. The application requests resources. The ApplicationMaster negotiates containers with the ResourceManager. NodeManagers launch and monitor work on the selected machines.
4. Map tasks read input splits. Hadoop's InputFormat creates logical input splits, and the framework starts a map task for each split. A mapper transforms input key-value pairs into intermediate key-value pairs. For the log example, it might emit (service, 1) for every valid request.
5. The framework shuffles and sorts. A partitioner assigns each key to a reducer. Reducers fetch and merge their partitions, then group records with the same key. This network and disk activity can dominate a job's cost.
6. Reduce tasks produce results. Each reducer processes a key and its associated values. In the example, it sums the counts for each service and writes output through an OutputFormat, commonly back to HDFS.
7. The system handles failures. Failed tasks can be scheduled again. HDFS detects missing DataNode heartbeats and arranges new replicas for under-replicated blocks.

The MapReduce tutorial separates the reducer side into shuffle, sort, and reduce phases. A job can have zero reducers when it needs only independent map transformations. Multi-stage workflows, however, often require several jobs and repeated materialization of intermediate data. The resulting execution model favors throughput-oriented batch work over workflows that revisit intermediate state frequently.

Hadoop architecture

Hadoop uses a master-worker architecture, but storage and compute have separate control planes: HDFS and YARN. The NameNode coordinates the HDFS namespace. The ResourceManager arbitrates compute resources. DataNodes store blocks, while NodeManagers supervise work on cluster machines. In many deployments, a worker machine runs both a DataNode and a NodeManager so a scheduler can place computation near an input replica.

HDFS storage plane

The HDFS architecture guide describes the NameNode as the authority for the filesystem namespace and block placement. It tracks directories, permissions, files, and the mapping from file blocks to DataNodes. DataNodes store the block contents on their local storage and serve client reads and writes. They send heartbeats and block reports so the NameNode can track their health and inventory.

An HDFS client first contacts the NameNode for metadata, then transfers data directly to or from DataNodes. This keeps bulk data off the metadata service. Replica placement can account for rack topology, balancing write traffic against resilience to node and rack failures. Block size and replication factor are configurable rather than universal properties of every file.

HDFS can be configured for high availability with active and standby NameNodes. High availability protects service continuity. HDFS Federation addresses namespace scaling separately by assigning independent namespaces and block pools to multiple NameNodes.

YARN compute plane

YARN separates global resource allocation from application-specific coordination. The ResourceManager contains a scheduler that grants containers according to resource requests and configured queue policy. It does not directly manage every task. Each application's ApplicationMaster requests containers, tracks progress, and works with NodeManagers to start and monitor processes.

A container represents resources such as memory and CPU on a worker. This lets YARN host frameworks with different execution models. MapReduce is one YARN application framework, and Spark can also run on YARN.

Processing plane

MapReduce supplies Hadoop's original batch framework. It runs independent map tasks followed by a key-based exchange and reduction. Data locality influences task placement when suitable containers and replicas are available. Intermediate output can spill to local disks before reducers fetch it.

The separation among these planes explains why a current Hadoop environment may look different from a classic one. HDFS can remain the storage layer and YARN the scheduler while Spark, Hive on Tez, or another engine performs the computation.

Key components of Hadoop

The Apache Hadoop project lists four modules. Each answers a different architectural question.

Hadoop Common. Shared libraries and utilities provide configuration, I/O, security, RPC, scripts, and filesystem interfaces.

HDFS. HDFS is optimized for high-throughput access to large files. The NameNode keeps namespace and block-location metadata, while DataNodes hold block data. The architecture favors streaming reads and append-oriented data over arbitrary in-place updates.

YARN. YARN turns a cluster into a resource pool. Its scheduler divides capacity among queues and applications, while each ApplicationMaster supplies framework-specific coordination.

MapReduce. Mappers create intermediate key-value records, a partitioner selects their reducers, and the framework shuffles and groups the data. Reducers produce final records. Iterative and interactive work pays for repeated job startup, serialization, network exchange, and disk I/O.

The Apache Hadoop project lists Hive, HBase, Spark, ZooKeeper, and others separately as Hadoop-related projects. They are not among the four Hadoop modules above.

Advantages of Hadoop

Scale-out storage and computation. HDFS distributes blocks and MapReduce distributes work. Teams can expand capacity by adding nodes, subject to cluster constraints.

Fault-aware design. HDFS replication, heartbeats, and re-replication address storage failures. YARN and application frameworks monitor processes and recover failed components or tasks.

Data locality. The MapReduce framework can schedule tasks on nodes where input data is present. For large scans, moving code to the data can avoid transferring the full input through the network. Shuffle data and final results still move when the job requires it.

Shared infrastructure. YARN allows several applications and engines to draw from the same resource pool. Queues and scheduler policies give operators controls for dividing capacity among teams and workloads.

Open ecosystem and portable interfaces. Filesystem APIs, command-line tools, and integration points let more than one engine process the same HDFS data.

The advantages are strongest for large, throughput-oriented workloads running on infrastructure an organization is prepared to operate. Operating Hadoop still requires cluster engineering across many machines.

Limitations of Hadoop

MapReduce has high latency. Job setup, task scheduling, serialization, shuffle, sorting, and disk materialization add overhead. The model works well for long batch scans but poorly for iterative machine learning, interactive SQL, and event-by-event processing. Multi-step algorithms may read and write the same intermediate dataset repeatedly.

The platform is operationally demanding. HDFS and YARN introduce several long-running services, configuration files, logs, metrics, queues, storage policies, and recovery procedures. Production deployments also need authentication, authorization, encryption, upgrades, capacity planning, and tested high-availability arrangements.

Small files pressure the namespace. The NameNode holds the filesystem namespace and block map in memory. A large number of small files creates many metadata objects while providing little data per block, which can constrain the NameNode. Compaction and file-layout discipline matter.

HDFS couples storage to cluster infrastructure. Adding HDFS capacity traditionally means adding or expanding machines that run the storage layer. Cloud object stores separate durable storage from transient compute more directly, which can simplify elasticity for cloud-native workloads.

HDFS is not an OLTP store. Its access and write model favors large files, sequential access, one writer at a time, and append or truncate operations. Applications needing indexed point lookups, high rates of small updates, or multi-row transactions need a database or another serving system.

Hadoop vs Spark

Hadoop MapReduce and Apache Spark are not equivalent products. Hadoop includes storage and resource management; Spark is a processing engine. The direct comparison is therefore MapReduce versus Spark, while Spark can still use HDFS and YARN from the Hadoop stack.

Spark can replace MapReduce processing without replacing the rest of Hadoop. Spark's current documentation describes HDFS and YARN integration, alongside standalone and Kubernetes deployment. For an existing Hadoop estate, adopting Spark can be a compute-layer change rather than a storage migration.

The larger limitation is fit. Hadoop's classic architecture remains coherent for self-managed, throughput-oriented batch platforms. A new system built around elastic object storage, interactive queries, or continuous processing may reach the same goals with fewer Hadoop services.

Conclusion

Hadoop divides a distributed data platform into distinct layers. HDFS stores blocks and recovers from storage failures. YARN allocates resources across applications. MapReduce turns batch programs into parallel map, shuffle, sort, and reduce tasks. Hadoop Common supplies their shared foundation. Together, these components explain how Hadoop scales beyond one machine and why it remains relevant in established on-premises data platforms.

The same separation also clarifies Hadoop's limits. MapReduce is optimized for throughput rather than low latency, HDFS favors large sequential files rather than transactions, and operating the complete stack requires sustained cluster expertise. Spark can take over the processing layer while continuing to use HDFS and YARN, so the practical choice is often about which Hadoop components a workload still needs.

When analysis shifts from table scans and aggregates to multi-hop relationships, PuppyGraph can map existing warehouse and lakehouse tables to nodes and edges and execute graph queries without a graph-specific ETL pipeline or a separate graph data copy. This complements batch and SQL engines rather than replacing their transformation and reporting roles.

Try the forever-free PuppyGraph Developer Edition and book a demo with the team to see how openCypher and Gremlin queries traverse warehouse and lakehouse tables, with no graph-specific ETL, when relationship analysis would otherwise require a separate processing pipeline.