Apache Hadoop

Introduction to Apache Hadoop

Before 2006, the term “Big Data” meant buying a massive, multi-million dollar Oracle or Teradata mainframe. These machines were technological marvels, but they were fundamentally limited. You could not store Petabytes of data, and you absolutely could not store “unstructured” data like raw text logs or images.

In 2006, Doug Cutting and Mike Cafarella, inspired by whitepapers published by Google, created Apache Hadoop.

Hadoop was the software framework that single-handedly birthed the modern Big Data industry. It allowed enterprises to take thousands of cheap, standard, “commodity” computers, wire them together, and treat them as a single, infinitely scalable supercomputer.

The Two Pillars of Hadoop

Hadoop is not a database. It is a framework consisting of two massive, foundational sub-projects: one for storage, and one for compute.

1. HDFS (Hadoop Distributed File System)

HDFS solved the storage problem. If you have a 1-Terabyte log file, it will not fit on a standard 500GB hard drive. HDFS takes that 1TB file and automatically chops it into 128-Megabyte “Blocks.” It then scatters those blocks across 1,000 different servers in a server rack. To guarantee fault tolerance, HDFS defaults to 3x replication. It makes three copies of every block and places them on different physical servers. If a server catches fire and dies, the system doesn’t lose a single byte of data. The “NameNode” (the brain of HDFS) simply notices the missing blocks and instantly recreates them from the backups.

2. MapReduce

MapReduce solved the compute problem. In a traditional database, you pull the data over the network to the CPU. If you try to pull Petabytes of data over a network cable, the network instantly crashes. MapReduce flips the paradigm: It moves the code to the data. The central server sends a tiny Java program out to the 1,000 servers. Each server runs the code locally, right next to the physical hard drive where the 128MB block of data lives. The servers process their tiny chunk of data simultaneously, and send only the final aggregated answer back to the central server.

The Hadoop Ecosystem Explosion

Because HDFS was so revolutionary, an entire “Zoo” of open-source projects sprang up around it to make it easier to use:

• Apache Hive: Allowed analysts to write SQL queries against data sitting in HDFS, instead of writing complex Java MapReduce code.
• Apache HBase: A massive NoSQL database built on top of HDFS.
• Apache Zookeeper: A centralized service for maintaining configuration information across the cluster.

The Fall of Hadoop

Today, Hadoop is considered legacy technology, largely replaced by Cloud Object Storage (Amazon S3) and Apache Spark.

Why did it die?

1. Coupled Architecture: In Hadoop, the hard drives (HDFS) and the CPUs (MapReduce/YARN) were physically trapped inside the exact same server racks. If you ran out of storage but had plenty of CPU, you still had to buy expensive servers containing both. The Cloud separated Storage and Compute, making Hadoop’s architecture economically obsolete.
2. Agonizing Complexity: Maintaining an on-premise Hadoop cluster required an army of specialized system administrators. Upgrading the cluster often took weeks of planned downtime.

Conclusion

While you will rarely spin up a new Hadoop cluster today, understanding its architecture is mandatory. Hadoop proved that distributed, scale-out commodity hardware was the future of data engineering, laying the philosophical and technical groundwork for the modern Data Lake and Data Lakehouse architectures that dominate the industry today.