Introduction to Hadoop HDFS

Hadoop HDFS Tutorial: A Beginner’s Guide to Distributed Storage
Updated: February 1, 2025 | By Shubham Mishra

Hadoop Distributed File System (HDFS) is the storage component of the Hadoop ecosystem. It is designed to store large datasets across multiple machines in a distributed manner, ensuring high availability, fault tolerance, and scalability.

Key Features of HDFS

1. Distributed Storage – Data is split into blocks and stored across multiple nodes.
2. Fault Tolerance – Replication of data ensures availability even if a node fails.
3. Scalability – Can scale horizontally by adding more nodes.
4. High Throughput – Optimized for large data processing.
5. Write Once, Read Many – Ideal for batch processing and analytical workloads.

HDFS Architecture

HDFS follows a master-slave architecture, consisting of:

• NameNode – Manages metadata and file structure.
• DataNodes – Store actual data blocks and report to the NameNode.
• Secondary NameNode – Periodically saves NameNode metadata for backup.

How HDFS Works

1. File Splitting – Large files are divided into blocks (default 128MB or 256MB).
2. Replication – Each block is replicated (default replication factor is 3).
3. Storage and Retrieval – Blocks are distributed across DataNodes, and NameNode manages access.
4. Data Processing – HDFS is tightly integrated with Hadoop MapReduce for data computation.

Commands in HDFS

Here are some commonly used HDFS commands:

# Create a directory in HDFS
hdfs dfs -mkdir /user/hadoop

# Upload a file to HDFS
hdfs dfs -put localfile.txt /user/hadoop/

# List files in HDFS
hdfs dfs -ls /user/hadoop/

# Read a file from HDFS
hdfs dfs -cat /user/hadoop/localfile.txt

# Delete a file in HDFS
hdfs dfs -rm /user/hadoop/localfile.txt

Advantages of HDFS

• Cost-Effective – Uses commodity hardware to store big data.
• Resilient to Failures – Data replication ensures minimal risk of data loss.
• Efficient Data Processing – Works seamlessly with Apache Hadoop’s ecosystem.
• Parallel Processing – Enhances speed and performance.

Use Cases of HDFS

• Big Data Analytics – Used by organizations for analyzing vast amounts of data.
• Machine Learning – Serves as a data source for ML models.
• Data Warehousing – Stores structured and unstructured data efficiently.
• Log Processing – Helps in analyzing and managing server logs.

Conclusion

HDFS is the backbone of the Hadoop ecosystem, offering a scalable and reliable storage solution for big data applications. Its ability to handle vast amounts of data efficiently makes it an essential component for enterprises dealing with large-scale data processing.

By mastering HDFS, you can unlock the full potential of Hadoop and work with big data effectively. Start exploring HDFS today and take your data management skills to the next level!