Apache Hadoop commands HDFS

Updated:10/20/2023 by Computer Hope

The File System (FS) shell includes various shell-like commands that directly interact with the Hadoop Distributed File System (HDFS) as well as other file systems that Hadoop supports, such as Local FS, WebHDFS, S3 FS, and others. The FS shell is invoked by:

Hadoop HDFS Commands

Most of the commands in FS shell behave like corresponding Unix commands. Differences are described with each of the commands. Error information is sent to stderr and the output is sent to stdout.
If HDFS is being used, hdfs dfs is a synonym.
Relative paths can be used. For HDFS, the current working directory is the HDFS home directory /user/ that often has to be created manually. The HDFS home directory can also be implicitly accessed, e.g., when using the HDFS trash folder, the .Trash directory in the home directory.


HDFS Commands

We will start with some very basic help commands and this go over some of the most helpful Hadoop HDFS commands..

 1) fsck  

HDFS Command to check the health of the Hadoop file system.

Command: hdfs fsck /
2) ls

HDFS Command to display the list of Files and Directories in HDFS.

Command: hdfs dfs –ls /
3) mkdir

HDFS Command to create the directory in HDFS.

Command: hdfs dfs –mkdir /new_deveoperIndian
4) touchz

HDFS Command to create a file in HDFS with file size 0 bytes.

command: hdfs dfs –touchz /directory/new_deveoperIndian
5) du

HDFS Command to check the file size.

Command: hdfs dfs –du –s /new_developerIndian/testfile
6) cat

HDFS Command that reads a file on HDFS and prints the content of that file to the console as output.

Command: hdfs dfs –cat /new_developerIndian/testfile
7) text

HDFS Command that takes a source file and outputs the file in text format.

Command: hdfs dfs –text /new_developerIndian/testfile
8) textcopyFromLocal

HDFS Command to copy the file from a Local file system to HDFS.

Command: hdfs dfs –copyFromLocal /home/edureka/test /new_developerIndian
9) copyToLocal

HDFS Command to copy the file from HDFS to Local File System.

Command: hdfs dfs –copyToLocal /new_developerIndian/test /home/edureka
10) put

HDFS Command to copy single source or multiple sources from local file system to the destination file system.

Command: hdfs dfs –put /home/edureka/test /user
11) get

HDFS Command to copy files from hdfs to the local file system.

Command: hdfs dfs –get /user/test /home/edureka
12) count

HDFS Command to count the number of directories, files, and bytes under the paths that match the specified file pattern.

Command: hdfs dfs –count /user
13) rm

HDFS Command to remove the file from HDFS.

Command: hdfs dfs –rm /new_developerIndian/test
14) rm -r

HDFS Command to remove the entire directory and all of its content from HDFS.

Command: hdfs dfs -rm -r /new_developerIndian
15) cp

HDFS Command to copy files from source to destination. This command allows multiple sources as well, in which case the destination must be a directory.

Command: hdfs dfs -cp /user/hadoop/file1 /user/hadoop/file2
16) mv

HDFS Command to move files from source to destination. This command allows multiple sources as well, in which case the destination needs to be a directory.

Command: hdfs dfs -mv /user/hadoop/file1 /user/hadoop/file2
17) expunge

HDFS Command that makes the trash empty.

Command: hdfs dfs -expunge
18) rmdir

HDFS Command to remove the directory.

Command: hdfs dfs –rmdir /user/hadoop
19) usage

HDFS Command that returns the help for an individual command.

Command: hdfs dfs -usage mkdir
20) help

HDFS Command that displays help for given command or all commands if none is specified.

Command: hdfs dfs -help

Use case of Hadoop HDFS

  • Hdfs is used for Very Large Files.
  • HDF work with Streaming Data Access.
  • It is support low cost hardware.
  • Log and Event Data Storage.
  • used in Batch Processing.
  • we used in Data Warehousing.
  • It used in Batch Processing.
  • It used in machine learning model and Processing.

Conclusion :

In this article , we see hadoop HDFS is a fundamental building block of the Hadoop ecosystem. here we learn about use case and some very basic hadoop hdfs command. check next article for other command and more detail.