In this chapter, our focus was to cover the architecture of HDFS and its components. We covered the internals of NameNode and DataNode internals. We explained the work of the Quorum Journal Manager and HDFS high availability in Hadoop 3. Data management was also a focus of this chapter, and we covered edit logs and fsimage in detail. We looked at a brief overview of the checkpoint process. We also covered the internals of the HDFS write and read operations. The interface and HDFS commands were explained with examples.

In the next chapter, we will cover Yet Another Resource Manager (YARN) in detail. We will dive deep into the YARN architecture and will cover its components in detail. We will gain an understanding of the different types of schedulers that are available in YARN and their detailed uses. A few new features were added to YARN in the Hadoop 3 release...