Search for a command to run...
Articles tagged with #hadoop
Spark Streaming from Amazon SQS So this is a journal log, to describe how was I able to integrate Spark Streaming and Amazon SQS and what all problems I faced while doing so. Problem Statement Seems like when we read s3 files from spark Streaming, s...
Long Running Spark Streaming Jobs Fault Tolerance Spark and YARN both were not really designed for real time processing, but they have adapted to the need of near real time processing. For a Spark job in cluster mode, driver and Application Master s...
Delta Lake [Part 4] Change data feed change data feed represents row-level changes between versions, This includes the row data along with metadata indicating whether the specified row was inserted, deleted, or updated. we can read this field using S...
Delta Lake [Part 3] - Delta table Streaming Delta Lake is deeply integrated with Spark Structured Streaming through readStream and writeStream along with following features, Coalesce small files Maintains Exactly-once processing Discovering new file...
HDFS - [Hadoop Distributed File System] HDFS architecture looks like this - NameNode holds all the metadata about files and directory structure and then there are DataNodes which basically hold files in the form of blocks so DataNode holds blocks o...
