October 20, 11:59 PM ISTOct 20, 11:59 PM IST
September 30, 2:32 AM ISTSep 30, 2:32 AM IST
1 left (3 attempts every 24 hours)
1 hour per attempt1h per attempt
1 left (1 total within the time limit)
To pass you need at least 80%. We keep your highest score.
Graded Assignment • 30 min
Which of the following statements best describe data wrangling?
Great job! This definition is presented in the Batch Transformation Patterns and Use Cases video.
Which of the following statements best describe a soft delete?
Great job! Soft delete means that you mark the record as “deleted” without permanently removing it from your target system. This operation can improve performance and allow data to be recovered should circumstances arise.
How is modeling the data using dbt different from modeling the data using AWS Glue?
Nice job! As you saw in the Batch Transformation Patterns and Use Cases video, dbt is not an execution tool like Spark. It is a SQL-tool that facilitates transformation within the database or data warehouse. AWS Glue is a processing engine based on Spark, enabling data extraction and transformation before loading into the target system.
What is the purpose of the shuffle task that comes after the mapping task within a MapReduce process?
The shuffle step redistributes the key-value pairs generated by the map tasks so that all pairs with the same key are sent to the same reducer.
Hadoop and Spark are both distributed processing frameworks. What is the difference between these two frameworks?
Nice job! Spark is designed around in-memory processing, while Hadoop is not.
Which of the following statements is true about Spark DataFrames?
Spark DataFrames are indeed a high-level abstraction built on top of RDDs. With DataFrames, you can leverage high-level operations like filtering, selecting, and aggregating without delving into the intricacies of data partitioning and distribution, which are handled by Spark internally.
Which of the following best describes the job of the driver node in the Spark's underlying architecture?
Great work! The driver node is the central controller of a Spark application; It transforms each job into a sequence of stages, and represents these stages as a DAG. Each stage is further broken down into tasks written in Spark code that can run in parallel. Once the DAG execution plan is developed, the driver communicates with the cluster manager to request computational and memory resources for the executors. Each task is assigned to a single core within an executor, and each executor works on a single data partition.
In Spark, transformations are evaluated in a lazy way. What does lazy evaluation mean?
Transformations are evaluated lazily, meaning that they are not executed immediately. Instead, they are recorded as a lineage and only executed when an action is invoked.
In which of the following cases should you choose to processing your data using Pandas over Spark DataFrames?
If the dataset is not very large, you can use Pandas instead of Spark.
In which of the following cases should you choose coding with SQL in PySpark over working with DataFrames using Python to transform your data?
When working with simpler transformations, Spark DataFrames and SQL queries generally result in comparable performance. However, you might find writing SQL queries to be simpler, and easier for your team to understand.
Which of the following describes a main difference between the distributed stream processing tools Spark Streaming and Apache Flink?
That’s right! Spark Streaming processes data in a micro-batch way, providing near real-time performance. Apache Flink is a true stream processing tool that processes one event at a time in a continuous real-time manner, resulting in lower latency than micro-batch processing systems like Spark Streaming.
