movieratings - Details for Query 10

Details for Query 10

Submitted Time: 2025/03/08 12:38:47
Duration: 1.3 min
Succeeded Jobs: 33 34 35 36 37 38

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" CollectLimit "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n63 ms (0 ms, 0 ms, 9 ms (stage 69.0: task 1978))"; 2 [labelType="html" label="HashAggregate time in aggregation build total (min, med, max (stageId: taskId)) 27 ms (0 ms, 0 ms, 7 ms (stage 69.0: task 1978)) peak memory total (min, med, max (stageId: taskId)) 791.5 MiB (256.0 KiB, 256.0 KiB, 64.3 MiB (stage 69.0: task 1978)) number of output rows: 12 avg hash probe bucket list iters (min, med, max (stageId: taskId)): (1, 1, 1 (stage 69.0: task 1978))"]; } 3 [labelType="html" label="Exchange shuffle records written: 786 shuffle write time total (min, med, max (stageId: taskId)) 228 ms (5 ms, 7 ms, 12 ms (stage 64.0: task 1953)) records read: 336 local bytes read total (min, med, max (stageId: taskId)) 10.7 KiB (0.0 B, 0.0 B, 1024.0 B (stage 69.0: task 1978)) fetch wait time total (min, med, max (stageId: taskId)) 10 ms (0 ms, 0 ms, 5 ms (stage 69.0: task 1978)) remote bytes read total (min, med, max (stageId: taskId)) 10.2 KiB (0.0 B, 0.0 B, 1024.0 B (stage 69.0: task 1979)) local blocks read: 172 remote blocks read: 164 data size total (min, med, max (stageId: taskId)) 18.4 KiB (672.0 B, 672.0 B, 696.0 B (stage 64.0: task 1952)) shuffle bytes written total (min, med, max (stageId: taskId)) 49.1 KiB (1791.0 B, 1792.0 B, 1856.0 B (stage 64.0: task 1952))"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n2.4 m (4.2 s, 5.0 s, 6.5 s (stage 64.0: task 1945))"; 5 [labelType="html" label="HashAggregate time in aggregation build total (min, med, max (stageId: taskId)) 2.4 m (4.2 s, 5.0 s, 6.5 s (stage 64.0: task 1950)) peak memory total (min, med, max (stageId: taskId)) 7.0 MiB (256.0 KiB, 256.0 KiB, 256.0 KiB (stage 64.0: task 1946)) number of output rows: 786"]; 6 [labelType="html" label=" Project "]; 7 [labelType="html" label="Scan ExistingRDD number of output rows: 33,832,162"]; } 2->0; 3->2; 5->3; 6->5; 7->6; }

CollectLimit 11

HashAggregate(keys=[year#127], functions=[count(1)])

WholeStageCodegen (2)

Exchange hashpartitioning(year#127, 200), true, [id=#381]

HashAggregate(keys=[year#127], functions=[partial_count(1)])

Project [year(cast(from_unixtime(timestamp_str#3L, yyyy-MM-dd, Some(GMT)) as date)) AS year#127]

Scan ExistingRDD[userId#0L,movieId#1L,rating#2,timestamp_str#3L]

WholeStageCodegen (1)

Details

== Parsed Logical Plan ==
GlobalLimit 11
+- LocalLimit 11
   +- Project [cast(year#127 as string) AS year#320, cast(ratings_per_year#315L as string) AS ratings_per_year#321]
      +- Aggregate [year#127], [year#127, count(1) AS ratings_per_year#315L]
         +- Project [userId#0L, movieId#1L, rating#2, timestamp_str#3L, date#39, month#120, year#127, CASE WHEN ((month#120 >= 1) AND (month#120 <= 6)) THEN Early Year ELSE Late Year END AS time_of_rating#135]
            +- Project [userId#0L, movieId#1L, rating#2, timestamp_str#3L, date#39, month#120, year(cast(date#39 as date)) AS year#127]
               +- Project [userId#0L, movieId#1L, rating#2, timestamp_str#3L, date#39, month(cast(date#39 as date)) AS month#120]
                  +- Sort [date#39 ASC NULLS FIRST], true
                     +- Project [userId#0L, movieId#1L, rating#2, timestamp_str#3L, from_unixtime(timestamp_str#3L, yyyy-MM-dd, Some(GMT)) AS date#39]
                        +- LogicalRDD [userId#0L, movieId#1L, rating#2, timestamp_str#3L], false

== Analyzed Logical Plan ==
year: string, ratings_per_year: string
GlobalLimit 11
+- LocalLimit 11
   +- Project [cast(year#127 as string) AS year#320, cast(ratings_per_year#315L as string) AS ratings_per_year#321]
      +- Aggregate [year#127], [year#127, count(1) AS ratings_per_year#315L]
         +- Project [userId#0L, movieId#1L, rating#2, timestamp_str#3L, date#39, month#120, year#127, CASE WHEN ((month#120 >= 1) AND (month#120 <= 6)) THEN Early Year ELSE Late Year END AS time_of_rating#135]
            +- Project [userId#0L, movieId#1L, rating#2, timestamp_str#3L, date#39, month#120, year(cast(date#39 as date)) AS year#127]
               +- Project [userId#0L, movieId#1L, rating#2, timestamp_str#3L, date#39, month(cast(date#39 as date)) AS month#120]
                  +- Sort [date#39 ASC NULLS FIRST], true
                     +- Project [userId#0L, movieId#1L, rating#2, timestamp_str#3L, from_unixtime(timestamp_str#3L, yyyy-MM-dd, Some(GMT)) AS date#39]
                        +- LogicalRDD [userId#0L, movieId#1L, rating#2, timestamp_str#3L], false

== Optimized Logical Plan ==
GlobalLimit 11
+- LocalLimit 11
   +- Aggregate [year#127], [cast(year#127 as string) AS year#320, cast(count(1) as string) AS ratings_per_year#321]
      +- Project [year(cast(from_unixtime(timestamp_str#3L, yyyy-MM-dd, Some(GMT)) as date)) AS year#127]
         +- LogicalRDD [userId#0L, movieId#1L, rating#2, timestamp_str#3L], false

== Physical Plan ==
CollectLimit 11
+- *(2) HashAggregate(keys=[year#127], functions=[count(1)], output=[year#320, ratings_per_year#321])
   +- Exchange hashpartitioning(year#127, 200), true, [id=#381]
      +- *(1) HashAggregate(keys=[year#127], functions=[partial_count(1)], output=[year#127, count#325L])
         +- *(1) Project [year(cast(from_unixtime(timestamp_str#3L, yyyy-MM-dd, Some(GMT)) as date)) AS year#127]
            +- *(1) Scan ExistingRDD[userId#0L,movieId#1L,rating#2,timestamp_str#3L]