movieratings - Details for Query 13

Details for Query 13

Submitted Time: 2025/03/08 12:44:00
Duration: 1.2 min
Succeeded Jobs: 42 43 44 45 46

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" CollectLimit "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: total (min, med, max (stageId: taskId))\n45 ms (0 ms, 0 ms, 11 ms (stage 98.0: task 3108))"; 2 [labelType="html" label="HashAggregate time in aggregation build total (min, med, max (stageId: taskId)) 16 ms (0 ms, 0 ms, 10 ms (stage 98.0: task 3108)) peak memory total (min, med, max (stageId: taskId)) 178.0 MiB (256.0 KiB, 256.0 KiB, 64.3 MiB (stage 98.0: task 3107)) number of output rows: 2 avg hash probe bucket list iters (min, med, max (stageId: taskId)): (1, 1, 1 (stage 98.0: task 3107))"]; } 3 [labelType="html" label="Exchange shuffle records written: 400 shuffle write time total (min, med, max (stageId: taskId)) 709 ms (2 ms, 3 ms, 5 ms (stage 85.0: task 2959)) records read: 400 local bytes read total (min, med, max (stageId: taskId)) 16.6 KiB (0.0 B, 0.0 B, 8.4 KiB (stage 98.0: task 3107)) fetch wait time total (min, med, max (stageId: taskId)) 2 ms (0 ms, 0 ms, 2 ms (stage 98.0: task 3108)) remote bytes read total (min, med, max (stageId: taskId)) 16.6 KiB (0.0 B, 0.0 B, 9.0 KiB (stage 98.0: task 3107)) local blocks read: 200 remote blocks read: 200 data size total (min, med, max (stageId: taskId)) 17.2 KiB (88.0 B, 88.0 B, 88.0 B (stage 85.0: task 2782)) shuffle bytes written total (min, med, max (stageId: taskId)) 33.2 KiB (169.0 B, 170.0 B, 171.0 B (stage 85.0: task 2825))"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n1.3 s (5 ms, 6 ms, 14 ms (stage 85.0: task 2818))"; 5 [labelType="html" label="HashAggregate time in aggregation build total (min, med, max (stageId: taskId)) 328 ms (1 ms, 2 ms, 5 ms (stage 85.0: task 2783)) peak memory total (min, med, max (stageId: taskId)) 50.0 MiB (256.0 KiB, 256.0 KiB, 256.0 KiB (stage 85.0: task 2782)) number of output rows: 400"]; 6 [labelType="html" label="HashAggregate time in aggregation build total (min, med, max (stageId: taskId)) 185 ms (0 ms, 1 ms, 2 ms (stage 85.0: task 2782)) peak memory total (min, med, max (stageId: taskId)) 12.5 GiB (64.3 MiB, 64.3 MiB, 64.3 MiB (stage 85.0: task 2782)) number of output rows: 330,975 avg hash probe bucket list iters (min, med, max (stageId: taskId)): (1, 1, 1 (stage 85.0: task 2782))"]; } 7 [labelType="html" label="Exchange shuffle records written: 331,002 shuffle write time total (min, med, max (stageId: taskId)) 465 ms (13 ms, 15 ms, 22 ms (stage 84.0: task 2755)) records read: 331,002 local bytes read total (min, med, max (stageId: taskId)) 2.2 MiB (8.6 KiB, 12.2 KiB, 13.5 KiB (stage 85.0: task 2973)) fetch wait time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 85.0: task 2782)) remote bytes read total (min, med, max (stageId: taskId)) 2.2 MiB (8.6 KiB, 10.2 KiB, 13.6 KiB (stage 85.0: task 2932)) local blocks read: 2,812 remote blocks read: 2,788 data size total (min, med, max (stageId: taskId)) 7.6 MiB (233.1 KiB, 279.2 KiB, 294.1 KiB (stage 84.0: task 2761)) shuffle bytes written total (min, med, max (stageId: taskId)) 4.4 MiB (136.4 KiB, 159.4 KiB, 169.5 KiB (stage 84.0: task 2758))"]; subgraph cluster8 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n2.3 m (4.2 s, 4.4 s, 6.5 s (stage 84.0: task 2754))"; 9 [labelType="html" label="HashAggregate time in aggregation build total (min, med, max (stageId: taskId)) 2.3 m (4.2 s, 4.4 s, 6.5 s (stage 84.0: task 2754)) peak memory total (min, med, max (stageId: taskId)) 7.0 MiB (256.0 KiB, 256.0 KiB, 256.0 KiB (stage 84.0: task 2755)) number of output rows: 331,002"]; 10 [labelType="html" label=" Project "]; 11 [labelType="html" label="Scan ExistingRDD number of output rows: 33,832,162"]; } 2->0; 3->2; 5->3; 6->5; 7->6; 9->7; 10->9; 11->10; }

CollectLimit 21

HashAggregate(keys=[user_type#372], functions=[count(1)])

WholeStageCodegen (3)

Exchange hashpartitioning(user_type#372, 200), true, [id=#546]

HashAggregate(keys=[user_type#372], functions=[partial_count(1)])

HashAggregate(keys=[userId#0L], functions=[count(1)])

WholeStageCodegen (2)

Exchange hashpartitioning(userId#0L, 200), true, [id=#541]

HashAggregate(keys=[userId#0L], functions=[partial_count(1)])

Project [userId#0L]

Scan ExistingRDD[userId#0L,movieId#1L,rating#2,timestamp_str#3L]

WholeStageCodegen (1)

Details

== Parsed Logical Plan ==
GlobalLimit 21
+- LocalLimit 21
   +- Project [cast(user_type#372 as string) AS user_type#400, cast(user_count#395L as string) AS user_count#401]
      +- Aggregate [user_type#372], [user_type#372, count(1) AS user_count#395L]
         +- Sort [num_ratings#369L DESC NULLS LAST], true
            +- Project [userId#0L, num_ratings#369L, CASE WHEN (num_ratings#369L > cast(50 as bigint)) THEN Frequent Raters ELSE Infrequent Raters END AS user_type#372]
               +- Aggregate [userId#0L], [userId#0L, count(1) AS num_ratings#369L]
                  +- Sort [date#39 ASC NULLS FIRST], true
                     +- Project [userId#0L, movieId#1L, rating#2, timestamp_str#3L, from_unixtime(timestamp_str#3L, yyyy-MM-dd, Some(GMT)) AS date#39]
                        +- LogicalRDD [userId#0L, movieId#1L, rating#2, timestamp_str#3L], false

== Analyzed Logical Plan ==
user_type: string, user_count: string
GlobalLimit 21
+- LocalLimit 21
   +- Project [cast(user_type#372 as string) AS user_type#400, cast(user_count#395L as string) AS user_count#401]
      +- Aggregate [user_type#372], [user_type#372, count(1) AS user_count#395L]
         +- Sort [num_ratings#369L DESC NULLS LAST], true
            +- Project [userId#0L, num_ratings#369L, CASE WHEN (num_ratings#369L > cast(50 as bigint)) THEN Frequent Raters ELSE Infrequent Raters END AS user_type#372]
               +- Aggregate [userId#0L], [userId#0L, count(1) AS num_ratings#369L]
                  +- Sort [date#39 ASC NULLS FIRST], true
                     +- Project [userId#0L, movieId#1L, rating#2, timestamp_str#3L, from_unixtime(timestamp_str#3L, yyyy-MM-dd, Some(GMT)) AS date#39]
                        +- LogicalRDD [userId#0L, movieId#1L, rating#2, timestamp_str#3L], false

== Optimized Logical Plan ==
GlobalLimit 21
+- LocalLimit 21
   +- Aggregate [user_type#372], [user_type#372, cast(count(1) as string) AS user_count#401]
      +- Aggregate [userId#0L], [CASE WHEN (count(1) > 50) THEN Frequent Raters ELSE Infrequent Raters END AS user_type#372]
         +- Project [userId#0L]
            +- LogicalRDD [userId#0L, movieId#1L, rating#2, timestamp_str#3L], false

== Physical Plan ==
CollectLimit 21
+- *(3) HashAggregate(keys=[user_type#372], functions=[count(1)], output=[user_type#372, user_count#401])
   +- Exchange hashpartitioning(user_type#372, 200), true, [id=#546]
      +- *(2) HashAggregate(keys=[user_type#372], functions=[partial_count(1)], output=[user_type#372, count#405L])
         +- *(2) HashAggregate(keys=[userId#0L], functions=[count(1)], output=[user_type#372])
            +- Exchange hashpartitioning(userId#0L, 200), true, [id=#541]
               +- *(1) HashAggregate(keys=[userId#0L], functions=[partial_count(1)], output=[userId#0L, count#386L])
                  +- *(1) Project [userId#0L]
                     +- *(1) Scan ExistingRDD[userId#0L,movieId#1L,rating#2,timestamp_str#3L]