task1 - Details for Query 3

Details for Query 3

Submitted Time: 2024/12/12 01:01:02
Duration: 26 s
Succeeded Jobs: 5

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" TakeOrderedAndProject "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n252 ms (0 ms, 1 ms, 28 ms (stage 7.0: task 65))"; 2 [labelType="html" label="HashAggregate time in aggregation build total (min, med, max (stageId: taskId)) 8 ms (0 ms, 0 ms, 8 ms (stage 7.0: task 65)) peak memory total (min, med, max (stageId: taskId)) 498.0 MiB (256.0 KiB, 256.0 KiB, 64.3 MiB (stage 7.0: task 68)) number of output rows: 7 avg hash probe bucket list iters (min, med, max (stageId: taskId)): (1, 1, 1 (stage 7.0: task 68))"]; } 3 [labelType="html" label="Exchange shuffle records written: 14 shuffle write time total (min, med, max (stageId: taskId)) 14 ms (0 ms, 0 ms, 7 ms (stage 6.0: task 46)) records read: 14 local bytes read total (min, med, max (stageId: taskId)) 561.0 B (0.0 B, 0.0 B, 81.0 B (stage 7.0: task 65)) fetch wait time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 7.0: task 68)) remote bytes read total (min, med, max (stageId: taskId)) 566.0 B (0.0 B, 0.0 B, 81.0 B (stage 7.0: task 68)) local blocks read: 7 remote blocks read: 7 data size total (min, med, max (stageId: taskId)) 560.0 B (0.0 B, 0.0 B, 280.0 B (stage 6.0: task 46)) shuffle bytes written total (min, med, max (stageId: taskId)) 1127.0 B (0.0 B, 0.0 B, 567.0 B (stage 6.0: task 46))"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n49.8 s (763 ms, 2.7 s, 3.5 s (stage 6.0: task 44))"; 5 [labelType="html" label="HashAggregate time in aggregation build total (min, med, max (stageId: taskId)) 49.5 s (758 ms, 2.7 s, 3.4 s (stage 6.0: task 44)) peak memory total (min, med, max (stageId: taskId)) 5.0 MiB (256.0 KiB, 256.0 KiB, 256.0 KiB (stage 6.0: task 45)) number of output rows: 14"]; 6 [labelType="html" label=" Project "]; 7 [labelType="html" label="Filter number of output rows: 2,202"]; } 8 [labelType="html" label="Scan csv number of files read: 7 metadata time: 0 ms size of files read: 2.2 GiB number of output rows: 22,400,728"]; 2->0; 3->2; 5->3; 6->5; 7->6; 8->7; }

TakeOrderedAndProject(limit=21, orderBy=[trip_date#93 ASC NULLS FIRST], output=[trip_date#93,trip_count#135])

HashAggregate(keys=[trip_date#93], functions=[count(1)])

WholeStageCodegen (2)

Exchange hashpartitioning(trip_date#93, 200), true, [id=#82]

HashAggregate(keys=[trip_date#93], functions=[partial_count(1)])

Project [date_format(cast(tpep_pickup_datetime#40 as timestamp), yyyy-MM-dd, Some(GMT)) AS trip_date#93]

Filter (((((isnotnull(fare_amount#47) AND isnotnull(trip_distance#43)) AND (cast(fare_amount#47 as int) > 50)) AND (cast(trip_distance#43 as int) < 1)) AND (date_format(cast(tpep_pickup_datetime#40 as timestamp), yyyy-MM-dd, Some(GMT)) >= 2023-02-01)) AND (date_format(cast(tpep_pickup_datetime#40 as timestamp), yyyy-MM-dd, Some(GMT)) <= 2023-02-07))

WholeStageCodegen (1)

FileScan csv [tpep_pickup_datetime#40,trip_distance#43,fare_amount#47] Batched: false, DataFilters: [isnotnull(fare_amount#47), isnotnull(trip_distance#43), (cast(fare_amount#47 as int) > 50), (cas..., Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/nyc_taxi/yellow_tripdata/2023], PartitionFilters: [], PushedFilters: [IsNotNull(fare_amount), IsNotNull(trip_distance)], ReadSchema: struct<tpep_pickup_datetime:string,trip_distance:string,fare_amount:string>

Details

== Parsed Logical Plan ==
GlobalLimit 21
+- LocalLimit 21
   +- Project [cast(trip_date#93 as string) AS trip_date#134, cast(trip_count#129L as string) AS trip_count#135]
      +- Sort [trip_date#93 ASC NULLS FIRST], true
         +- Aggregate [trip_date#93], [trip_date#93, count(1) AS trip_count#129L]
            +- Filter ((trip_date#93 >= 2023-02-01) AND (trip_date#93 <= 2023-02-07))
               +- Project [tpep_pickup_datetime#40, tpep_dropoff_datetime#41, passenger_count#42, trip_distance#43, PULocationID#44, DOLocationID#45, payment_type#46, fare_amount#47, extra#48, mta_tax#49, tip_amount#50, tolls_amount#51, total_amount#52, congestion_surcharge#53, airport_fee#54, taxi_type#55, date_format(cast(tpep_pickup_datetime#40 as timestamp), yyyy-MM-dd, Some(GMT)) AS trip_date#93]
                  +- Filter ((cast(fare_amount#47 as int) > 50) AND (cast(trip_distance#43 as int) < 1))
                     +- Relation[tpep_pickup_datetime#40,tpep_dropoff_datetime#41,passenger_count#42,trip_distance#43,PULocationID#44,DOLocationID#45,payment_type#46,fare_amount#47,extra#48,mta_tax#49,tip_amount#50,tolls_amount#51,total_amount#52,congestion_surcharge#53,airport_fee#54,taxi_type#55] csv

== Analyzed Logical Plan ==
trip_date: string, trip_count: string
GlobalLimit 21
+- LocalLimit 21
   +- Project [cast(trip_date#93 as string) AS trip_date#134, cast(trip_count#129L as string) AS trip_count#135]
      +- Sort [trip_date#93 ASC NULLS FIRST], true
         +- Aggregate [trip_date#93], [trip_date#93, count(1) AS trip_count#129L]
            +- Filter ((trip_date#93 >= 2023-02-01) AND (trip_date#93 <= 2023-02-07))
               +- Project [tpep_pickup_datetime#40, tpep_dropoff_datetime#41, passenger_count#42, trip_distance#43, PULocationID#44, DOLocationID#45, payment_type#46, fare_amount#47, extra#48, mta_tax#49, tip_amount#50, tolls_amount#51, total_amount#52, congestion_surcharge#53, airport_fee#54, taxi_type#55, date_format(cast(tpep_pickup_datetime#40 as timestamp), yyyy-MM-dd, Some(GMT)) AS trip_date#93]
                  +- Filter ((cast(fare_amount#47 as int) > 50) AND (cast(trip_distance#43 as int) < 1))
                     +- Relation[tpep_pickup_datetime#40,tpep_dropoff_datetime#41,passenger_count#42,trip_distance#43,PULocationID#44,DOLocationID#45,payment_type#46,fare_amount#47,extra#48,mta_tax#49,tip_amount#50,tolls_amount#51,total_amount#52,congestion_surcharge#53,airport_fee#54,taxi_type#55] csv

== Optimized Logical Plan ==
GlobalLimit 21
+- LocalLimit 21
   +- Project [trip_date#93, cast(trip_count#129L as string) AS trip_count#135]
      +- Sort [trip_date#93 ASC NULLS FIRST], true
         +- Aggregate [trip_date#93], [trip_date#93, count(1) AS trip_count#129L]
            +- Project [date_format(cast(tpep_pickup_datetime#40 as timestamp), yyyy-MM-dd, Some(GMT)) AS trip_date#93]
               +- Filter (((((isnotnull(fare_amount#47) AND isnotnull(trip_distance#43)) AND (cast(fare_amount#47 as int) > 50)) AND (cast(trip_distance#43 as int) < 1)) AND (date_format(cast(tpep_pickup_datetime#40 as timestamp), yyyy-MM-dd, Some(GMT)) >= 2023-02-01)) AND (date_format(cast(tpep_pickup_datetime#40 as timestamp), yyyy-MM-dd, Some(GMT)) <= 2023-02-07))
                  +- Relation[tpep_pickup_datetime#40,tpep_dropoff_datetime#41,passenger_count#42,trip_distance#43,PULocationID#44,DOLocationID#45,payment_type#46,fare_amount#47,extra#48,mta_tax#49,tip_amount#50,tolls_amount#51,total_amount#52,congestion_surcharge#53,airport_fee#54,taxi_type#55] csv

== Physical Plan ==
TakeOrderedAndProject(limit=21, orderBy=[trip_date#93 ASC NULLS FIRST], output=[trip_date#93,trip_count#135])
+- *(2) HashAggregate(keys=[trip_date#93], functions=[count(1)], output=[trip_date#93, trip_count#129L])
   +- Exchange hashpartitioning(trip_date#93, 200), true, [id=#82]
      +- *(1) HashAggregate(keys=[trip_date#93], functions=[partial_count(1)], output=[trip_date#93, count#139L])
         +- *(1) Project [date_format(cast(tpep_pickup_datetime#40 as timestamp), yyyy-MM-dd, Some(GMT)) AS trip_date#93]
            +- *(1) Filter (((((isnotnull(fare_amount#47) AND isnotnull(trip_distance#43)) AND (cast(fare_amount#47 as int) > 50)) AND (cast(trip_distance#43 as int) < 1)) AND (date_format(cast(tpep_pickup_datetime#40 as timestamp), yyyy-MM-dd, Some(GMT)) >= 2023-02-01)) AND (date_format(cast(tpep_pickup_datetime#40 as timestamp), yyyy-MM-dd, Some(GMT)) <= 2023-02-07))
               +- FileScan csv [tpep_pickup_datetime#40,trip_distance#43,fare_amount#47] Batched: false, DataFilters: [isnotnull(fare_amount#47), isnotnull(trip_distance#43), (cast(fare_amount#47 as int) > 50), (cas..., Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/nyc_taxi/yellow_tripdata/2023], PartitionFilters: [], PushedFilters: [IsNotNull(fare_amount), IsNotNull(trip_distance)], ReadSchema: struct<tpep_pickup_datetime:string,trip_distance:string,fare_amount:string>