task1 - Details for Query 12

Details for Query 12

Submitted Time: 2024/12/18 02:10:58
Duration: 36 s
Succeeded Jobs: 28

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" TakeOrderedAndProject "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n258 ms (0 ms, 1 ms, 20 ms (stage 38.0: task 238))"; 2 [labelType="html" label="HashAggregate time in aggregation build total (min, med, max (stageId: taskId)) 1 ms (0 ms, 0 ms, 1 ms (stage 38.0: task 238)) peak memory total (min, med, max (stageId: taskId)) 498.0 MiB (256.0 KiB, 256.0 KiB, 64.3 MiB (stage 38.0: task 239)) number of output rows: 7 avg hash probe bucket list iters (min, med, max (stageId: taskId)): (1, 1, 1 (stage 38.0: task 239))"]; } 3 [labelType="html" label="Exchange shuffle records written: 14 shuffle write time total (min, med, max (stageId: taskId)) 7 ms (0 ms, 0 ms, 3 ms (stage 37.0: task 220)) records read: 14 local bytes read total (min, med, max (stageId: taskId)) 448.0 B (0.0 B, 0.0 B, 64.0 B (stage 38.0: task 239)) fetch wait time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 38.0: task 239)) remote bytes read total (min, med, max (stageId: taskId)) 448.0 B (0.0 B, 0.0 B, 64.0 B (stage 38.0: task 239)) local blocks read: 7 remote blocks read: 7 data size total (min, med, max (stageId: taskId)) 336.0 B (0.0 B, 0.0 B, 168.0 B (stage 37.0: task 220)) shuffle bytes written total (min, med, max (stageId: taskId)) 896.0 B (0.0 B, 0.0 B, 448.0 B (stage 37.0: task 220))"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n1.1 m (1.1 s, 3.8 s, 4.5 s (stage 37.0: task 223))"; 5 [labelType="html" label="HashAggregate time in aggregation build total (min, med, max (stageId: taskId)) 1.1 m (1.1 s, 3.8 s, 4.5 s (stage 37.0: task 223)) peak memory total (min, med, max (stageId: taskId)) 5.0 MiB (256.0 KiB, 256.0 KiB, 256.0 KiB (stage 37.0: task 218)) number of output rows: 14"]; 6 [labelType="html" label=" Project "]; 7 [labelType="html" label="Filter number of output rows: 2,202"]; } 8 [labelType="html" label="Scan csv number of files read: 7 metadata time: 0 ms size of files read: 2.2 GiB number of output rows: 22,400,728"]; 2->0; 3->2; 5->3; 6->5; 7->6; 8->7; }

TakeOrderedAndProject(limit=21, orderBy=[date#1089 ASC NULLS FIRST], output=[date#1130,trip_count#1131])

HashAggregate(keys=[date#1089], functions=[count(1)])

WholeStageCodegen (2)

Exchange hashpartitioning(date#1089, 200), true, [id=#598]

HashAggregate(keys=[date#1089], functions=[partial_count(1)])

Project [cast(cast(unix_timestamp(tpep_pickup_datetime#70, yyyy-MM-dd HH:mm:ss, Some(GMT)) as timestamp) as date) AS date#1089]

Filter (((((isnotnull(trip_distance#73) AND isnotnull(fare_amount#77)) AND (cast(fare_amount#77 as int) > 50)) AND (cast(trip_distance#73 as int) < 1)) AND (cast(cast(unix_timestamp(tpep_pickup_datetime#70, yyyy-MM-dd HH:mm:ss, Some(GMT)) as timestamp) as date) >= 19389)) AND (cast(cast(unix_timestamp(tpep_pickup_datetime#70, yyyy-MM-dd HH:mm:ss, Some(GMT)) as timestamp) as date) <= 19395))

WholeStageCodegen (1)

FileScan csv [tpep_pickup_datetime#70,trip_distance#73,fare_amount#77] Batched: false, DataFilters: [isnotnull(trip_distance#73), isnotnull(fare_amount#77), (cast(fare_amount#77 as int) > 50), (cas..., Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/nyc_taxi/yellow_tripdata/2023], PartitionFilters: [], PushedFilters: [IsNotNull(trip_distance), IsNotNull(fare_amount)], ReadSchema: struct<tpep_pickup_datetime:string,trip_distance:string,fare_amount:string>

Details

== Parsed Logical Plan ==
GlobalLimit 21
+- LocalLimit 21
   +- Project [cast(date#1089 as string) AS date#1130, cast(trip_count#1125L as string) AS trip_count#1131]
      +- Sort [date#1089 ASC NULLS FIRST], true
         +- Aggregate [date#1089], [date#1089, count(1) AS trip_count#1125L]
            +- Filter ((date#1089 >= cast(2023-02-01 as date)) AND (date#1089 <= cast(2023-02-07 as date)))
               +- Project [tpep_pickup_datetime#70, tpep_dropoff_datetime#71, passenger_count#72, trip_distance#73, PULocationID#74, DOLocationID#75, payment_type#76, fare_amount#77, extra#78, mta_tax#79, tip_amount#80, tolls_amount#81, total_amount#82, congestion_surcharge#83, airport_fee#84, taxi_type#85, to_date('tpep_pickup_datetime, Some(yyyy-MM-dd HH:mm:ss)) AS date#1089]
                  +- Filter ((cast(fare_amount#77 as int) > 50) AND (cast(trip_distance#73 as int) < 1))
                     +- Relation[tpep_pickup_datetime#70,tpep_dropoff_datetime#71,passenger_count#72,trip_distance#73,PULocationID#74,DOLocationID#75,payment_type#76,fare_amount#77,extra#78,mta_tax#79,tip_amount#80,tolls_amount#81,total_amount#82,congestion_surcharge#83,airport_fee#84,taxi_type#85] csv

== Analyzed Logical Plan ==
date: string, trip_count: string
GlobalLimit 21
+- LocalLimit 21
   +- Project [cast(date#1089 as string) AS date#1130, cast(trip_count#1125L as string) AS trip_count#1131]
      +- Sort [date#1089 ASC NULLS FIRST], true
         +- Aggregate [date#1089], [date#1089, count(1) AS trip_count#1125L]
            +- Filter ((date#1089 >= cast(2023-02-01 as date)) AND (date#1089 <= cast(2023-02-07 as date)))
               +- Project [tpep_pickup_datetime#70, tpep_dropoff_datetime#71, passenger_count#72, trip_distance#73, PULocationID#74, DOLocationID#75, payment_type#76, fare_amount#77, extra#78, mta_tax#79, tip_amount#80, tolls_amount#81, total_amount#82, congestion_surcharge#83, airport_fee#84, taxi_type#85, to_date('tpep_pickup_datetime, Some(yyyy-MM-dd HH:mm:ss)) AS date#1089]
                  +- Filter ((cast(fare_amount#77 as int) > 50) AND (cast(trip_distance#73 as int) < 1))
                     +- Relation[tpep_pickup_datetime#70,tpep_dropoff_datetime#71,passenger_count#72,trip_distance#73,PULocationID#74,DOLocationID#75,payment_type#76,fare_amount#77,extra#78,mta_tax#79,tip_amount#80,tolls_amount#81,total_amount#82,congestion_surcharge#83,airport_fee#84,taxi_type#85] csv

== Optimized Logical Plan ==
GlobalLimit 21
+- LocalLimit 21
   +- Project [cast(date#1089 as string) AS date#1130, cast(trip_count#1125L as string) AS trip_count#1131]
      +- Sort [date#1089 ASC NULLS FIRST], true
         +- Aggregate [date#1089], [date#1089, count(1) AS trip_count#1125L]
            +- Project [cast(cast(unix_timestamp(tpep_pickup_datetime#70, yyyy-MM-dd HH:mm:ss, Some(GMT)) as timestamp) as date) AS date#1089]
               +- Filter (((((isnotnull(trip_distance#73) AND isnotnull(fare_amount#77)) AND (cast(fare_amount#77 as int) > 50)) AND (cast(trip_distance#73 as int) < 1)) AND (cast(cast(unix_timestamp(tpep_pickup_datetime#70, yyyy-MM-dd HH:mm:ss, Some(GMT)) as timestamp) as date) >= 19389)) AND (cast(cast(unix_timestamp(tpep_pickup_datetime#70, yyyy-MM-dd HH:mm:ss, Some(GMT)) as timestamp) as date) <= 19395))
                  +- Relation[tpep_pickup_datetime#70,tpep_dropoff_datetime#71,passenger_count#72,trip_distance#73,PULocationID#74,DOLocationID#75,payment_type#76,fare_amount#77,extra#78,mta_tax#79,tip_amount#80,tolls_amount#81,total_amount#82,congestion_surcharge#83,airport_fee#84,taxi_type#85] csv

== Physical Plan ==
TakeOrderedAndProject(limit=21, orderBy=[date#1089 ASC NULLS FIRST], output=[date#1130,trip_count#1131])
+- *(2) HashAggregate(keys=[date#1089], functions=[count(1)], output=[date#1089, trip_count#1125L])
   +- Exchange hashpartitioning(date#1089, 200), true, [id=#598]
      +- *(1) HashAggregate(keys=[date#1089], functions=[partial_count(1)], output=[date#1089, count#1135L])
         +- *(1) Project [cast(cast(unix_timestamp(tpep_pickup_datetime#70, yyyy-MM-dd HH:mm:ss, Some(GMT)) as timestamp) as date) AS date#1089]
            +- *(1) Filter (((((isnotnull(trip_distance#73) AND isnotnull(fare_amount#77)) AND (cast(fare_amount#77 as int) > 50)) AND (cast(trip_distance#73 as int) < 1)) AND (cast(cast(unix_timestamp(tpep_pickup_datetime#70, yyyy-MM-dd HH:mm:ss, Some(GMT)) as timestamp) as date) >= 19389)) AND (cast(cast(unix_timestamp(tpep_pickup_datetime#70, yyyy-MM-dd HH:mm:ss, Some(GMT)) as timestamp) as date) <= 19395))
               +- FileScan csv [tpep_pickup_datetime#70,trip_distance#73,fare_amount#77] Batched: false, DataFilters: [isnotnull(trip_distance#73), isnotnull(fare_amount#77), (cast(fare_amount#77 as int) > 50), (cas..., Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/nyc_taxi/yellow_tripdata/2023], PartitionFilters: [], PushedFilters: [IsNotNull(trip_distance), IsNotNull(fare_amount)], ReadSchema: struct<tpep_pickup_datetime:string,trip_distance:string,fare_amount:string>