NYC Yellow Taxi Data Analysis - Details for Query 3

Details for Query 3

Submitted Time: 2024/12/16 22:47:14
Duration: 1.4 min
Succeeded Jobs: 5

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" CollectLimit "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (8)"; 2 [labelType="html" label=" Project "]; } 3 [labelType="html" label="SortMergeJoin number of output rows: 11"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (5)"; 5 [labelType="html" label="Sort sort time: 0 ms peak memory: 136.0 MiB spill size: 0.0 B"]; } 6 [labelType="html" label="Exchange shuffle records written: 22,400,728 shuffle write time total (min, med, max (stageId: taskId)) 2.7 s (0 ms, 7 ms, 101 ms (stage 7.0: task 145)) records read: 222,916 local bytes read: 4.7 MiB fetch wait time: 0 ms remote bytes read: 8.0 MiB local blocks read: 57 remote blocks read: 67 data size total (min, med, max (stageId: taskId)) 7.5 GiB (0.0 B, 517.0 KiB, 495.9 MiB (stage 7.0: task 134)) shuffle bytes written total (min, med, max (stageId: taskId)) 1281.5 MiB (0.0 B, 113.0 KiB, 83.0 MiB (stage 7.0: task 134))"]; subgraph cluster7 { isCluster="true"; label="WholeStageCodegen (4)\n \nduration: total (min, med, max (stageId: taskId))\n53.9 s (0 ms, 15 ms, 3.6 s (stage 7.0: task 145))"; 8 [labelType="html" label=" Project "]; } 9 [labelType="html" label="SortMergeJoin number of output rows: 22,400,728"]; subgraph cluster10 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n56.4 s (4 ms, 22 ms, 3.6 s (stage 7.0: task 145))"; 11 [labelType="html" label="Sort sort time total (min, med, max (stageId: taskId)) 183 ms (0 ms, 0 ms, 22 ms (stage 7.0: task 50)) peak memory total (min, med, max (stageId: taskId)) 14.9 GiB (64.0 KiB, 64.1 MiB, 512.0 MiB (stage 7.0: task 134)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 7.0: task 48))"]; } 12 [labelType="html" label="Exchange shuffle records written: 22,400,728 shuffle write time total (min, med, max (stageId: taskId)) 2.2 s (48 ms, 113 ms, 158 ms (stage 6.0: task 30)) records read: 22,400,728 local bytes read total (min, med, max (stageId: taskId)) 709.9 MiB (0.0 B, 47.0 KiB, 51.3 MiB (stage 7.0: task 134)) fetch wait time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 7.0: task 48)) remote bytes read total (min, med, max (stageId: taskId)) 671.8 MiB (0.0 B, 48.0 KiB, 45.7 MiB (stage 7.0: task 145)) local blocks read: 1,426 remote blocks read: 1,423 data size total (min, med, max (stageId: taskId)) 6.3 GiB (101.8 MiB, 369.5 MiB, 369.8 MiB (stage 6.0: task 29)) shuffle bytes written total (min, med, max (stageId: taskId)) 1381.7 MiB (22.3 MiB, 78.5 MiB, 78.8 MiB (stage 6.0: task 39))"]; 13 [labelType="html" label="Scan csv number of files read: 7 metadata time: 1 ms size of files read: 2.2 GiB number of output rows: 22,400,728"]; subgraph cluster14 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: total (min, med, max (stageId: taskId))\n28.6 s (2 ms, 11 ms, 2.6 s (stage 7.0: task 134))"; 15 [labelType="html" label="Sort sort time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 7.0: task 48)) peak memory total (min, med, max (stageId: taskId)) 9.1 GiB (64.0 KiB, 64.1 MiB, 64.1 MiB (stage 7.0: task 48)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 7.0: task 48))"]; } 16 [labelType="html" label="Exchange shuffle records written: 265 shuffle write time total (min, med, max (stageId: taskId)) 60 ms (26 ms, 34 ms, 34 ms (stage 5.0: task 26)) records read: 266 local bytes read total (min, med, max (stageId: taskId)) 14.1 KiB (0.0 B, 110.0 B, 244.0 B (stage 7.0: task 148)) fetch wait time total (min, med, max (stageId: taskId)) 768 ms (0 ms, 0 ms, 60 ms (stage 7.0: task 153)) remote bytes read total (min, med, max (stageId: taskId)) 13.6 KiB (0.0 B, 0.0 B, 330.0 B (stage 7.0: task 71)) local blocks read: 103 remote blocks read: 97 data size total (min, med, max (stageId: taskId)) 22.2 KiB (10.9 KiB, 11.3 KiB, 11.3 KiB (stage 5.0: task 25)) shuffle bytes written total (min, med, max (stageId: taskId)) 27.6 KiB (13.6 KiB, 14.0 KiB, 14.0 KiB (stage 5.0: task 25))"]; subgraph cluster17 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n196 ms (88 ms, 108 ms, 108 ms (stage 5.0: task 26))"; 18 [labelType="html" label="Filter number of output rows: 265"]; 19 [labelType="html" label="Scan ExistingRDD number of output rows: 265"]; } subgraph cluster20 { isCluster="true"; label="WholeStageCodegen (7)\n \nduration: 163 ms"; 21 [labelType="html" label="Sort sort time: 0 ms peak memory: 64.1 MiB spill size: 0.0 B"]; } 2->0; 3->2; 5->3; 6->5; 8->6; 9->8; 11->9; 12->11; 13->12; 15->9; 16->15; 18->16; 19->18; 21->3; 16->21; }

CollectLimit 11

Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Pickup_Borough#147, Pickup_Zone#168, Pickup_service_zone#189, Borough#15 AS Dropoff_Borough#467, Zone#16 AS Dropoff_Zone#468, service_zone#17 AS Dropoff_service_zone#469, concat_ws( to , Pickup_Borough#147, Borough#15) AS route#470, cast(month(cast(tpep_pickup_datetime#55 as date)) as string) AS Month#471]

WholeStageCodegen (8)

SortMergeJoin [cast(DOLocationID#60 as int)], [LocationID#14], LeftOuter

Sort [cast(DOLocationID#60 as int) ASC NULLS FIRST], false, 0

WholeStageCodegen (5)

Exchange hashpartitioning(cast(DOLocationID#60 as int), 200), true, [id=#139]

Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, DOLocationID#60, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Borough#15 AS Pickup_Borough#147, Zone#16 AS Pickup_Zone#168, service_zone#17 AS Pickup_service_zone#189]

WholeStageCodegen (4)

SortMergeJoin [cast(PULocationID#59 as int)], [LocationID#14], LeftOuter

Sort [cast(PULocationID#59 as int) ASC NULLS FIRST], false, 0

WholeStageCodegen (1)

Exchange hashpartitioning(cast(PULocationID#59 as int), 200), true, [id=#112]

FileScan csv [tpep_pickup_datetime#55,tpep_dropoff_datetime#56,passenger_count#57,trip_distance#58,PULocationID#59,DOLocationID#60,payment_type#61,fare_amount#62,extra#63,mta_tax#64,tip_amount#65,tolls_amount#66,total_amount#67,congestion_surcharge#68,airport_fee#69,taxi_type#70] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/nyc_taxi/yellow_tripdata/2023/yellow_tripdata_..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<tpep_pickup_datetime:string,tpep_dropoff_datetime:string,passenger_count:string,trip_dista...

Sort [LocationID#14 ASC NULLS FIRST], false, 0

WholeStageCodegen (3)

Exchange hashpartitioning(LocationID#14, 200), true, [id=#131]

Filter isnotnull(LocationID#14)

Scan ExistingRDD[LocationID#14,Borough#15,Zone#16,service_zone#17]

WholeStageCodegen (2)

Sort [LocationID#14 ASC NULLS FIRST], false, 0

WholeStageCodegen (7)

Details

== Parsed Logical Plan ==
GlobalLimit 11
+- LocalLimit 11
   +- Project [cast(tpep_pickup_datetime#55 as string) AS tpep_pickup_datetime#450, cast(tpep_dropoff_datetime#56 as string) AS tpep_dropoff_datetime#451, cast(passenger_count#57 as string) AS passenger_count#452, cast(trip_distance#58 as string) AS trip_distance#453, cast(payment_type#61 as string) AS payment_type#454, cast(fare_amount#62 as string) AS fare_amount#455, cast(extra#63 as string) AS extra#456, cast(mta_tax#64 as string) AS mta_tax#457, cast(tip_amount#65 as string) AS tip_amount#458, cast(tolls_amount#66 as string) AS tolls_amount#459, cast(total_amount#67 as string) AS total_amount#460, cast(congestion_surcharge#68 as string) AS congestion_surcharge#461, cast(airport_fee#69 as string) AS airport_fee#462, cast(taxi_type#70 as string) AS taxi_type#463, cast(Pickup_Borough#147 as string) AS Pickup_Borough#464, cast(Pickup_Zone#168 as string) AS Pickup_Zone#465, cast(Pickup_service_zone#189 as string) AS Pickup_service_zone#466, cast(Dropoff_Borough#294 as string) AS Dropoff_Borough#467, cast(Dropoff_Zone#317 as string) AS Dropoff_Zone#468, cast(Dropoff_service_zone#340 as string) AS Dropoff_service_zone#469, cast(route#383 as string) AS route#470, cast(Month#405 as string) AS Month#471]
      +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Pickup_Borough#147, Pickup_Zone#168, Pickup_service_zone#189, Dropoff_Borough#294, Dropoff_Zone#317, Dropoff_service_zone#340, route#383, month(cast(tpep_pickup_datetime#55 as date)) AS Month#405]
         +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Pickup_Borough#147, Pickup_Zone#168, Pickup_service_zone#189, Dropoff_Borough#294, Dropoff_Zone#317, Dropoff_service_zone#340, concat_ws( to , Pickup_Borough#147, Dropoff_Borough#294) AS route#383]
            +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Pickup_Borough#147, Pickup_Zone#168, Pickup_service_zone#189, Dropoff_Borough#294, Dropoff_Zone#317, Dropoff_service_zone#340]
               +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, DOLocationID#60, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Pickup_Borough#147, Pickup_Zone#168, Pickup_service_zone#189, LocationID#14, Dropoff_Borough#294, Dropoff_Zone#317, service_zone#17 AS Dropoff_service_zone#340]
                  +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, DOLocationID#60, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Pickup_Borough#147, Pickup_Zone#168, Pickup_service_zone#189, LocationID#14, Dropoff_Borough#294, Zone#16 AS Dropoff_Zone#317, service_zone#17]
                     +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, DOLocationID#60, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Pickup_Borough#147, Pickup_Zone#168, Pickup_service_zone#189, LocationID#14, Borough#15 AS Dropoff_Borough#294, Zone#16, service_zone#17]
                        +- Join LeftOuter, (cast(DOLocationID#60 as int) = LocationID#14)
                           :- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, DOLocationID#60, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Pickup_Borough#147, Pickup_Zone#168, Pickup_service_zone#189]
                           :  +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, PULocationID#59, DOLocationID#60, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, LocationID#14, Pickup_Borough#147, Pickup_Zone#168, service_zone#17 AS Pickup_service_zone#189]
                           :     +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, PULocationID#59, DOLocationID#60, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, LocationID#14, Pickup_Borough#147, Zone#16 AS Pickup_Zone#168, service_zone#17]
                           :        +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, PULocationID#59, DOLocationID#60, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, LocationID#14, Borough#15 AS Pickup_Borough#147, Zone#16, service_zone#17]
                           :           +- Join LeftOuter, (cast(PULocationID#59 as int) = LocationID#14)
                           :              :- Relation[tpep_pickup_datetime#55,tpep_dropoff_datetime#56,passenger_count#57,trip_distance#58,PULocationID#59,DOLocationID#60,payment_type#61,fare_amount#62,extra#63,mta_tax#64,tip_amount#65,tolls_amount#66,total_amount#67,congestion_surcharge#68,airport_fee#69,taxi_type#70] csv
                           :              +- LogicalRDD [LocationID#14, Borough#15, Zone#16, service_zone#17], false
                           +- LogicalRDD [LocationID#14, Borough#15, Zone#16, service_zone#17], false

== Analyzed Logical Plan ==
tpep_pickup_datetime: string, tpep_dropoff_datetime: string, passenger_count: string, trip_distance: string, payment_type: string, fare_amount: string, extra: string, mta_tax: string, tip_amount: string, tolls_amount: string, total_amount: string, congestion_surcharge: string, airport_fee: string, taxi_type: string, Pickup_Borough: string, Pickup_Zone: string, Pickup_service_zone: string, Dropoff_Borough: string, Dropoff_Zone: string, Dropoff_service_zone: string, route: string, Month: string
GlobalLimit 11
+- LocalLimit 11
   +- Project [cast(tpep_pickup_datetime#55 as string) AS tpep_pickup_datetime#450, cast(tpep_dropoff_datetime#56 as string) AS tpep_dropoff_datetime#451, cast(passenger_count#57 as string) AS passenger_count#452, cast(trip_distance#58 as string) AS trip_distance#453, cast(payment_type#61 as string) AS payment_type#454, cast(fare_amount#62 as string) AS fare_amount#455, cast(extra#63 as string) AS extra#456, cast(mta_tax#64 as string) AS mta_tax#457, cast(tip_amount#65 as string) AS tip_amount#458, cast(tolls_amount#66 as string) AS tolls_amount#459, cast(total_amount#67 as string) AS total_amount#460, cast(congestion_surcharge#68 as string) AS congestion_surcharge#461, cast(airport_fee#69 as string) AS airport_fee#462, cast(taxi_type#70 as string) AS taxi_type#463, cast(Pickup_Borough#147 as string) AS Pickup_Borough#464, cast(Pickup_Zone#168 as string) AS Pickup_Zone#465, cast(Pickup_service_zone#189 as string) AS Pickup_service_zone#466, cast(Dropoff_Borough#294 as string) AS Dropoff_Borough#467, cast(Dropoff_Zone#317 as string) AS Dropoff_Zone#468, cast(Dropoff_service_zone#340 as string) AS Dropoff_service_zone#469, cast(route#383 as string) AS route#470, cast(Month#405 as string) AS Month#471]
      +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Pickup_Borough#147, Pickup_Zone#168, Pickup_service_zone#189, Dropoff_Borough#294, Dropoff_Zone#317, Dropoff_service_zone#340, route#383, month(cast(tpep_pickup_datetime#55 as date)) AS Month#405]
         +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Pickup_Borough#147, Pickup_Zone#168, Pickup_service_zone#189, Dropoff_Borough#294, Dropoff_Zone#317, Dropoff_service_zone#340, concat_ws( to , Pickup_Borough#147, Dropoff_Borough#294) AS route#383]
            +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Pickup_Borough#147, Pickup_Zone#168, Pickup_service_zone#189, Dropoff_Borough#294, Dropoff_Zone#317, Dropoff_service_zone#340]
               +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, DOLocationID#60, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Pickup_Borough#147, Pickup_Zone#168, Pickup_service_zone#189, LocationID#14, Dropoff_Borough#294, Dropoff_Zone#317, service_zone#17 AS Dropoff_service_zone#340]
                  +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, DOLocationID#60, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Pickup_Borough#147, Pickup_Zone#168, Pickup_service_zone#189, LocationID#14, Dropoff_Borough#294, Zone#16 AS Dropoff_Zone#317, service_zone#17]
                     +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, DOLocationID#60, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Pickup_Borough#147, Pickup_Zone#168, Pickup_service_zone#189, LocationID#14, Borough#15 AS Dropoff_Borough#294, Zone#16, service_zone#17]
                        +- Join LeftOuter, (cast(DOLocationID#60 as int) = LocationID#14)
                           :- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, DOLocationID#60, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Pickup_Borough#147, Pickup_Zone#168, Pickup_service_zone#189]
                           :  +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, PULocationID#59, DOLocationID#60, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, LocationID#14, Pickup_Borough#147, Pickup_Zone#168, service_zone#17 AS Pickup_service_zone#189]
                           :     +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, PULocationID#59, DOLocationID#60, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, LocationID#14, Pickup_Borough#147, Zone#16 AS Pickup_Zone#168, service_zone#17]
                           :        +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, PULocationID#59, DOLocationID#60, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, LocationID#14, Borough#15 AS Pickup_Borough#147, Zone#16, service_zone#17]
                           :           +- Join LeftOuter, (cast(PULocationID#59 as int) = LocationID#14)
                           :              :- Relation[tpep_pickup_datetime#55,tpep_dropoff_datetime#56,passenger_count#57,trip_distance#58,PULocationID#59,DOLocationID#60,payment_type#61,fare_amount#62,extra#63,mta_tax#64,tip_amount#65,tolls_amount#66,total_amount#67,congestion_surcharge#68,airport_fee#69,taxi_type#70] csv
                           :              +- LogicalRDD [LocationID#14, Borough#15, Zone#16, service_zone#17], false
                           +- LogicalRDD [LocationID#14, Borough#15, Zone#16, service_zone#17], false

== Optimized Logical Plan ==
GlobalLimit 11
+- LocalLimit 11
   +- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Pickup_Borough#147, Pickup_Zone#168, Pickup_service_zone#189, Borough#15 AS Dropoff_Borough#467, Zone#16 AS Dropoff_Zone#468, service_zone#17 AS Dropoff_service_zone#469, concat_ws( to , Pickup_Borough#147, Borough#15) AS route#470, cast(month(cast(tpep_pickup_datetime#55 as date)) as string) AS Month#471]
      +- Join LeftOuter, (cast(DOLocationID#60 as int) = LocationID#14)
         :- Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, DOLocationID#60, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Borough#15 AS Pickup_Borough#147, Zone#16 AS Pickup_Zone#168, service_zone#17 AS Pickup_service_zone#189]
         :  +- Join LeftOuter, (cast(PULocationID#59 as int) = LocationID#14)
         :     :- Relation[tpep_pickup_datetime#55,tpep_dropoff_datetime#56,passenger_count#57,trip_distance#58,PULocationID#59,DOLocationID#60,payment_type#61,fare_amount#62,extra#63,mta_tax#64,tip_amount#65,tolls_amount#66,total_amount#67,congestion_surcharge#68,airport_fee#69,taxi_type#70] csv
         :     +- Filter isnotnull(LocationID#14)
         :        +- LogicalRDD [LocationID#14, Borough#15, Zone#16, service_zone#17], false
         +- Filter isnotnull(LocationID#14)
            +- LogicalRDD [LocationID#14, Borough#15, Zone#16, service_zone#17], false

== Physical Plan ==
CollectLimit 11
+- *(8) Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Pickup_Borough#147, Pickup_Zone#168, Pickup_service_zone#189, Borough#15 AS Dropoff_Borough#467, Zone#16 AS Dropoff_Zone#468, service_zone#17 AS Dropoff_service_zone#469, concat_ws( to , Pickup_Borough#147, Borough#15) AS route#470, cast(month(cast(tpep_pickup_datetime#55 as date)) as string) AS Month#471]
   +- SortMergeJoin [cast(DOLocationID#60 as int)], [LocationID#14], LeftOuter
      :- *(5) Sort [cast(DOLocationID#60 as int) ASC NULLS FIRST], false, 0
      :  +- Exchange hashpartitioning(cast(DOLocationID#60 as int), 200), true, [id=#139]
      :     +- *(4) Project [tpep_pickup_datetime#55, tpep_dropoff_datetime#56, passenger_count#57, trip_distance#58, DOLocationID#60, payment_type#61, fare_amount#62, extra#63, mta_tax#64, tip_amount#65, tolls_amount#66, total_amount#67, congestion_surcharge#68, airport_fee#69, taxi_type#70, Borough#15 AS Pickup_Borough#147, Zone#16 AS Pickup_Zone#168, service_zone#17 AS Pickup_service_zone#189]
      :        +- SortMergeJoin [cast(PULocationID#59 as int)], [LocationID#14], LeftOuter
      :           :- *(1) Sort [cast(PULocationID#59 as int) ASC NULLS FIRST], false, 0
      :           :  +- Exchange hashpartitioning(cast(PULocationID#59 as int), 200), true, [id=#112]
      :           :     +- FileScan csv [tpep_pickup_datetime#55,tpep_dropoff_datetime#56,passenger_count#57,trip_distance#58,PULocationID#59,DOLocationID#60,payment_type#61,fare_amount#62,extra#63,mta_tax#64,tip_amount#65,tolls_amount#66,total_amount#67,congestion_surcharge#68,airport_fee#69,taxi_type#70] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/nyc_taxi/yellow_tripdata/2023/yellow_tripdata_..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<tpep_pickup_datetime:string,tpep_dropoff_datetime:string,passenger_count:string,trip_dista...
      :           +- *(3) Sort [LocationID#14 ASC NULLS FIRST], false, 0
      :              +- Exchange hashpartitioning(LocationID#14, 200), true, [id=#131]
      :                 +- *(2) Filter isnotnull(LocationID#14)
      :                    +- *(2) Scan ExistingRDD[LocationID#14,Borough#15,Zone#16,service_zone#17]
      +- *(7) Sort [LocationID#14 ASC NULLS FIRST], false, 0
         +- ReusedExchange [LocationID#14, Borough#15, Zone#16, service_zone#17], Exchange hashpartitioning(LocationID#14, 200), true, [id=#131]