ChicagoTaxiTripsAnalysis - Details for Query 3

Details for Query 3

Submitted Time: 2025/03/20 20:17:19
Duration: 54 s
Succeeded Jobs: 3 4 5 6

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" CollectLimit "]; 1 [labelType="html" label="SortAggregate number of output rows: 23"]; subgraph cluster2 { isCluster="true"; label="WholeStageCodegen (4)\n \nduration: total (min, med, max (stageId: taskId))\n588 ms (1 ms, 2 ms, 136 ms (stage 7.0: task 70))"; 3 [labelType="html" label="Sort sort time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 5.0: task 69)) peak memory total (min, med, max (stageId: taskId)) 1090.8 MiB (64.0 KiB, 64.0 KiB, 64.1 MiB (stage 9.0: task 74)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 5.0: task 69))"]; } 4 [labelType="html" label="Exchange shuffle records written: 3,432 shuffle write time total (min, med, max (stageId: taskId)) 371 ms (5 ms, 7 ms, 22 ms (stage 4.0: task 26)) records read: 1,012 local bytes read total (min, med, max (stageId: taskId)) 56.4 KiB (0.0 B, 0.0 B, 4.0 KiB (stage 11.0: task 96)) fetch wait time total (min, med, max (stageId: taskId)) 28 ms (0 ms, 0 ms, 15 ms (stage 7.0: task 70)) remote bytes read total (min, med, max (stageId: taskId)) 54.2 KiB (0.0 B, 0.0 B, 3.5 KiB (stage 11.0: task 94)) local blocks read: 380 remote blocks read: 368 data size total (min, med, max (stageId: taskId)) 522.5 KiB (11.6 KiB, 12.2 KiB, 12.2 KiB (stage 4.0: task 47)) shuffle bytes written total (min, med, max (stageId: taskId)) 412.7 KiB (9.3 KiB, 9.4 KiB, 9.4 KiB (stage 4.0: task 25))"]; 5 [labelType="html" label="SortAggregate number of output rows: 3,432"]; subgraph cluster6 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: total (min, med, max (stageId: taskId))\n1.7 m (778 ms, 2.4 s, 3.6 s (stage 4.0: task 26))"; 7 [labelType="html" label="Sort sort time total (min, med, max (stageId: taskId)) 5.2 s (33 ms, 116 ms, 204 ms (stage 4.0: task 26)) peak memory total (min, med, max (stageId: taskId)) 3.1 GiB (68.0 MiB, 72.0 MiB, 72.0 MiB (stage 4.0: task 25)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 4.0: task 25))"]; } 8 [labelType="html" label=" Union "]; subgraph cluster9 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n47.0 s (0 ms, 776 ms, 3.0 s (stage 4.0: task 26))"; 10 [labelType="html" label=" Project "]; } 11 [labelType="html" label="Scan csv number of files read: 1 metadata time: 0 ms size of files read: 2.7 GiB number of output rows: 6,905,288"]; subgraph cluster12 { isCluster="true"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n44.4 s (0 ms, 684 ms, 2.4 s (stage 4.0: task 56))"; 13 [labelType="html" label=" Project "]; } 14 [labelType="html" label="Scan csv number of files read: 1 metadata time: 0 ms size of files read: 2.7 GiB number of output rows: 6,905,288"]; 1->0; 3->1; 4->3; 5->4; 7->5; 8->7; 10->8; 11->10; 13->8; 14->13; }

CollectLimit 21

SortAggregate(key=[id#90], functions=[first(Longitude#91, false), first(Latitude#92, false), first(Census Tract#93, false)])

Sort [id#90 ASC NULLS FIRST], false, 0

WholeStageCodegen (4)

Exchange hashpartitioning(id#90, 200), true, [id=#93]

SortAggregate(key=[id#90], functions=[partial_first(Longitude#91, false), partial_first(Latitude#92, false), partial_first(Census Tract#93, false)])

Sort [id#90 ASC NULLS FIRST], false, 0

WholeStageCodegen (3)

Union

Project [Pickup Community Area#24 AS id#90, Pickup Centroid Longitude AS Longitude#91, Pickup Centroid Latitude AS Latitude#92, Pickup Census Tract AS Census Tract#93]

WholeStageCodegen (1)

FileScan csv [Pickup Community Area#24] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/Chicago_Taxitrips/chicago_taxi_trips.csv], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<Pickup Community Area:string>

Project [Dropoff Community Area#25 AS id#98, Dropoff Centroid Longitude AS Longitude#99, Dropoff Centroid Latitude AS Latitude#100, Dropoff Census Tract AS Census Tract#101]

WholeStageCodegen (2)

FileScan csv [Dropoff Community Area#25] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/Chicago_Taxitrips/chicago_taxi_trips.csv], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<Dropoff Community Area:string>

Details

== Parsed Logical Plan ==
GlobalLimit 21
+- LocalLimit 21
   +- Project [cast(id#90 as string) AS id#132, cast(Longitude#91 as string) AS Longitude#133, cast(Latitude#92 as string) AS Latitude#134, cast(Census Tract#93 as string) AS Census Tract#135]
      +- Deduplicate [id#90]
         +- Union
            :- Project [Pickup Community Area#24 AS id#90, Pickup Centroid Longitude AS Longitude#91, Pickup Centroid Latitude AS Latitude#92, Pickup Census Tract AS Census Tract#93]
            :  +- Relation[Trip ID#16,Taxi ID#17,Trip Start Timestamp#18,Trip End Timestamp#19,Trip Seconds#20,Trip Miles#21,Pickup Census Tract#22,Dropoff Census Tract#23,Pickup Community Area#24,Dropoff Community Area#25,Fare#26,Tips#27,Tolls#28,Extras#29,Trip Total#30,Payment Type#31,Company#32,Pickup Centroid Latitude#33,Pickup Centroid Longitude#34,Pickup Centroid Location#35,Dropoff Centroid Latitude#36,Dropoff Centroid Longitude#37,Dropoff Centroid  Location#38] csv
            +- Project [Dropoff Community Area#25 AS id#98, Dropoff Centroid Longitude AS Longitude#99, Dropoff Centroid Latitude AS Latitude#100, Dropoff Census Tract AS Census Tract#101]
               +- Relation[Trip ID#16,Taxi ID#17,Trip Start Timestamp#18,Trip End Timestamp#19,Trip Seconds#20,Trip Miles#21,Pickup Census Tract#22,Dropoff Census Tract#23,Pickup Community Area#24,Dropoff Community Area#25,Fare#26,Tips#27,Tolls#28,Extras#29,Trip Total#30,Payment Type#31,Company#32,Pickup Centroid Latitude#33,Pickup Centroid Longitude#34,Pickup Centroid Location#35,Dropoff Centroid Latitude#36,Dropoff Centroid Longitude#37,Dropoff Centroid  Location#38] csv

== Analyzed Logical Plan ==
id: string, Longitude: string, Latitude: string, Census Tract: string
GlobalLimit 21
+- LocalLimit 21
   +- Project [cast(id#90 as string) AS id#132, cast(Longitude#91 as string) AS Longitude#133, cast(Latitude#92 as string) AS Latitude#134, cast(Census Tract#93 as string) AS Census Tract#135]
      +- Deduplicate [id#90]
         +- Union
            :- Project [Pickup Community Area#24 AS id#90, Pickup Centroid Longitude AS Longitude#91, Pickup Centroid Latitude AS Latitude#92, Pickup Census Tract AS Census Tract#93]
            :  +- Relation[Trip ID#16,Taxi ID#17,Trip Start Timestamp#18,Trip End Timestamp#19,Trip Seconds#20,Trip Miles#21,Pickup Census Tract#22,Dropoff Census Tract#23,Pickup Community Area#24,Dropoff Community Area#25,Fare#26,Tips#27,Tolls#28,Extras#29,Trip Total#30,Payment Type#31,Company#32,Pickup Centroid Latitude#33,Pickup Centroid Longitude#34,Pickup Centroid Location#35,Dropoff Centroid Latitude#36,Dropoff Centroid Longitude#37,Dropoff Centroid  Location#38] csv
            +- Project [Dropoff Community Area#25 AS id#98, Dropoff Centroid Longitude AS Longitude#99, Dropoff Centroid Latitude AS Latitude#100, Dropoff Census Tract AS Census Tract#101]
               +- Relation[Trip ID#16,Taxi ID#17,Trip Start Timestamp#18,Trip End Timestamp#19,Trip Seconds#20,Trip Miles#21,Pickup Census Tract#22,Dropoff Census Tract#23,Pickup Community Area#24,Dropoff Community Area#25,Fare#26,Tips#27,Tolls#28,Extras#29,Trip Total#30,Payment Type#31,Company#32,Pickup Centroid Latitude#33,Pickup Centroid Longitude#34,Pickup Centroid Location#35,Dropoff Centroid Latitude#36,Dropoff Centroid Longitude#37,Dropoff Centroid  Location#38] csv

== Optimized Logical Plan ==
GlobalLimit 21
+- LocalLimit 21
   +- Aggregate [id#90], [id#90, first(Longitude#91, false) AS Longitude#91, first(Latitude#92, false) AS Latitude#92, first(Census Tract#93, false) AS Census Tract#93]
      +- Union
         :- Project [Pickup Community Area#24 AS id#90, Pickup Centroid Longitude AS Longitude#91, Pickup Centroid Latitude AS Latitude#92, Pickup Census Tract AS Census Tract#93]
         :  +- Relation[Trip ID#16,Taxi ID#17,Trip Start Timestamp#18,Trip End Timestamp#19,Trip Seconds#20,Trip Miles#21,Pickup Census Tract#22,Dropoff Census Tract#23,Pickup Community Area#24,Dropoff Community Area#25,Fare#26,Tips#27,Tolls#28,Extras#29,Trip Total#30,Payment Type#31,Company#32,Pickup Centroid Latitude#33,Pickup Centroid Longitude#34,Pickup Centroid Location#35,Dropoff Centroid Latitude#36,Dropoff Centroid Longitude#37,Dropoff Centroid  Location#38] csv
         +- Project [Dropoff Community Area#25 AS id#98, Dropoff Centroid Longitude AS Longitude#99, Dropoff Centroid Latitude AS Latitude#100, Dropoff Census Tract AS Census Tract#101]
            +- Relation[Trip ID#16,Taxi ID#17,Trip Start Timestamp#18,Trip End Timestamp#19,Trip Seconds#20,Trip Miles#21,Pickup Census Tract#22,Dropoff Census Tract#23,Pickup Community Area#24,Dropoff Community Area#25,Fare#26,Tips#27,Tolls#28,Extras#29,Trip Total#30,Payment Type#31,Company#32,Pickup Centroid Latitude#33,Pickup Centroid Longitude#34,Pickup Centroid Location#35,Dropoff Centroid Latitude#36,Dropoff Centroid Longitude#37,Dropoff Centroid  Location#38] csv

== Physical Plan ==
CollectLimit 21
+- SortAggregate(key=[id#90], functions=[first(Longitude#91, false), first(Latitude#92, false), first(Census Tract#93, false)], output=[id#90, Longitude#91, Latitude#92, Census Tract#93])
   +- *(4) Sort [id#90 ASC NULLS FIRST], false, 0
      +- Exchange hashpartitioning(id#90, 200), true, [id=#93]
         +- SortAggregate(key=[id#90], functions=[partial_first(Longitude#91, false), partial_first(Latitude#92, false), partial_first(Census Tract#93, false)], output=[id#90, first#149, valueSet#150, first#151, valueSet#152, first#153, valueSet#154])
            +- *(3) Sort [id#90 ASC NULLS FIRST], false, 0
               +- Union
                  :- *(1) Project [Pickup Community Area#24 AS id#90, Pickup Centroid Longitude AS Longitude#91, Pickup Centroid Latitude AS Latitude#92, Pickup Census Tract AS Census Tract#93]
                  :  +- FileScan csv [Pickup Community Area#24] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/Chicago_Taxitrips/chicago_taxi_trips.csv], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<Pickup Community Area:string>
                  +- *(2) Project [Dropoff Community Area#25 AS id#98, Dropoff Centroid Longitude AS Longitude#99, Dropoff Centroid Latitude AS Latitude#100, Dropoff Census Tract AS Census Tract#101]
                     +- FileScan csv [Dropoff Community Area#25] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/Chicago_Taxitrips/chicago_taxi_trips.csv], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<Dropoff Community Area:string>