GlobalAirportsAnalysis - Details for Query 3

Details for Query 3

Submitted Time: 2025/02/17 15:09:42
Duration: 2.8 min
Succeeded Jobs: 5

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label="Execute InsertIntoHadoopFsRelationCommand number of written files: 1 written output: 2.7 GiB number of output rows: 6,905,288 number of dynamic part: 0"]; 1 [labelType="html" label=" Coalesce "]; subgraph cluster2 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: 2.7 m"; 3 [labelType="html" label=" Project "]; } 4 [labelType="html" label="Scan csv number of files read: 1 metadata time: 0 ms size of files read: 2.7 GiB number of output rows: 6,905,288"]; 1->0; 3->1; 4->3; }

Execute InsertIntoHadoopFsRelationCommand s3a://object-bucket-eex654-45a8c32f-acaa-44b7-aa06-d996030307e6/trip_counts_by_hour.csv, false, CSV, Map(header -> true, path -> s3a://object-bucket-eex654-45a8c32f-acaa-44b7-aa06-d996030307e6/trip_counts_by_hour.csv), Overwrite, [Trip ID, Taxi ID, Trip Start Timestamp, Trip End Timestamp, Trip Seconds, Trip Miles, Pickup Census Tract, Dropoff Census Tract, Pickup Community Area, Dropoff Community Area, Fare, Tips, Tolls, Extras, Trip Total, Payment Type, Company, Pickup Centroid Latitude, Pickup Centroid Longitude, Pickup Centroid Location, Dropoff Centroid Latitude, Dropoff Centroid Longitude, Dropoff Centroid Location, Start Hour]

Coalesce 1

Project [Trip ID#78, Taxi ID#79, CASE WHEN isnull(gettimestamp(Trip Start Timestamp#80, MM/dd/yyyy hh:mm:ss a, Some(GMT))) THEN gettimestamp(gettimestamp(Trip Start Timestamp#80, MM/dd/yyyy hh:mm:ss a, Some(GMT)), MM/dd/yyyy HH:mm, Some(GMT)) ELSE gettimestamp(Trip Start Timestamp#80, MM/dd/yyyy hh:mm:ss a, Some(GMT)) END AS Trip Start Timestamp#148, Trip End Timestamp#81, Trip Seconds#82, Trip Miles#83, Pickup Census Tract#84L, Dropoff Census Tract#85L, Pickup Community Area#86, Dropoff Community Area#87, Fare#88, Tips#89, Tolls#90, Extras#91, Trip Total#92, Payment Type#93, Company#94, Pickup Centroid Latitude#95, Pickup Centroid Longitude#96, Pickup Centroid Location#97, Dropoff Centroid Latitude#98, Dropoff Centroid Longitude#99, Dropoff Centroid Location#100, hour(CASE WHEN isnull(gettimestamp(Trip Start Timestamp#80, MM/dd/yyyy hh:mm:ss a, Some(GMT))) THEN gettimestamp(gettimestamp(Trip Start Timestamp#80, MM/dd/yyyy hh:mm:ss a, Some(GMT)), MM/dd/yyyy HH:mm, Some(GMT)) ELSE gettimestamp(Trip Start Timestamp#80, MM/dd/yyyy hh:mm:ss a, Some(GMT)) END, Some(GMT)) AS Start Hour#172]

WholeStageCodegen (1)

FileScan csv [Trip ID#78,Taxi ID#79,Trip Start Timestamp#80,Trip End Timestamp#81,Trip Seconds#82,Trip Miles#83,Pickup Census Tract#84L,Dropoff Census Tract#85L,Pickup Community Area#86,Dropoff Community Area#87,Fare#88,Tips#89,Tolls#90,Extras#91,Trip Total#92,Payment Type#93,Company#94,Pickup Centroid Latitude#95,Pickup Centroid Longitude#96,Pickup Centroid Location#97,Dropoff Centroid Latitude#98,Dropoff Centroid Longitude#99,Dropoff Centroid Location#100] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/Chicago_Taxitrips/chicago_taxi_trips.csv], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<Trip ID:string,Taxi ID:string,Trip Start Timestamp:string,Trip End Timestamp:string,Trip S...

Details

== Parsed Logical Plan ==
InsertIntoHadoopFsRelationCommand s3a://object-bucket-eex654-45a8c32f-acaa-44b7-aa06-d996030307e6/trip_counts_by_hour.csv, false, CSV, Map(header -> true, path -> s3a://object-bucket-eex654-45a8c32f-acaa-44b7-aa06-d996030307e6/trip_counts_by_hour.csv), Overwrite, [Trip ID, Taxi ID, Trip Start Timestamp, Trip End Timestamp, Trip Seconds, Trip Miles, Pickup Census Tract, Dropoff Census Tract, Pickup Community Area, Dropoff Community Area, Fare, Tips, Tolls, Extras, Trip Total, Payment Type, Company, Pickup Centroid Latitude, Pickup Centroid Longitude, Pickup Centroid Location, Dropoff Centroid Latitude, Dropoff Centroid Longitude, Dropoff Centroid  Location, Start Hour]
+- Repartition 1, false
   +- Project [Trip ID#78, Taxi ID#79, Trip Start Timestamp#148, Trip End Timestamp#81, Trip Seconds#82, Trip Miles#83, Pickup Census Tract#84L, Dropoff Census Tract#85L, Pickup Community Area#86, Dropoff Community Area#87, Fare#88, Tips#89, Tolls#90, Extras#91, Trip Total#92, Payment Type#93, Company#94, Pickup Centroid Latitude#95, Pickup Centroid Longitude#96, Pickup Centroid Location#97, Dropoff Centroid Latitude#98, Dropoff Centroid Longitude#99, Dropoff Centroid  Location#100, hour(Trip Start Timestamp#148, Some(GMT)) AS Start Hour#172]
      +- Project [Trip ID#78, Taxi ID#79, CASE WHEN isnull(Trip Start Timestamp#124) THEN to_timestamp('Trip Start Timestamp, Some(MM/dd/yyyy HH:mm)) ELSE Trip Start Timestamp#124 END AS Trip Start Timestamp#148, Trip End Timestamp#81, Trip Seconds#82, Trip Miles#83, Pickup Census Tract#84L, Dropoff Census Tract#85L, Pickup Community Area#86, Dropoff Community Area#87, Fare#88, Tips#89, Tolls#90, Extras#91, Trip Total#92, Payment Type#93, Company#94, Pickup Centroid Latitude#95, Pickup Centroid Longitude#96, Pickup Centroid Location#97, Dropoff Centroid Latitude#98, Dropoff Centroid Longitude#99, Dropoff Centroid  Location#100]
         +- Project [Trip ID#78, Taxi ID#79, to_timestamp('Trip Start Timestamp, Some(MM/dd/yyyy hh:mm:ss a)) AS Trip Start Timestamp#124, Trip End Timestamp#81, Trip Seconds#82, Trip Miles#83, Pickup Census Tract#84L, Dropoff Census Tract#85L, Pickup Community Area#86, Dropoff Community Area#87, Fare#88, Tips#89, Tolls#90, Extras#91, Trip Total#92, Payment Type#93, Company#94, Pickup Centroid Latitude#95, Pickup Centroid Longitude#96, Pickup Centroid Location#97, Dropoff Centroid Latitude#98, Dropoff Centroid Longitude#99, Dropoff Centroid  Location#100]
            +- Relation[Trip ID#78,Taxi ID#79,Trip Start Timestamp#80,Trip End Timestamp#81,Trip Seconds#82,Trip Miles#83,Pickup Census Tract#84L,Dropoff Census Tract#85L,Pickup Community Area#86,Dropoff Community Area#87,Fare#88,Tips#89,Tolls#90,Extras#91,Trip Total#92,Payment Type#93,Company#94,Pickup Centroid Latitude#95,Pickup Centroid Longitude#96,Pickup Centroid Location#97,Dropoff Centroid Latitude#98,Dropoff Centroid Longitude#99,Dropoff Centroid  Location#100] csv

== Analyzed Logical Plan ==

InsertIntoHadoopFsRelationCommand s3a://object-bucket-eex654-45a8c32f-acaa-44b7-aa06-d996030307e6/trip_counts_by_hour.csv, false, CSV, Map(header -> true, path -> s3a://object-bucket-eex654-45a8c32f-acaa-44b7-aa06-d996030307e6/trip_counts_by_hour.csv), Overwrite, [Trip ID, Taxi ID, Trip Start Timestamp, Trip End Timestamp, Trip Seconds, Trip Miles, Pickup Census Tract, Dropoff Census Tract, Pickup Community Area, Dropoff Community Area, Fare, Tips, Tolls, Extras, Trip Total, Payment Type, Company, Pickup Centroid Latitude, Pickup Centroid Longitude, Pickup Centroid Location, Dropoff Centroid Latitude, Dropoff Centroid Longitude, Dropoff Centroid  Location, Start Hour]
+- Repartition 1, false
   +- Project [Trip ID#78, Taxi ID#79, Trip Start Timestamp#148, Trip End Timestamp#81, Trip Seconds#82, Trip Miles#83, Pickup Census Tract#84L, Dropoff Census Tract#85L, Pickup Community Area#86, Dropoff Community Area#87, Fare#88, Tips#89, Tolls#90, Extras#91, Trip Total#92, Payment Type#93, Company#94, Pickup Centroid Latitude#95, Pickup Centroid Longitude#96, Pickup Centroid Location#97, Dropoff Centroid Latitude#98, Dropoff Centroid Longitude#99, Dropoff Centroid  Location#100, hour(Trip Start Timestamp#148, Some(GMT)) AS Start Hour#172]
      +- Project [Trip ID#78, Taxi ID#79, CASE WHEN isnull(Trip Start Timestamp#124) THEN to_timestamp('Trip Start Timestamp, Some(MM/dd/yyyy HH:mm)) ELSE Trip Start Timestamp#124 END AS Trip Start Timestamp#148, Trip End Timestamp#81, Trip Seconds#82, Trip Miles#83, Pickup Census Tract#84L, Dropoff Census Tract#85L, Pickup Community Area#86, Dropoff Community Area#87, Fare#88, Tips#89, Tolls#90, Extras#91, Trip Total#92, Payment Type#93, Company#94, Pickup Centroid Latitude#95, Pickup Centroid Longitude#96, Pickup Centroid Location#97, Dropoff Centroid Latitude#98, Dropoff Centroid Longitude#99, Dropoff Centroid  Location#100]
         +- Project [Trip ID#78, Taxi ID#79, to_timestamp('Trip Start Timestamp, Some(MM/dd/yyyy hh:mm:ss a)) AS Trip Start Timestamp#124, Trip End Timestamp#81, Trip Seconds#82, Trip Miles#83, Pickup Census Tract#84L, Dropoff Census Tract#85L, Pickup Community Area#86, Dropoff Community Area#87, Fare#88, Tips#89, Tolls#90, Extras#91, Trip Total#92, Payment Type#93, Company#94, Pickup Centroid Latitude#95, Pickup Centroid Longitude#96, Pickup Centroid Location#97, Dropoff Centroid Latitude#98, Dropoff Centroid Longitude#99, Dropoff Centroid  Location#100]
            +- Relation[Trip ID#78,Taxi ID#79,Trip Start Timestamp#80,Trip End Timestamp#81,Trip Seconds#82,Trip Miles#83,Pickup Census Tract#84L,Dropoff Census Tract#85L,Pickup Community Area#86,Dropoff Community Area#87,Fare#88,Tips#89,Tolls#90,Extras#91,Trip Total#92,Payment Type#93,Company#94,Pickup Centroid Latitude#95,Pickup Centroid Longitude#96,Pickup Centroid Location#97,Dropoff Centroid Latitude#98,Dropoff Centroid Longitude#99,Dropoff Centroid  Location#100] csv

== Optimized Logical Plan ==
InsertIntoHadoopFsRelationCommand s3a://object-bucket-eex654-45a8c32f-acaa-44b7-aa06-d996030307e6/trip_counts_by_hour.csv, false, CSV, Map(header -> true, path -> s3a://object-bucket-eex654-45a8c32f-acaa-44b7-aa06-d996030307e6/trip_counts_by_hour.csv), Overwrite, [Trip ID, Taxi ID, Trip Start Timestamp, Trip End Timestamp, Trip Seconds, Trip Miles, Pickup Census Tract, Dropoff Census Tract, Pickup Community Area, Dropoff Community Area, Fare, Tips, Tolls, Extras, Trip Total, Payment Type, Company, Pickup Centroid Latitude, Pickup Centroid Longitude, Pickup Centroid Location, Dropoff Centroid Latitude, Dropoff Centroid Longitude, Dropoff Centroid  Location, Start Hour]
+- Repartition 1, false
   +- Project [Trip ID#78, Taxi ID#79, CASE WHEN isnull(gettimestamp(Trip Start Timestamp#80, MM/dd/yyyy hh:mm:ss a, Some(GMT))) THEN gettimestamp(gettimestamp(Trip Start Timestamp#80, MM/dd/yyyy hh:mm:ss a, Some(GMT)), MM/dd/yyyy HH:mm, Some(GMT)) ELSE gettimestamp(Trip Start Timestamp#80, MM/dd/yyyy hh:mm:ss a, Some(GMT)) END AS Trip Start Timestamp#148, Trip End Timestamp#81, Trip Seconds#82, Trip Miles#83, Pickup Census Tract#84L, Dropoff Census Tract#85L, Pickup Community Area#86, Dropoff Community Area#87, Fare#88, Tips#89, Tolls#90, Extras#91, Trip Total#92, Payment Type#93, Company#94, Pickup Centroid Latitude#95, Pickup Centroid Longitude#96, Pickup Centroid Location#97, Dropoff Centroid Latitude#98, Dropoff Centroid Longitude#99, Dropoff Centroid  Location#100, hour(CASE WHEN isnull(gettimestamp(Trip Start Timestamp#80, MM/dd/yyyy hh:mm:ss a, Some(GMT))) THEN gettimestamp(gettimestamp(Trip Start Timestamp#80, MM/dd/yyyy hh:mm:ss a, Some(GMT)), MM/dd/yyyy HH:mm, Some(GMT)) ELSE gettimestamp(Trip Start Timestamp#80, MM/dd/yyyy hh:mm:ss a, Some(GMT)) END, Some(GMT)) AS Start Hour#172]
      +- Relation[Trip ID#78,Taxi ID#79,Trip Start Timestamp#80,Trip End Timestamp#81,Trip Seconds#82,Trip Miles#83,Pickup Census Tract#84L,Dropoff Census Tract#85L,Pickup Community Area#86,Dropoff Community Area#87,Fare#88,Tips#89,Tolls#90,Extras#91,Trip Total#92,Payment Type#93,Company#94,Pickup Centroid Latitude#95,Pickup Centroid Longitude#96,Pickup Centroid Location#97,Dropoff Centroid Latitude#98,Dropoff Centroid Longitude#99,Dropoff Centroid  Location#100] csv

== Physical Plan ==
Execute InsertIntoHadoopFsRelationCommand s3a://object-bucket-eex654-45a8c32f-acaa-44b7-aa06-d996030307e6/trip_counts_by_hour.csv, false, CSV, Map(header -> true, path -> s3a://object-bucket-eex654-45a8c32f-acaa-44b7-aa06-d996030307e6/trip_counts_by_hour.csv), Overwrite, [Trip ID, Taxi ID, Trip Start Timestamp, Trip End Timestamp, Trip Seconds, Trip Miles, Pickup Census Tract, Dropoff Census Tract, Pickup Community Area, Dropoff Community Area, Fare, Tips, Tolls, Extras, Trip Total, Payment Type, Company, Pickup Centroid Latitude, Pickup Centroid Longitude, Pickup Centroid Location, Dropoff Centroid Latitude, Dropoff Centroid Longitude, Dropoff Centroid  Location, Start Hour]
+- Coalesce 1
   +- *(1) Project [Trip ID#78, Taxi ID#79, CASE WHEN isnull(gettimestamp(Trip Start Timestamp#80, MM/dd/yyyy hh:mm:ss a, Some(GMT))) THEN gettimestamp(gettimestamp(Trip Start Timestamp#80, MM/dd/yyyy hh:mm:ss a, Some(GMT)), MM/dd/yyyy HH:mm, Some(GMT)) ELSE gettimestamp(Trip Start Timestamp#80, MM/dd/yyyy hh:mm:ss a, Some(GMT)) END AS Trip Start Timestamp#148, Trip End Timestamp#81, Trip Seconds#82, Trip Miles#83, Pickup Census Tract#84L, Dropoff Census Tract#85L, Pickup Community Area#86, Dropoff Community Area#87, Fare#88, Tips#89, Tolls#90, Extras#91, Trip Total#92, Payment Type#93, Company#94, Pickup Centroid Latitude#95, Pickup Centroid Longitude#96, Pickup Centroid Location#97, Dropoff Centroid Latitude#98, Dropoff Centroid Longitude#99, Dropoff Centroid  Location#100, hour(CASE WHEN isnull(gettimestamp(Trip Start Timestamp#80, MM/dd/yyyy hh:mm:ss a, Some(GMT))) THEN gettimestamp(gettimestamp(Trip Start Timestamp#80, MM/dd/yyyy hh:mm:ss a, Some(GMT)), MM/dd/yyyy HH:mm, Some(GMT)) ELSE gettimestamp(Trip Start Timestamp#80, MM/dd/yyyy hh:mm:ss a, Some(GMT)) END, Some(GMT)) AS Start Hour#172]
      +- FileScan csv [Trip ID#78,Taxi ID#79,Trip Start Timestamp#80,Trip End Timestamp#81,Trip Seconds#82,Trip Miles#83,Pickup Census Tract#84L,Dropoff Census Tract#85L,Pickup Community Area#86,Dropoff Community Area#87,Fare#88,Tips#89,Tolls#90,Extras#91,Trip Total#92,Payment Type#93,Company#94,Pickup Centroid Latitude#95,Pickup Centroid Longitude#96,Pickup Centroid Location#97,Dropoff Centroid Latitude#98,Dropoff Centroid Longitude#99,Dropoff Centroid  Location#100] Batched: false, DataFilters: [], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/Chicago_Taxitrips/chicago_taxi_trips.csv], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<Trip ID:string,Taxi ID:string,Trip Start Timestamp:string,Trip End Timestamp:string,Trip S...