task2 - Details for Query 5

Details for Query 5

Submitted Time: 2024/11/28 21:55:09
Duration: 6 s
Succeeded Jobs: 7

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" CollectLimit "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (5)"; 2 [labelType="html" label=" Project "]; 3 [labelType="html" label="SortMergeJoin number of output rows: 11"]; } subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (2)"; 5 [labelType="html" label="Sort sort time: 0 ms peak memory: 64.1 MiB spill size: 0.0 B"]; } 6 [labelType="html" label="Exchange shuffle records written: 144,178 shuffle write time total (min, med, max (stageId: taskId)) 83 ms (0 ms, 0 ms, 42 ms (stage 11.0: task 425)) records read: 706 local bytes read: 127.5 KiB fetch wait time: 0 ms remote bytes read: 117.0 KiB local blocks read: 1 remote blocks read: 1 data size total (min, med, max (stageId: taskId)) 172.0 MiB (0.0 B, 0.0 B, 91.2 MiB (stage 11.0: task 424)) shuffle bytes written total (min, med, max (stageId: taskId)) 49.1 MiB (0.0 B, 0.0 B, 25.8 MiB (stage 11.0: task 424))"]; subgraph cluster7 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n8.1 s (157 ms, 2.0 s, 2.1 s (stage 11.0: task 424))"; 8 [labelType="html" label=" Project "]; 9 [labelType="html" label="Filter number of output rows: 144,178"]; } 10 [labelType="html" label="Scan csv number of files read: 1 metadata time: 0 ms size of files read: 519.0 MiB number of output rows: 500,001"]; subgraph cluster11 { isCluster="true"; label="WholeStageCodegen (4)"; 12 [labelType="html" label="Sort sort time: 7 ms peak memory: 64.1 MiB spill size: 0.0 B"]; } 13 [labelType="html" label="Exchange shuffle records written: 504,708 shuffle write time total (min, med, max (stageId: taskId)) 134 ms (67 ms, 67 ms, 67 ms (stage 12.0: task 428)) records read: 2,385 local bytes read: 241.2 KiB fetch wait time: 0 ms remote bytes read: 212.0 KiB local blocks read: 1 remote blocks read: 1 data size total (min, med, max (stageId: taskId)) 217.8 MiB (104.9 MiB, 112.9 MiB, 112.9 MiB (stage 12.0: task 428)) shuffle bytes written total (min, med, max (stageId: taskId)) 88.8 MiB (42.6 MiB, 46.2 MiB, 46.2 MiB (stage 12.0: task 428))"]; subgraph cluster14 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: total (min, med, max (stageId: taskId))\n3.8 s (1.9 s, 1.9 s, 1.9 s (stage 12.0: task 428))"; 15 [labelType="html" label=" Project "]; 16 [labelType="html" label="Filter number of output rows: 504,708"]; } 17 [labelType="html" label="Scan csv number of files read: 1 metadata time: 0 ms size of files read: 173.6 MiB number of output rows: 504,708"]; 2->0; 3->2; 5->3; 6->5; 8->6; 9->8; 10->9; 12->3; 13->12; 15->13; 16->15; 17->16; }

CollectLimit 6

Project [cast(number#16 as string) AS number#351, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, cast(difficulty#26L as string) AS difficulty#361, cast(total_difficulty#27 as string) AS total_difficulty#362, cast(size#28 as string) AS size#363, extra_data#29, cast(gas_limit#30 as string) AS gas_limit#365, cast(gas_used#31 as string) AS gas_used#366, cast(timestamp#32 as string) AS timestamp#367, cast(transaction_count#33 as string) AS transaction_count#368, base_fee_per_gas#34, formatted_date#136, hash#70, cast(nonce#71 as string) AS nonce#372, block_hash#72, cast(block_number#73 as string) AS block_number#374, ... 11 more fields]

SortMergeJoin [hash#17], [block_hash#72], Inner

WholeStageCodegen (5)

Sort [hash#17 ASC NULLS FIRST], false, 0

WholeStageCodegen (2)

Exchange hashpartitioning(hash#17, 200), true, [id=#179]

Project [number#16, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, difficulty#26L, total_difficulty#27, size#28, extra_data#29, gas_limit#30, gas_used#31, timestamp#32, transaction_count#33, base_fee_per_gas#34, from_unixtime(cast(timestamp#32 as bigint), yyyy-MM-dd, Some(GMT)) AS formatted_date#136]

Filter ((from_unixtime(cast(timestamp#32 as bigint), yyyy-MM, Some(GMT)) = 2015-09) AND isnotnull(hash#17))

WholeStageCodegen (1)

FileScan csv [number#16,hash#17,parent_hash#18,nonce#19,sha3_uncles#20,logs_bloom#21,transactions_root#22,state_root#23,receipts_root#24,miner#25,difficulty#26L,total_difficulty#27,size#28,extra_data#29,gas_limit#30,gas_used#31,timestamp#32,transaction_count#33,base_fee_per_gas#34] Batched: false, DataFilters: [(from_unixtime(cast(timestamp#32 as bigint), yyyy-MM, Some(GMT)) = 2015-09), isnotnull(hash#17)], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/ethereum/blocks.csv], PartitionFilters: [], PushedFilters: [IsNotNull(hash)], ReadSchema: struct<number:int,hash:string,parent_hash:string,nonce:string,sha3_uncles:string,logs_bloom:strin...

Sort [block_hash#72 ASC NULLS FIRST], false, 0

WholeStageCodegen (4)

Exchange hashpartitioning(block_hash#72, 200), true, [id=#188]

Project [hash#70, nonce#71, block_hash#72, block_number#73, transaction_index#74, from_address#75, to_address#76, value#77, gas#78, gas_price#79L, input#80, block_timestamp#81, max_fee_per_gas#82, max_priority_fee_per_gas#83, transaction_type#84]

Filter isnotnull(block_hash#72)

WholeStageCodegen (3)

FileScan csv [hash#70,nonce#71,block_hash#72,block_number#73,transaction_index#74,from_address#75,to_address#76,value#77,gas#78,gas_price#79L,input#80,block_timestamp#81,max_fee_per_gas#82,max_priority_fee_per_gas#83,transaction_type#84] Batched: false, DataFilters: [isnotnull(block_hash#72)], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/ethereum/transactions.csv], PartitionFilters: [], PushedFilters: [IsNotNull(block_hash)], ReadSchema: struct<hash:string,nonce:int,block_hash:string,block_number:int,transaction_index:int,from_addres...

Details

== Parsed Logical Plan ==
GlobalLimit 6
+- LocalLimit 6
   +- Project [cast(number#16 as string) AS number#351, cast(hash#17 as string) AS hash#352, cast(parent_hash#18 as string) AS parent_hash#353, cast(nonce#19 as string) AS nonce#354, cast(sha3_uncles#20 as string) AS sha3_uncles#355, cast(logs_bloom#21 as string) AS logs_bloom#356, cast(transactions_root#22 as string) AS transactions_root#357, cast(state_root#23 as string) AS state_root#358, cast(receipts_root#24 as string) AS receipts_root#359, cast(miner#25 as string) AS miner#360, cast(difficulty#26L as string) AS difficulty#361, cast(total_difficulty#27 as string) AS total_difficulty#362, cast(size#28 as string) AS size#363, cast(extra_data#29 as string) AS extra_data#364, cast(gas_limit#30 as string) AS gas_limit#365, cast(gas_used#31 as string) AS gas_used#366, cast(timestamp#32 as string) AS timestamp#367, cast(transaction_count#33 as string) AS transaction_count#368, cast(base_fee_per_gas#34 as string) AS base_fee_per_gas#369, cast(formatted_date#136 as string) AS formatted_date#370, cast(hash#70 as string) AS hash#371, cast(nonce#71 as string) AS nonce#372, cast(block_hash#72 as string) AS block_hash#373, cast(block_number#73 as string) AS block_number#374, ... 11 more fields]
      +- Filter (from_unixtime(cast(timestamp#32 as bigint), yyyy-MM, Some(GMT)) = 2015-09)
         +- Join Inner, (hash#17 = block_hash#72)
            :- Project [number#16, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, difficulty#26L, total_difficulty#27, size#28, extra_data#29, gas_limit#30, gas_used#31, timestamp#32, transaction_count#33, base_fee_per_gas#34, from_unixtime(cast(timestamp#32 as bigint), yyyy-MM-dd, Some(GMT)) AS formatted_date#136]
            :  +- Relation[number#16,hash#17,parent_hash#18,nonce#19,sha3_uncles#20,logs_bloom#21,transactions_root#22,state_root#23,receipts_root#24,miner#25,difficulty#26L,total_difficulty#27,size#28,extra_data#29,gas_limit#30,gas_used#31,timestamp#32,transaction_count#33,base_fee_per_gas#34] csv
            +- Relation[hash#70,nonce#71,block_hash#72,block_number#73,transaction_index#74,from_address#75,to_address#76,value#77,gas#78,gas_price#79L,input#80,block_timestamp#81,max_fee_per_gas#82,max_priority_fee_per_gas#83,transaction_type#84] csv

== Analyzed Logical Plan ==
number: string, hash: string, parent_hash: string, nonce: string, sha3_uncles: string, logs_bloom: string, transactions_root: string, state_root: string, receipts_root: string, miner: string, difficulty: string, total_difficulty: string, size: string, extra_data: string, gas_limit: string, gas_used: string, timestamp: string, transaction_count: string, base_fee_per_gas: string, formatted_date: string, hash: string, nonce: string, block_hash: string, block_number: string, ... 11 more fields
GlobalLimit 6
+- LocalLimit 6
   +- Project [cast(number#16 as string) AS number#351, cast(hash#17 as string) AS hash#352, cast(parent_hash#18 as string) AS parent_hash#353, cast(nonce#19 as string) AS nonce#354, cast(sha3_uncles#20 as string) AS sha3_uncles#355, cast(logs_bloom#21 as string) AS logs_bloom#356, cast(transactions_root#22 as string) AS transactions_root#357, cast(state_root#23 as string) AS state_root#358, cast(receipts_root#24 as string) AS receipts_root#359, cast(miner#25 as string) AS miner#360, cast(difficulty#26L as string) AS difficulty#361, cast(total_difficulty#27 as string) AS total_difficulty#362, cast(size#28 as string) AS size#363, cast(extra_data#29 as string) AS extra_data#364, cast(gas_limit#30 as string) AS gas_limit#365, cast(gas_used#31 as string) AS gas_used#366, cast(timestamp#32 as string) AS timestamp#367, cast(transaction_count#33 as string) AS transaction_count#368, cast(base_fee_per_gas#34 as string) AS base_fee_per_gas#369, cast(formatted_date#136 as string) AS formatted_date#370, cast(hash#70 as string) AS hash#371, cast(nonce#71 as string) AS nonce#372, cast(block_hash#72 as string) AS block_hash#373, cast(block_number#73 as string) AS block_number#374, ... 11 more fields]
      +- Filter (from_unixtime(cast(timestamp#32 as bigint), yyyy-MM, Some(GMT)) = 2015-09)
         +- Join Inner, (hash#17 = block_hash#72)
            :- Project [number#16, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, difficulty#26L, total_difficulty#27, size#28, extra_data#29, gas_limit#30, gas_used#31, timestamp#32, transaction_count#33, base_fee_per_gas#34, from_unixtime(cast(timestamp#32 as bigint), yyyy-MM-dd, Some(GMT)) AS formatted_date#136]
            :  +- Relation[number#16,hash#17,parent_hash#18,nonce#19,sha3_uncles#20,logs_bloom#21,transactions_root#22,state_root#23,receipts_root#24,miner#25,difficulty#26L,total_difficulty#27,size#28,extra_data#29,gas_limit#30,gas_used#31,timestamp#32,transaction_count#33,base_fee_per_gas#34] csv
            +- Relation[hash#70,nonce#71,block_hash#72,block_number#73,transaction_index#74,from_address#75,to_address#76,value#77,gas#78,gas_price#79L,input#80,block_timestamp#81,max_fee_per_gas#82,max_priority_fee_per_gas#83,transaction_type#84] csv

== Optimized Logical Plan ==
GlobalLimit 6
+- LocalLimit 6
   +- Project [cast(number#16 as string) AS number#351, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, cast(difficulty#26L as string) AS difficulty#361, cast(total_difficulty#27 as string) AS total_difficulty#362, cast(size#28 as string) AS size#363, extra_data#29, cast(gas_limit#30 as string) AS gas_limit#365, cast(gas_used#31 as string) AS gas_used#366, cast(timestamp#32 as string) AS timestamp#367, cast(transaction_count#33 as string) AS transaction_count#368, base_fee_per_gas#34, formatted_date#136, hash#70, cast(nonce#71 as string) AS nonce#372, block_hash#72, cast(block_number#73 as string) AS block_number#374, ... 11 more fields]
      +- Join Inner, (hash#17 = block_hash#72)
         :- Project [number#16, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, difficulty#26L, total_difficulty#27, size#28, extra_data#29, gas_limit#30, gas_used#31, timestamp#32, transaction_count#33, base_fee_per_gas#34, from_unixtime(cast(timestamp#32 as bigint), yyyy-MM-dd, Some(GMT)) AS formatted_date#136]
         :  +- Filter ((from_unixtime(cast(timestamp#32 as bigint), yyyy-MM, Some(GMT)) = 2015-09) AND isnotnull(hash#17))
         :     +- Relation[number#16,hash#17,parent_hash#18,nonce#19,sha3_uncles#20,logs_bloom#21,transactions_root#22,state_root#23,receipts_root#24,miner#25,difficulty#26L,total_difficulty#27,size#28,extra_data#29,gas_limit#30,gas_used#31,timestamp#32,transaction_count#33,base_fee_per_gas#34] csv
         +- Filter isnotnull(block_hash#72)
            +- Relation[hash#70,nonce#71,block_hash#72,block_number#73,transaction_index#74,from_address#75,to_address#76,value#77,gas#78,gas_price#79L,input#80,block_timestamp#81,max_fee_per_gas#82,max_priority_fee_per_gas#83,transaction_type#84] csv

== Physical Plan ==
CollectLimit 6
+- *(5) Project [cast(number#16 as string) AS number#351, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, cast(difficulty#26L as string) AS difficulty#361, cast(total_difficulty#27 as string) AS total_difficulty#362, cast(size#28 as string) AS size#363, extra_data#29, cast(gas_limit#30 as string) AS gas_limit#365, cast(gas_used#31 as string) AS gas_used#366, cast(timestamp#32 as string) AS timestamp#367, cast(transaction_count#33 as string) AS transaction_count#368, base_fee_per_gas#34, formatted_date#136, hash#70, cast(nonce#71 as string) AS nonce#372, block_hash#72, cast(block_number#73 as string) AS block_number#374, ... 11 more fields]
   +- *(5) SortMergeJoin [hash#17], [block_hash#72], Inner
      :- *(2) Sort [hash#17 ASC NULLS FIRST], false, 0
      :  +- Exchange hashpartitioning(hash#17, 200), true, [id=#179]
      :     +- *(1) Project [number#16, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, difficulty#26L, total_difficulty#27, size#28, extra_data#29, gas_limit#30, gas_used#31, timestamp#32, transaction_count#33, base_fee_per_gas#34, from_unixtime(cast(timestamp#32 as bigint), yyyy-MM-dd, Some(GMT)) AS formatted_date#136]
      :        +- *(1) Filter ((from_unixtime(cast(timestamp#32 as bigint), yyyy-MM, Some(GMT)) = 2015-09) AND isnotnull(hash#17))
      :           +- FileScan csv [number#16,hash#17,parent_hash#18,nonce#19,sha3_uncles#20,logs_bloom#21,transactions_root#22,state_root#23,receipts_root#24,miner#25,difficulty#26L,total_difficulty#27,size#28,extra_data#29,gas_limit#30,gas_used#31,timestamp#32,transaction_count#33,base_fee_per_gas#34] Batched: false, DataFilters: [(from_unixtime(cast(timestamp#32 as bigint), yyyy-MM, Some(GMT)) = 2015-09), isnotnull(hash#17)], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/ethereum/blocks.csv], PartitionFilters: [], PushedFilters: [IsNotNull(hash)], ReadSchema: struct<number:int,hash:string,parent_hash:string,nonce:string,sha3_uncles:string,logs_bloom:strin...
      +- *(4) Sort [block_hash#72 ASC NULLS FIRST], false, 0
         +- Exchange hashpartitioning(block_hash#72, 200), true, [id=#188]
            +- *(3) Project [hash#70, nonce#71, block_hash#72, block_number#73, transaction_index#74, from_address#75, to_address#76, value#77, gas#78, gas_price#79L, input#80, block_timestamp#81, max_fee_per_gas#82, max_priority_fee_per_gas#83, transaction_type#84]
               +- *(3) Filter isnotnull(block_hash#72)
                  +- FileScan csv [hash#70,nonce#71,block_hash#72,block_number#73,transaction_index#74,from_address#75,to_address#76,value#77,gas#78,gas_price#79L,input#80,block_timestamp#81,max_fee_per_gas#82,max_priority_fee_per_gas#83,transaction_type#84] Batched: false, DataFilters: [isnotnull(block_hash#72)], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/ethereum/transactions.csv], PartitionFilters: [], PushedFilters: [IsNotNull(block_hash)], ReadSchema: struct<hash:string,nonce:int,block_hash:string,block_number:int,transaction_index:int,from_addres...