task2 - Details for Query 7

Details for Query 7

Submitted Time: 2024/11/28 21:55:30
Duration: 7 s
Succeeded Jobs: 12

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" CollectLimit "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (5)"; 2 [labelType="html" label=" Project "]; 3 [labelType="html" label="SortMergeJoin number of output rows: 6"]; } subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (2)"; 5 [labelType="html" label="Sort sort time: 0 ms peak memory: 64.1 MiB spill size: 0.0 B"]; } 6 [labelType="html" label="Exchange shuffle records written: 156,095 shuffle write time total (min, med, max (stageId: taskId)) 86 ms (0 ms, 0 ms, 56 ms (stage 34.0: task 941)) records read: 785 local bytes read: 190.2 KiB fetch wait time: 0 ms remote bytes read: 95.4 KiB local blocks read: 1 remote blocks read: 1 data size total (min, med, max (stageId: taskId)) 187.4 MiB (0.0 B, 0.0 B, 120.4 MiB (stage 34.0: task 941)) shuffle bytes written total (min, med, max (stageId: taskId)) 55.4 MiB (0.0 B, 0.0 B, 35.8 MiB (stage 34.0: task 941))"]; subgraph cluster7 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n8.4 s (176 ms, 1.9 s, 2.5 s (stage 34.0: task 941))"; 8 [labelType="html" label=" Project "]; 9 [labelType="html" label="Filter number of output rows: 156,095"]; } 10 [labelType="html" label="Scan csv number of files read: 1 metadata time: 0 ms size of files read: 519.0 MiB number of output rows: 500,001"]; subgraph cluster11 { isCluster="true"; label="WholeStageCodegen (4)"; 12 [labelType="html" label="Sort sort time: 0 ms peak memory: 64.1 MiB spill size: 0.0 B"]; } 13 [labelType="html" label="Exchange shuffle records written: 504,708 shuffle write time total (min, med, max (stageId: taskId)) 146 ms (64 ms, 81 ms, 81 ms (stage 35.0: task 944)) records read: 2,385 local bytes read: 241.2 KiB fetch wait time: 0 ms remote bytes read: 212.0 KiB local blocks read: 1 remote blocks read: 1 data size total (min, med, max (stageId: taskId)) 217.8 MiB (104.9 MiB, 112.9 MiB, 112.9 MiB (stage 35.0: task 943)) shuffle bytes written total (min, med, max (stageId: taskId)) 88.8 MiB (42.6 MiB, 46.2 MiB, 46.2 MiB (stage 35.0: task 943))"]; subgraph cluster14 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: total (min, med, max (stageId: taskId))\n3.6 s (1.8 s, 1.8 s, 1.8 s (stage 35.0: task 943))"; 15 [labelType="html" label=" Project "]; 16 [labelType="html" label="Filter number of output rows: 504,708"]; } 17 [labelType="html" label="Scan csv number of files read: 1 metadata time: 0 ms size of files read: 173.6 MiB number of output rows: 504,708"]; 2->0; 3->2; 5->3; 6->5; 8->6; 9->8; 10->9; 12->3; 13->12; 15->13; 16->15; 17->16; }

CollectLimit 6

Project [cast(number#16 as string) AS number#592, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, cast(difficulty#26L as string) AS difficulty#602, cast(total_difficulty#27 as string) AS total_difficulty#603, cast(size#28 as string) AS size#604, extra_data#29, cast(gas_limit#30 as string) AS gas_limit#606, cast(gas_used#31 as string) AS gas_used#607, cast(timestamp#32 as string) AS timestamp#608, cast(transaction_count#33 as string) AS transaction_count#609, base_fee_per_gas#34, formatted_date#136, hash#70, cast(nonce#71 as string) AS nonce#613, block_hash#72, cast(block_number#73 as string) AS block_number#615, ... 11 more fields]

SortMergeJoin [hash#17], [block_hash#72], Inner

WholeStageCodegen (5)

Sort [hash#17 ASC NULLS FIRST], false, 0

WholeStageCodegen (2)

Exchange hashpartitioning(hash#17, 200), true, [id=#322]

Project [number#16, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, difficulty#26L, total_difficulty#27, size#28, extra_data#29, gas_limit#30, gas_used#31, timestamp#32, transaction_count#33, base_fee_per_gas#34, from_unixtime(cast(timestamp#32 as bigint), yyyy-MM-dd, Some(GMT)) AS formatted_date#136]

Filter ((from_unixtime(cast(timestamp#32 as bigint), yyyy-MM, Some(GMT)) = 2015-10) AND isnotnull(hash#17))

WholeStageCodegen (1)

FileScan csv [number#16,hash#17,parent_hash#18,nonce#19,sha3_uncles#20,logs_bloom#21,transactions_root#22,state_root#23,receipts_root#24,miner#25,difficulty#26L,total_difficulty#27,size#28,extra_data#29,gas_limit#30,gas_used#31,timestamp#32,transaction_count#33,base_fee_per_gas#34] Batched: false, DataFilters: [(from_unixtime(cast(timestamp#32 as bigint), yyyy-MM, Some(GMT)) = 2015-10), isnotnull(hash#17)], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/ethereum/blocks.csv], PartitionFilters: [], PushedFilters: [IsNotNull(hash)], ReadSchema: struct<number:int,hash:string,parent_hash:string,nonce:string,sha3_uncles:string,logs_bloom:strin...

Sort [block_hash#72 ASC NULLS FIRST], false, 0

WholeStageCodegen (4)

Exchange hashpartitioning(block_hash#72, 200), true, [id=#331]

Project [hash#70, nonce#71, block_hash#72, block_number#73, transaction_index#74, from_address#75, to_address#76, value#77, gas#78, gas_price#79L, input#80, block_timestamp#81, max_fee_per_gas#82, max_priority_fee_per_gas#83, transaction_type#84]

Filter isnotnull(block_hash#72)

WholeStageCodegen (3)

FileScan csv [hash#70,nonce#71,block_hash#72,block_number#73,transaction_index#74,from_address#75,to_address#76,value#77,gas#78,gas_price#79L,input#80,block_timestamp#81,max_fee_per_gas#82,max_priority_fee_per_gas#83,transaction_type#84] Batched: false, DataFilters: [isnotnull(block_hash#72)], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/ethereum/transactions.csv], PartitionFilters: [], PushedFilters: [IsNotNull(block_hash)], ReadSchema: struct<hash:string,nonce:int,block_hash:string,block_number:int,transaction_index:int,from_addres...

Details

== Parsed Logical Plan ==
GlobalLimit 6
+- LocalLimit 6
   +- Project [cast(number#16 as string) AS number#592, cast(hash#17 as string) AS hash#593, cast(parent_hash#18 as string) AS parent_hash#594, cast(nonce#19 as string) AS nonce#595, cast(sha3_uncles#20 as string) AS sha3_uncles#596, cast(logs_bloom#21 as string) AS logs_bloom#597, cast(transactions_root#22 as string) AS transactions_root#598, cast(state_root#23 as string) AS state_root#599, cast(receipts_root#24 as string) AS receipts_root#600, cast(miner#25 as string) AS miner#601, cast(difficulty#26L as string) AS difficulty#602, cast(total_difficulty#27 as string) AS total_difficulty#603, cast(size#28 as string) AS size#604, cast(extra_data#29 as string) AS extra_data#605, cast(gas_limit#30 as string) AS gas_limit#606, cast(gas_used#31 as string) AS gas_used#607, cast(timestamp#32 as string) AS timestamp#608, cast(transaction_count#33 as string) AS transaction_count#609, cast(base_fee_per_gas#34 as string) AS base_fee_per_gas#610, cast(formatted_date#136 as string) AS formatted_date#611, cast(hash#70 as string) AS hash#612, cast(nonce#71 as string) AS nonce#613, cast(block_hash#72 as string) AS block_hash#614, cast(block_number#73 as string) AS block_number#615, ... 11 more fields]
      +- Filter (from_unixtime(cast(timestamp#32 as bigint), yyyy-MM, Some(GMT)) = 2015-10)
         +- Join Inner, (hash#17 = block_hash#72)
            :- Project [number#16, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, difficulty#26L, total_difficulty#27, size#28, extra_data#29, gas_limit#30, gas_used#31, timestamp#32, transaction_count#33, base_fee_per_gas#34, from_unixtime(cast(timestamp#32 as bigint), yyyy-MM-dd, Some(GMT)) AS formatted_date#136]
            :  +- Relation[number#16,hash#17,parent_hash#18,nonce#19,sha3_uncles#20,logs_bloom#21,transactions_root#22,state_root#23,receipts_root#24,miner#25,difficulty#26L,total_difficulty#27,size#28,extra_data#29,gas_limit#30,gas_used#31,timestamp#32,transaction_count#33,base_fee_per_gas#34] csv
            +- Relation[hash#70,nonce#71,block_hash#72,block_number#73,transaction_index#74,from_address#75,to_address#76,value#77,gas#78,gas_price#79L,input#80,block_timestamp#81,max_fee_per_gas#82,max_priority_fee_per_gas#83,transaction_type#84] csv

== Analyzed Logical Plan ==
number: string, hash: string, parent_hash: string, nonce: string, sha3_uncles: string, logs_bloom: string, transactions_root: string, state_root: string, receipts_root: string, miner: string, difficulty: string, total_difficulty: string, size: string, extra_data: string, gas_limit: string, gas_used: string, timestamp: string, transaction_count: string, base_fee_per_gas: string, formatted_date: string, hash: string, nonce: string, block_hash: string, block_number: string, ... 11 more fields
GlobalLimit 6
+- LocalLimit 6
   +- Project [cast(number#16 as string) AS number#592, cast(hash#17 as string) AS hash#593, cast(parent_hash#18 as string) AS parent_hash#594, cast(nonce#19 as string) AS nonce#595, cast(sha3_uncles#20 as string) AS sha3_uncles#596, cast(logs_bloom#21 as string) AS logs_bloom#597, cast(transactions_root#22 as string) AS transactions_root#598, cast(state_root#23 as string) AS state_root#599, cast(receipts_root#24 as string) AS receipts_root#600, cast(miner#25 as string) AS miner#601, cast(difficulty#26L as string) AS difficulty#602, cast(total_difficulty#27 as string) AS total_difficulty#603, cast(size#28 as string) AS size#604, cast(extra_data#29 as string) AS extra_data#605, cast(gas_limit#30 as string) AS gas_limit#606, cast(gas_used#31 as string) AS gas_used#607, cast(timestamp#32 as string) AS timestamp#608, cast(transaction_count#33 as string) AS transaction_count#609, cast(base_fee_per_gas#34 as string) AS base_fee_per_gas#610, cast(formatted_date#136 as string) AS formatted_date#611, cast(hash#70 as string) AS hash#612, cast(nonce#71 as string) AS nonce#613, cast(block_hash#72 as string) AS block_hash#614, cast(block_number#73 as string) AS block_number#615, ... 11 more fields]
      +- Filter (from_unixtime(cast(timestamp#32 as bigint), yyyy-MM, Some(GMT)) = 2015-10)
         +- Join Inner, (hash#17 = block_hash#72)
            :- Project [number#16, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, difficulty#26L, total_difficulty#27, size#28, extra_data#29, gas_limit#30, gas_used#31, timestamp#32, transaction_count#33, base_fee_per_gas#34, from_unixtime(cast(timestamp#32 as bigint), yyyy-MM-dd, Some(GMT)) AS formatted_date#136]
            :  +- Relation[number#16,hash#17,parent_hash#18,nonce#19,sha3_uncles#20,logs_bloom#21,transactions_root#22,state_root#23,receipts_root#24,miner#25,difficulty#26L,total_difficulty#27,size#28,extra_data#29,gas_limit#30,gas_used#31,timestamp#32,transaction_count#33,base_fee_per_gas#34] csv
            +- Relation[hash#70,nonce#71,block_hash#72,block_number#73,transaction_index#74,from_address#75,to_address#76,value#77,gas#78,gas_price#79L,input#80,block_timestamp#81,max_fee_per_gas#82,max_priority_fee_per_gas#83,transaction_type#84] csv

== Optimized Logical Plan ==
GlobalLimit 6
+- LocalLimit 6
   +- Project [cast(number#16 as string) AS number#592, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, cast(difficulty#26L as string) AS difficulty#602, cast(total_difficulty#27 as string) AS total_difficulty#603, cast(size#28 as string) AS size#604, extra_data#29, cast(gas_limit#30 as string) AS gas_limit#606, cast(gas_used#31 as string) AS gas_used#607, cast(timestamp#32 as string) AS timestamp#608, cast(transaction_count#33 as string) AS transaction_count#609, base_fee_per_gas#34, formatted_date#136, hash#70, cast(nonce#71 as string) AS nonce#613, block_hash#72, cast(block_number#73 as string) AS block_number#615, ... 11 more fields]
      +- Join Inner, (hash#17 = block_hash#72)
         :- Project [number#16, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, difficulty#26L, total_difficulty#27, size#28, extra_data#29, gas_limit#30, gas_used#31, timestamp#32, transaction_count#33, base_fee_per_gas#34, from_unixtime(cast(timestamp#32 as bigint), yyyy-MM-dd, Some(GMT)) AS formatted_date#136]
         :  +- Filter ((from_unixtime(cast(timestamp#32 as bigint), yyyy-MM, Some(GMT)) = 2015-10) AND isnotnull(hash#17))
         :     +- Relation[number#16,hash#17,parent_hash#18,nonce#19,sha3_uncles#20,logs_bloom#21,transactions_root#22,state_root#23,receipts_root#24,miner#25,difficulty#26L,total_difficulty#27,size#28,extra_data#29,gas_limit#30,gas_used#31,timestamp#32,transaction_count#33,base_fee_per_gas#34] csv
         +- Filter isnotnull(block_hash#72)
            +- Relation[hash#70,nonce#71,block_hash#72,block_number#73,transaction_index#74,from_address#75,to_address#76,value#77,gas#78,gas_price#79L,input#80,block_timestamp#81,max_fee_per_gas#82,max_priority_fee_per_gas#83,transaction_type#84] csv

== Physical Plan ==
CollectLimit 6
+- *(5) Project [cast(number#16 as string) AS number#592, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, cast(difficulty#26L as string) AS difficulty#602, cast(total_difficulty#27 as string) AS total_difficulty#603, cast(size#28 as string) AS size#604, extra_data#29, cast(gas_limit#30 as string) AS gas_limit#606, cast(gas_used#31 as string) AS gas_used#607, cast(timestamp#32 as string) AS timestamp#608, cast(transaction_count#33 as string) AS transaction_count#609, base_fee_per_gas#34, formatted_date#136, hash#70, cast(nonce#71 as string) AS nonce#613, block_hash#72, cast(block_number#73 as string) AS block_number#615, ... 11 more fields]
   +- *(5) SortMergeJoin [hash#17], [block_hash#72], Inner
      :- *(2) Sort [hash#17 ASC NULLS FIRST], false, 0
      :  +- Exchange hashpartitioning(hash#17, 200), true, [id=#322]
      :     +- *(1) Project [number#16, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, difficulty#26L, total_difficulty#27, size#28, extra_data#29, gas_limit#30, gas_used#31, timestamp#32, transaction_count#33, base_fee_per_gas#34, from_unixtime(cast(timestamp#32 as bigint), yyyy-MM-dd, Some(GMT)) AS formatted_date#136]
      :        +- *(1) Filter ((from_unixtime(cast(timestamp#32 as bigint), yyyy-MM, Some(GMT)) = 2015-10) AND isnotnull(hash#17))
      :           +- FileScan csv [number#16,hash#17,parent_hash#18,nonce#19,sha3_uncles#20,logs_bloom#21,transactions_root#22,state_root#23,receipts_root#24,miner#25,difficulty#26L,total_difficulty#27,size#28,extra_data#29,gas_limit#30,gas_used#31,timestamp#32,transaction_count#33,base_fee_per_gas#34] Batched: false, DataFilters: [(from_unixtime(cast(timestamp#32 as bigint), yyyy-MM, Some(GMT)) = 2015-10), isnotnull(hash#17)], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/ethereum/blocks.csv], PartitionFilters: [], PushedFilters: [IsNotNull(hash)], ReadSchema: struct<number:int,hash:string,parent_hash:string,nonce:string,sha3_uncles:string,logs_bloom:strin...
      +- *(4) Sort [block_hash#72 ASC NULLS FIRST], false, 0
         +- Exchange hashpartitioning(block_hash#72, 200), true, [id=#331]
            +- *(3) Project [hash#70, nonce#71, block_hash#72, block_number#73, transaction_index#74, from_address#75, to_address#76, value#77, gas#78, gas_price#79L, input#80, block_timestamp#81, max_fee_per_gas#82, max_priority_fee_per_gas#83, transaction_type#84]
               +- *(3) Filter isnotnull(block_hash#72)
                  +- FileScan csv [hash#70,nonce#71,block_hash#72,block_number#73,transaction_index#74,from_address#75,to_address#76,value#77,gas#78,gas_price#79L,input#80,block_timestamp#81,max_fee_per_gas#82,max_priority_fee_per_gas#83,transaction_type#84] Batched: false, DataFilters: [isnotnull(block_hash#72)], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/ethereum/transactions.csv], PartitionFilters: [], PushedFilters: [IsNotNull(block_hash)], ReadSchema: struct<hash:string,nonce:int,block_hash:string,block_number:int,transaction_index:int,from_addres...