task2 - Details for Query 6

Details for Query 6

Submitted Time: 2024/12/17 23:14:28
Duration: 11 s
Succeeded Jobs: 8

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" CollectLimit "]; subgraph cluster1 { isCluster="true"; label="WholeStageCodegen (5)"; 2 [labelType="html" label=" Project "]; 3 [labelType="html" label="SortMergeJoin number of output rows: 11"]; } subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (2)"; 5 [labelType="html" label="Sort sort time: 10 ms peak memory: 64.1 MiB spill size: 0.0 B"]; } 6 [labelType="html" label="Exchange shuffle records written: 504,708 shuffle write time total (min, med, max (stageId: taskId)) 184 ms (88 ms, 95 ms, 95 ms (stage 13.0: task 430)) records read: 2,385 local bytes read: 241.2 KiB fetch wait time: 0 ms remote bytes read: 212.0 KiB local blocks read: 1 remote blocks read: 1 data size total (min, med, max (stageId: taskId)) 217.8 MiB (104.9 MiB, 112.9 MiB, 112.9 MiB (stage 13.0: task 429)) shuffle bytes written total (min, med, max (stageId: taskId)) 88.8 MiB (42.6 MiB, 46.2 MiB, 46.2 MiB (stage 13.0: task 429))"]; subgraph cluster7 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n5.8 s (2.8 s, 3.0 s, 3.0 s (stage 13.0: task 429))"; 8 [labelType="html" label=" Project "]; 9 [labelType="html" label="Filter number of output rows: 504,708"]; } 10 [labelType="html" label="Scan csv number of files read: 1 metadata time: 0 ms size of files read: 173.6 MiB number of output rows: 504,708"]; subgraph cluster11 { isCluster="true"; label="WholeStageCodegen (4)"; 12 [labelType="html" label="Sort sort time: 9 ms peak memory: 64.1 MiB spill size: 0.0 B"]; } 13 [labelType="html" label="Exchange shuffle records written: 500,001 shuffle write time total (min, med, max (stageId: taskId)) 387 ms (24 ms, 84 ms, 104 ms (stage 12.0: task 427)) records read: 2,468 local bytes read: 437.5 KiB fetch wait time: 0 ms remote bytes read: 424.5 KiB local blocks read: 2 remote blocks read: 3 data size total (min, med, max (stageId: taskId)) 599.6 MiB (8.0 MiB, 147.8 MiB, 148.1 MiB (stage 12.0: task 425)) shuffle bytes written total (min, med, max (stageId: taskId)) 170.8 MiB (2.4 MiB, 41.4 MiB, 43.9 MiB (stage 12.0: task 427))"]; subgraph cluster14 { isCluster="true"; label="WholeStageCodegen (3)\n \nduration: total (min, med, max (stageId: taskId))\n14.0 s (244 ms, 3.4 s, 3.7 s (stage 12.0: task 424))"; 15 [labelType="html" label=" Project "]; 16 [labelType="html" label="Filter number of output rows: 500,001"]; } 17 [labelType="html" label="Scan csv number of files read: 1 metadata time: 0 ms size of files read: 519.0 MiB number of output rows: 500,001"]; 2->0; 3->2; 5->3; 6->5; 8->6; 9->8; 10->9; 12->3; 13->12; 15->13; 16->15; 17->16; }

CollectLimit 11

Project [hash#70, cast(nonce#71 as string) AS nonce#452, block_hash#72, cast(block_number#73 as string) AS block_number#454, cast(transaction_index#74 as string) AS transaction_index#455, from_address#75, to_address#76, cast(value#77 as string) AS value#458, cast(gas#78 as string) AS gas#459, cast(gas_price#79L as string) AS gas_price#460, input#80, cast(block_timestamp#81 as string) AS block_timestamp#462, max_fee_per_gas#82, max_priority_fee_per_gas#83, cast(transaction_type#84 as string) AS transaction_type#465, cast(number#16 as string) AS number#466, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, ... 11 more fields]

SortMergeJoin [block_hash#72], [hash#17], Inner

WholeStageCodegen (5)

Sort [block_hash#72 ASC NULLS FIRST], false, 0

WholeStageCodegen (2)

Exchange hashpartitioning(block_hash#72, 200), true, [id=#191]

Project [hash#70, nonce#71, block_hash#72, block_number#73, transaction_index#74, from_address#75, to_address#76, value#77, gas#78, gas_price#79L, input#80, block_timestamp#81, max_fee_per_gas#82, max_priority_fee_per_gas#83, transaction_type#84]

Filter isnotnull(block_hash#72)

WholeStageCodegen (1)

FileScan csv [hash#70,nonce#71,block_hash#72,block_number#73,transaction_index#74,from_address#75,to_address#76,value#77,gas#78,gas_price#79L,input#80,block_timestamp#81,max_fee_per_gas#82,max_priority_fee_per_gas#83,transaction_type#84] Batched: false, DataFilters: [isnotnull(block_hash#72)], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/ethereum/transactions.csv], PartitionFilters: [], PushedFilters: [IsNotNull(block_hash)], ReadSchema: struct<hash:string,nonce:int,block_hash:string,block_number:int,transaction_index:int,from_addres...

Sort [hash#17 ASC NULLS FIRST], false, 0

WholeStageCodegen (4)

Exchange hashpartitioning(hash#17, 200), true, [id=#200]

Project [number#16, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, difficulty#26L, total_difficulty#27, size#28, extra_data#29, gas_limit#30, gas_used#31, timestamp#32, transaction_count#33, base_fee_per_gas#34, from_unixtime(cast(timestamp#32 as bigint), yyyy-MM-dd, Some(GMT)) AS formatted_date#136]

Filter isnotnull(hash#17)

WholeStageCodegen (3)

FileScan csv [number#16,hash#17,parent_hash#18,nonce#19,sha3_uncles#20,logs_bloom#21,transactions_root#22,state_root#23,receipts_root#24,miner#25,difficulty#26L,total_difficulty#27,size#28,extra_data#29,gas_limit#30,gas_used#31,timestamp#32,transaction_count#33,base_fee_per_gas#34] Batched: false, DataFilters: [isnotnull(hash#17)], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/ethereum/blocks.csv], PartitionFilters: [], PushedFilters: [IsNotNull(hash)], ReadSchema: struct<number:int,hash:string,parent_hash:string,nonce:string,sha3_uncles:string,logs_bloom:strin...

Details

== Parsed Logical Plan ==
GlobalLimit 11
+- LocalLimit 11
   +- Project [cast(hash#70 as string) AS hash#451, cast(nonce#71 as string) AS nonce#452, cast(block_hash#72 as string) AS block_hash#453, cast(block_number#73 as string) AS block_number#454, cast(transaction_index#74 as string) AS transaction_index#455, cast(from_address#75 as string) AS from_address#456, cast(to_address#76 as string) AS to_address#457, cast(value#77 as string) AS value#458, cast(gas#78 as string) AS gas#459, cast(gas_price#79L as string) AS gas_price#460, cast(input#80 as string) AS input#461, cast(block_timestamp#81 as string) AS block_timestamp#462, cast(max_fee_per_gas#82 as string) AS max_fee_per_gas#463, cast(max_priority_fee_per_gas#83 as string) AS max_priority_fee_per_gas#464, cast(transaction_type#84 as string) AS transaction_type#465, cast(number#16 as string) AS number#466, cast(hash#17 as string) AS hash#467, cast(parent_hash#18 as string) AS parent_hash#468, cast(nonce#19 as string) AS nonce#469, cast(sha3_uncles#20 as string) AS sha3_uncles#470, cast(logs_bloom#21 as string) AS logs_bloom#471, cast(transactions_root#22 as string) AS transactions_root#472, cast(state_root#23 as string) AS state_root#473, cast(receipts_root#24 as string) AS receipts_root#474, ... 11 more fields]
      +- Join Inner, (block_hash#72 = hash#17)
         :- Relation[hash#70,nonce#71,block_hash#72,block_number#73,transaction_index#74,from_address#75,to_address#76,value#77,gas#78,gas_price#79L,input#80,block_timestamp#81,max_fee_per_gas#82,max_priority_fee_per_gas#83,transaction_type#84] csv
         +- Project [number#16, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, difficulty#26L, total_difficulty#27, size#28, extra_data#29, gas_limit#30, gas_used#31, timestamp#32, transaction_count#33, base_fee_per_gas#34, from_unixtime(cast(timestamp#32 as bigint), yyyy-MM-dd, Some(GMT)) AS formatted_date#136]
            +- Relation[number#16,hash#17,parent_hash#18,nonce#19,sha3_uncles#20,logs_bloom#21,transactions_root#22,state_root#23,receipts_root#24,miner#25,difficulty#26L,total_difficulty#27,size#28,extra_data#29,gas_limit#30,gas_used#31,timestamp#32,transaction_count#33,base_fee_per_gas#34] csv

== Analyzed Logical Plan ==
hash: string, nonce: string, block_hash: string, block_number: string, transaction_index: string, from_address: string, to_address: string, value: string, gas: string, gas_price: string, input: string, block_timestamp: string, max_fee_per_gas: string, max_priority_fee_per_gas: string, transaction_type: string, number: string, hash: string, parent_hash: string, nonce: string, sha3_uncles: string, logs_bloom: string, transactions_root: string, state_root: string, receipts_root: string, ... 11 more fields
GlobalLimit 11
+- LocalLimit 11
   +- Project [cast(hash#70 as string) AS hash#451, cast(nonce#71 as string) AS nonce#452, cast(block_hash#72 as string) AS block_hash#453, cast(block_number#73 as string) AS block_number#454, cast(transaction_index#74 as string) AS transaction_index#455, cast(from_address#75 as string) AS from_address#456, cast(to_address#76 as string) AS to_address#457, cast(value#77 as string) AS value#458, cast(gas#78 as string) AS gas#459, cast(gas_price#79L as string) AS gas_price#460, cast(input#80 as string) AS input#461, cast(block_timestamp#81 as string) AS block_timestamp#462, cast(max_fee_per_gas#82 as string) AS max_fee_per_gas#463, cast(max_priority_fee_per_gas#83 as string) AS max_priority_fee_per_gas#464, cast(transaction_type#84 as string) AS transaction_type#465, cast(number#16 as string) AS number#466, cast(hash#17 as string) AS hash#467, cast(parent_hash#18 as string) AS parent_hash#468, cast(nonce#19 as string) AS nonce#469, cast(sha3_uncles#20 as string) AS sha3_uncles#470, cast(logs_bloom#21 as string) AS logs_bloom#471, cast(transactions_root#22 as string) AS transactions_root#472, cast(state_root#23 as string) AS state_root#473, cast(receipts_root#24 as string) AS receipts_root#474, ... 11 more fields]
      +- Join Inner, (block_hash#72 = hash#17)
         :- Relation[hash#70,nonce#71,block_hash#72,block_number#73,transaction_index#74,from_address#75,to_address#76,value#77,gas#78,gas_price#79L,input#80,block_timestamp#81,max_fee_per_gas#82,max_priority_fee_per_gas#83,transaction_type#84] csv
         +- Project [number#16, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, difficulty#26L, total_difficulty#27, size#28, extra_data#29, gas_limit#30, gas_used#31, timestamp#32, transaction_count#33, base_fee_per_gas#34, from_unixtime(cast(timestamp#32 as bigint), yyyy-MM-dd, Some(GMT)) AS formatted_date#136]
            +- Relation[number#16,hash#17,parent_hash#18,nonce#19,sha3_uncles#20,logs_bloom#21,transactions_root#22,state_root#23,receipts_root#24,miner#25,difficulty#26L,total_difficulty#27,size#28,extra_data#29,gas_limit#30,gas_used#31,timestamp#32,transaction_count#33,base_fee_per_gas#34] csv

== Optimized Logical Plan ==
GlobalLimit 11
+- LocalLimit 11
   +- Project [hash#70, cast(nonce#71 as string) AS nonce#452, block_hash#72, cast(block_number#73 as string) AS block_number#454, cast(transaction_index#74 as string) AS transaction_index#455, from_address#75, to_address#76, cast(value#77 as string) AS value#458, cast(gas#78 as string) AS gas#459, cast(gas_price#79L as string) AS gas_price#460, input#80, cast(block_timestamp#81 as string) AS block_timestamp#462, max_fee_per_gas#82, max_priority_fee_per_gas#83, cast(transaction_type#84 as string) AS transaction_type#465, cast(number#16 as string) AS number#466, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, ... 11 more fields]
      +- Join Inner, (block_hash#72 = hash#17)
         :- Filter isnotnull(block_hash#72)
         :  +- Relation[hash#70,nonce#71,block_hash#72,block_number#73,transaction_index#74,from_address#75,to_address#76,value#77,gas#78,gas_price#79L,input#80,block_timestamp#81,max_fee_per_gas#82,max_priority_fee_per_gas#83,transaction_type#84] csv
         +- Project [number#16, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, difficulty#26L, total_difficulty#27, size#28, extra_data#29, gas_limit#30, gas_used#31, timestamp#32, transaction_count#33, base_fee_per_gas#34, from_unixtime(cast(timestamp#32 as bigint), yyyy-MM-dd, Some(GMT)) AS formatted_date#136]
            +- Filter isnotnull(hash#17)
               +- Relation[number#16,hash#17,parent_hash#18,nonce#19,sha3_uncles#20,logs_bloom#21,transactions_root#22,state_root#23,receipts_root#24,miner#25,difficulty#26L,total_difficulty#27,size#28,extra_data#29,gas_limit#30,gas_used#31,timestamp#32,transaction_count#33,base_fee_per_gas#34] csv

== Physical Plan ==
CollectLimit 11
+- *(5) Project [hash#70, cast(nonce#71 as string) AS nonce#452, block_hash#72, cast(block_number#73 as string) AS block_number#454, cast(transaction_index#74 as string) AS transaction_index#455, from_address#75, to_address#76, cast(value#77 as string) AS value#458, cast(gas#78 as string) AS gas#459, cast(gas_price#79L as string) AS gas_price#460, input#80, cast(block_timestamp#81 as string) AS block_timestamp#462, max_fee_per_gas#82, max_priority_fee_per_gas#83, cast(transaction_type#84 as string) AS transaction_type#465, cast(number#16 as string) AS number#466, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, ... 11 more fields]
   +- *(5) SortMergeJoin [block_hash#72], [hash#17], Inner
      :- *(2) Sort [block_hash#72 ASC NULLS FIRST], false, 0
      :  +- Exchange hashpartitioning(block_hash#72, 200), true, [id=#191]
      :     +- *(1) Project [hash#70, nonce#71, block_hash#72, block_number#73, transaction_index#74, from_address#75, to_address#76, value#77, gas#78, gas_price#79L, input#80, block_timestamp#81, max_fee_per_gas#82, max_priority_fee_per_gas#83, transaction_type#84]
      :        +- *(1) Filter isnotnull(block_hash#72)
      :           +- FileScan csv [hash#70,nonce#71,block_hash#72,block_number#73,transaction_index#74,from_address#75,to_address#76,value#77,gas#78,gas_price#79L,input#80,block_timestamp#81,max_fee_per_gas#82,max_priority_fee_per_gas#83,transaction_type#84] Batched: false, DataFilters: [isnotnull(block_hash#72)], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/ethereum/transactions.csv], PartitionFilters: [], PushedFilters: [IsNotNull(block_hash)], ReadSchema: struct<hash:string,nonce:int,block_hash:string,block_number:int,transaction_index:int,from_addres...
      +- *(4) Sort [hash#17 ASC NULLS FIRST], false, 0
         +- Exchange hashpartitioning(hash#17, 200), true, [id=#200]
            +- *(3) Project [number#16, hash#17, parent_hash#18, nonce#19, sha3_uncles#20, logs_bloom#21, transactions_root#22, state_root#23, receipts_root#24, miner#25, difficulty#26L, total_difficulty#27, size#28, extra_data#29, gas_limit#30, gas_used#31, timestamp#32, transaction_count#33, base_fee_per_gas#34, from_unixtime(cast(timestamp#32 as bigint), yyyy-MM-dd, Some(GMT)) AS formatted_date#136]
               +- *(3) Filter isnotnull(hash#17)
                  +- FileScan csv [number#16,hash#17,parent_hash#18,nonce#19,sha3_uncles#20,logs_bloom#21,transactions_root#22,state_root#23,receipts_root#24,miner#25,difficulty#26L,total_difficulty#27,size#28,extra_data#29,gas_limit#30,gas_used#31,timestamp#32,transaction_count#33,base_fee_per_gas#34] Batched: false, DataFilters: [isnotnull(hash#17)], Format: CSV, Location: InMemoryFileIndex[s3a://data-repository-bkt/ECS765/ethereum/blocks.csv], PartitionFilters: [], PushedFilters: [IsNotNull(hash)], ReadSchema: struct<number:int,hash:string,parent_hash:string,nonce:string,sha3_uncles:string,logs_bloom:strin...