3.0.1

task2 application UI

Details for Query 7

Show the Stage ID and Task ID that corresponds to the max metric
WholeStageCodegen (7) duration: 1 msWholeStageCodegen (6) duration: total (min, med, max )211 ms (0 ms, 0 ms, 7 ms )WholeStageCodegen (5) duration: total (min, med, max )1.2 s (4 ms, 6 ms, 31 ms )WholeStageCodegen (2) duration: total (min, med, max )511 ms (0 ms, 3 ms, 55 ms )WholeStageCodegen (1) duration: total (min, med, max )7.0 s (133 ms, 1.6 s, 2.1 s )WholeStageCodegen (4) duration: total (min, med, max )470 ms (0 ms, 2 ms, 52 ms )WholeStageCodegen (3) duration: total (min, med, max )2.4 s (1.2 s, 1.3 s, 1.3 s )
Execute InsertIntoHadoopFsRelationCommand

number of written files: 1
written output: 1038.0 B
number of output rows: 31
number of dynamic part: 0

Coalesce

Sort

sort time: 0 ms
peak memory: 1985.9 MiB
spill size: 0.0 B
Exchange

shuffle records written: 31
shuffle write time total (min, med, max )
14 ms (0 ms, 0 ms, 0 ms )
records read: 31
local bytes read: 1427.0 B
fetch wait time: 0 ms
remote bytes read: 1176.0 B
local blocks read: 17
remote blocks read: 14
data size total (min, med, max )
1240.0 B (0.0 B, 0.0 B, 120.0 B )
shuffle bytes written total (min, med, max )
2.5 KiB (0.0 B, 0.0 B, 252.0 B )
HashAggregate

time in aggregation build total (min, med, max )
147 ms (0 ms, 0 ms, 5 ms )
peak memory total (min, med, max )
3.6 GiB (256.0 KiB, 256.0 KiB, 64.3 MiB )
number of output rows: 62
avg hash probe bucket list iters (min, med, max ):
(1, 1, 1 )
Exchange

shuffle records written: 6,200
shuffle write time total (min, med, max )
703 ms (3 ms, 3 ms, 8 ms )
records read: 12,400
local bytes read total (min, med, max )
467.1 KiB (0.0 B, 0.0 B, 13.0 KiB )
fetch wait time total (min, med, max )
0 ms (0 ms, 0 ms, 0 ms )
remote bytes read total (min, med, max )
467.8 KiB (0.0 B, 0.0 B, 11.1 KiB )
local blocks read: 5,584
remote blocks read: 5,616
data size total (min, med, max )
242.2 KiB (1240.0 B, 1240.0 B, 1240.0 B )
shuffle bytes written total (min, med, max )
467.5 KiB (2.3 KiB, 2.3 KiB, 2.3 KiB )
HashAggregate

time in aggregation build total (min, med, max )
227 ms (1 ms, 1 ms, 24 ms )
peak memory total (min, med, max )
50.0 MiB (256.0 KiB, 256.0 KiB, 256.0 KiB )
number of output rows: 6,200

Project

SortMergeJoin

number of output rows: 72,118
Sort

sort time total (min, med, max )
0 ms (0 ms, 0 ms, 0 ms )
peak memory total (min, med, max )
12.5 GiB (64.1 MiB, 64.1 MiB, 64.1 MiB )
spill size total (min, med, max )
0.0 B (0.0 B, 0.0 B, 0.0 B )
Exchange

shuffle records written: 156,095
shuffle write time total (min, med, max )
28 ms (0 ms, 0 ms, 15 ms )
records read: 156,095
local bytes read total (min, med, max )
5.1 MiB (15.9 KiB, 30.8 KiB, 36.6 KiB )
fetch wait time total (min, med, max )
0 ms (0 ms, 0 ms, 0 ms )
remote bytes read total (min, med, max )
4.9 MiB (16.2 KiB, 19.9 KiB, 37.5 KiB )
local blocks read: 200
remote blocks read: 200
data size total (min, med, max )
16.7 MiB (0.0 B, 0.0 B, 10.7 MiB )
shuffle bytes written total (min, med, max )
10.0 MiB (0.0 B, 0.0 B, 6.4 MiB )

Project

Filter

number of output rows: 156,095
Scan csv

number of files read: 1
metadata time: 0 ms
size of files read: 519.0 MiB
number of output rows: 500,001
Sort

sort time total (min, med, max )
0 ms (0 ms, 0 ms, 0 ms )
peak memory total (min, med, max )
12.5 GiB (64.1 MiB, 64.1 MiB, 64.1 MiB )
spill size total (min, med, max )
0.0 B (0.0 B, 0.0 B, 0.0 B )
Exchange

shuffle records written: 171,282
shuffle write time total (min, med, max )
30 ms (15 ms, 15 ms, 15 ms )
records read: 171,282
local bytes read total (min, med, max )
5.9 MiB (25.5 KiB, 29.7 KiB, 36.3 KiB )
fetch wait time total (min, med, max )
0 ms (0 ms, 0 ms, 0 ms )
remote bytes read total (min, med, max )
5.9 MiB (24.8 KiB, 30.2 KiB, 36.4 KiB )
local blocks read: 200
remote blocks read: 200
data size total (min, med, max )
20.9 MiB (10.1 MiB, 10.8 MiB, 10.8 MiB )
shuffle bytes written total (min, med, max )
11.8 MiB (5.6 MiB, 6.2 MiB, 6.2 MiB )

Project

Filter

number of output rows: 171,282
Scan csv

number of files read: 1
metadata time: 0 ms
size of files read: 173.6 MiB
number of output rows: 504,708
Details