Climate metrics jobs

Queued

5,788

oldest waiting 58 min

Running

on 10 of 10 worker pods

Longest wait now

58 min

oldest queued job

Succeeded, last hour

2,841

jobs finished

Failure rate, 24 h

8.1 %

1,281 failed of 15.8K

Run time p95, 24 h

9.6 s

median 4.7 s

2 Oct 16:45 – now · 3,282 succeeded, 1,260 failed, run time p95 9.8 s, longest wait 57 min

Queue

peak per 15 min

Queued
Running

Table

Time	Queued	Running
2 Oct 21:30	9,990	39
2 Oct 21:45	9,369	66
2 Oct 22:00	8,292	58
2 Oct 22:15	7,169	55
2 Oct 22:30	5,864	53

Worker pods

peak per 15 min

Ready
Busy
Starting
Waiting for a node

Table

Time	Ready	Busy	Starting
2 Oct 21:15	1	0	0
2 Oct 21:30	10	10	7
2 Oct 21:45	10	10	0
2 Oct 22:00	10	10	0
2 Oct 22:15	10	10	0
2 Oct 22:30	10	9	0

Throughput

jobs per 15 min

Succeeded
Failed
Cancelled
Submitted

Table

Time	Succeeded	Failed	Submitted
2 Oct 17:15	300	0	300
2 Oct 18:15	50	0	50
2 Oct 19:45	1	0	1
2 Oct 20:15	102	0	108
2 Oct 20:30	6	0	0
2 Oct 21:30	605	1	10K
2 Oct 21:45	1,029	14	0
2 Oct 22:00	808	320	0
2 Oct 22:15	381	924	0
2 Oct 22:30	0	1	0

Run time

succeeded jobs, started to finished

Median
95th percentile
Longest

Table

Time	Median	95th percentile	Longest
2 Oct 17:15	6.1 s	9.8 s	55 s
2 Oct 18:15	4.3 s	9.8 s	43 s
2 Oct 19:45	5.8 s	5.8 s	5.8 s
2 Oct 20:15	6.3 s	9.8 s	56 s
2 Oct 20:30	3.8 s	5.1 s	5.1 s
2 Oct 21:30	6.0 s	9.8 s	57 s
2 Oct 21:45	6.5 s	9.8 s	59 s
2 Oct 22:00	6.3 s	9.9 s	1 min 1 s
2 Oct 22:15	4.3 s	9.8 s	57 s

Wait time

created to started

95th percentile (started)
Longest (started)
Oldest still queued

Table

Time	95th percentile (started)	Longest (started)	Oldest still queued
2 Oct 17:15	9 min 40 s	10 min	–
2 Oct 18:15	2 min 40 s	2 min 40 s	–
2 Oct 19:45	0.4 s	0.4 s	–
2 Oct 20:15	3 min 25 s	3 min 25 s	–
2 Oct 20:30	6 min 6 s	6 min 6 s	–
2 Oct 21:30	12 min	12 min	11 min
2 Oct 21:45	27 min	27 min	26 min
2 Oct 22:00	42 min	42 min	41 min
2 Oct 22:15	57 min	57 min	56 min
2 Oct 22:30	57 min	57 min	57 min

Failures

by kind, per 15 min

Details →

compute_error
invalid_asset
worker_lost
internal

Table

Time	compute_error	invalid_asset	worker_lost	internal
2 Oct 21:30	1	0	0	0
2 Oct 21:45	14	0	0	0
2 Oct 22:00	320	0	0	0
2 Oct 22:15	924	0	0	0
2 Oct 22:30	1	0	0	0

Next in the queue

first in, first out

Asset	Submitted	Waiting
ESP_00008	2 Oct 21:33:17	58 min
ESP_00009	2 Oct 21:33:17	58 min
FRA_00005	2 Oct 21:33:17	58 min
FRA_00015	2 Oct 21:33:17	58 min
FRA_00017	2 Oct 21:33:17	58 min
FRA_00020	2 Oct 21:33:17	58 min
FRA_00028	2 Oct 21:33:17	58 min
FRA_00029	2 Oct 21:33:17	58 min
GBR_00027	2 Oct 21:33:17	58 min
ITA_00005	2 Oct 21:33:17	58 min

and 5,778 more

Worker pods now

10 ready, 0 starting, 0 waiting for a node

Pod	State	Job	Age
climate-metrics-worker-standby-57c4499f86-g7vrh standby · ip-10-40-15-254	ready	CAN_00003	2 h 37 min
climate-metrics-worker-7b6cddc9fc-2wn4r pool · ip-10-40-20-253	ready	JOR_00001	57 min
climate-metrics-worker-7b6cddc9fc-6sbnv pool · ip-10-40-20-253	ready	BRA_00003	57 min
climate-metrics-worker-7b6cddc9fc-7rvrq pool · ip-10-40-20-253	ready	PSE_00001	57 min
climate-metrics-worker-7b6cddc9fc-8fsxj pool · ip-10-40-20-253	ready	PSE_00001	57 min
climate-metrics-worker-7b6cddc9fc-fd7hr pool · ip-10-40-20-253	ready	CPV_00001	57 min
climate-metrics-worker-7b6cddc9fc-v2mxp pool · ip-10-40-20-253	ready	ESP_00010	57 min
climate-metrics-worker-7b6cddc9fc-w7kzj pool · ip-10-40-20-253	ready	CAN_00007	57 min
climate-metrics-worker-7b6cddc9fc-x5zvn pool · ip-10-40-93-92	ready	ESP_00001	54 min
climate-metrics-worker-7b6cddc9fc-zxdzw pool · ip-10-40-93-92	ready	IND_00002	54 min

Running

47 jobs

Asset	Worker	Step	Progress	Waited	Running for	Attempt
BRA_00001	climate-metrics-worker-7b6cddc9fc-6sbnv	PHYSICAL	2/6	56 min	2 min 3 s	1
NLD_00003	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	56 min	1 min 57 s	1
SWE_00001	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	56 min	1 min 54 s	1
BEL_00025	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	56 min	1 min 51 s	1
ESP_00003	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	56 min	1 min 48 s	1
GBR_00013	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	56 min	1 min 45 s	1
HUN_00001	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	56 min	1 min 40 s	1
AUS_00003	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	56 min	1 min 36 s	1
CHL_00002	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	56 min	1 min 33 s	1
DEU_00009	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	56 min	1 min 29 s	1
FRA_00026	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	56 min	1 min 27 s	1
MLT_00001	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	56 min	1 min 24 s	1
PRT_00001	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	56 min	1 min 20 s	1
TWN_00002	climate-metrics-worker-7b6cddc9fc-x5zvn	starting	0/6	56 min	1 min 19 s	1
USA_00086	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	56 min	1 min 17 s	2
AUS_00004	climate-metrics-worker-7b6cddc9fc-x5zvn	starting	0/6	56 min	1 min 16 s	1
PSE_00001	climate-metrics-worker-7b6cddc9fc-8fsxj	starting	0/6	56 min	1 min 15 s	1
NOR_00001	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	56 min	1 min 14 s	1
ESP_00009	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	56 min	1 min 12 s	1
SWE_00001	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	57 min	1 min 9 s	2
FRA_00020	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	57 min	1 min 6 s	1
SGP_00028	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	57 min	1 min 4 s	1
FRA_00030	climate-metrics-worker-7b6cddc9fc-2wn4r	starting	0/6	57 min	1 min 1 s	1
BRA_00003	climate-metrics-worker-7b6cddc9fc-6sbnv	PHYSICAL	2/6	57 min	60 s	1
GUF_00001	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	57 min	59 s	1
USA_00014	climate-metrics-worker-7b6cddc9fc-x5zvn	PHYSICAL	2/6	57 min	58 s	1
BRA_00005	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	57 min	54 s	1
JPN_00002	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	57 min	51 s	1
PAN_00001	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	57 min	48 s	1
VUT_00001	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	57 min	45 s	1
FRA_00009	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	57 min	39 s	1
HUN_00001	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	57 min	34 s	1
AUS_00002	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	57 min	30 s	1
BRA_00002	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	57 min	26 s	1
COL_00001	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	57 min	23 s	1
GBR_00025	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	57 min	18 s	1
PRT_00001	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	57 min	12 s	1
USA_00005	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	57 min	8.5 s	2
PSE_00001	climate-metrics-worker-7b6cddc9fc-7rvrq	starting	0/6	58 min	6.3 s	1
VUT_00001	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	58 min	5.9 s	2
CPV_00001	climate-metrics-worker-7b6cddc9fc-fd7hr	starting	0/6	58 min	4.0 s	2
JOR_00001	climate-metrics-worker-7b6cddc9fc-2wn4r	starting	0/6	58 min	2.9 s	1
CAN_00007	climate-metrics-worker-7b6cddc9fc-w7kzj	starting	0/6	58 min	2.8 s	1
IND_00002	climate-metrics-worker-7b6cddc9fc-zxdzw	starting	0/6	58 min	2.4 s	1
CAN_00003	climate-metrics-worker-standby-57c4499f86-g7vrh	starting	0/6	58 min	1.8 s	1
ESP_00001	climate-metrics-worker-7b6cddc9fc-x5zvn	starting	0/6	58 min	1.6 s	1
ESP_00010	climate-metrics-worker-7b6cddc9fc-v2mxp	starting	0/6	58 min	1.3 s	3