bigscience-bloom-7b1
Find this model in the BLOOM model summary
bigscience-bloom-7b1 Model Summary Plots
bigscience-bloom-7b1 Model Selected Details
layer_type
N
M
Q
alpha
D
alpha-hat
log_SN
% Rand
num_traps
num_fingers
rank_loss
layer_id
4
DENSE
12288
4096
3.00
3.07
0.02
6.29
2.05
77.85
0
0
0
5
DENSE
4096
4096
1.00
3.11
0.02
6.23
2.00
82.04
4
0
6
7
DENSE
16384
4096
4.00
2.37
0.05
6.06
2.56
87.05
1
0
0
8
DENSE
16384
4096
4.00
5.91
0.05
14.97
2.53
87.80
5
0
0
10
DENSE
12288
4096
3.00
3.51
0.03
7.16
2.04
83.05
0
0
0
11
DENSE
4096
4096
1.00
3.51
0.03
7.73
2.20
81.85
21
0
5
13
DENSE
16384
4096
4.00
3.38
0.06
8.62
2.55
82.88
1
0
0
14
DENSE
16384
4096
4.00
4.65
0.04
11.27
2.42
87.00
4
0
0
16
DENSE
12288
4096
3.00
4.24
0.01
7.85
1.85
86.04
0
0
0
17
DENSE
4096
4096
1.00
4.04
0.04
9.47
2.34
81.76
36
0
5
19
DENSE
16384
4096
4.00
2.76
0.03
6.68
2.42
83.29
1
0
0
20
DENSE
16384
4096
4.00
3.84
0.03
9.28
2.41
81.74
2
0
0
22
DENSE
12288
4096
3.00
3.61
0.02
7.03
1.95
82.41
0
0
0
23
DENSE
4096
4096
1.00
3.73
0.02
8.33
2.23
81.40
25
0
4
25
DENSE
16384
4096
4.00
3.22
0.01
7.29
2.26
78.85
0
0
0
26
DENSE
16384
4096
4.00
3.58
0.01
8.44
2.36
78.39
3
0
0
28
DENSE
12288
4096
3.00
3.06
0.04
6.04
1.97
80.35
0
0
0
29
DENSE
4096
4096
1.00
3.70
0.03
7.59
2.05
83.82
14
0
4
31
DENSE
16384
4096
4.00
3.01
0.01
6.41
2.13
76.53
0
0
0
32
DENSE
16384
4096
4.00
3.53
0.01
8.18
2.32
77.26
3
0
0
34
DENSE
12288
4096
3.00
3.10
0.04
6.05
1.95
80.06
0
0
0
35
DENSE
4096
4096
1.00
3.66
0.02
6.95
1.90
84.69
3
0
4
37
DENSE
16384
4096
4.00
2.98
0.01
6.31
2.12
76.94
0
0
0
38
DENSE
16384
4096
4.00
3.57
0.01
8.23
2.31
78.58
0
0
0
40
DENSE
12288
4096
3.00
3.34
0.03
6.39
1.92
81.50
0
0
0
41
DENSE
4096
4096
1.00
3.96
0.03
7.51
1.89
84.98
0
0
5
43
DENSE
16384
4096
4.00
3.02
0.01
6.60
2.18
79.46
0
0
0
44
DENSE
16384
4096
4.00
3.58
0.01
7.56
2.11
81.94
3
0
0
46
DENSE
12288
4096
3.00
3.04
0.02
5.86
1.93
80.56
0
0
0
47
DENSE
4096
4096
1.00
3.70
0.04
6.17
1.67
87.70
0
0
3
49
DENSE
16384
4096
4.00
3.08
0.01
8.34
2.71
80.48
0
0
0
50
DENSE
16384
4096
4.00
3.77
0.01
8.49
2.25
87.37
16
0
0
52
DENSE
12288
4096
3.00
3.18
0.03
6.25
1.97
81.34
0
0
0
53
DENSE
4096
4096
1.00
3.42
0.08
5.29
1.55
88.72
0
4
5
55
DENSE
16384
4096
4.00
3.19
0.01
8.29
2.60
83.01
0
0
0
56
DENSE
16384
4096
4.00
3.82
0.02
9.33
2.44
88.67
22
0
0
58
DENSE
12288
4096
3.00
3.31
0.05
5.97
1.80
82.18
0
0
0
59
DENSE
4096
4096
1.00
4.05
0.03
5.75
1.42
89.19
0
0
5
61
DENSE
16384
4096
4.00
3.09
0.01
8.00
2.59
82.52
0
0
0
62
DENSE
16384
4096
4.00
3.42
0.03
7.52
2.20
87.91
14
0
0
64
DENSE
12288
4096
3.00
2.85
0.05
5.18
1.82
82.34
0
0
0
65
DENSE
4096
4096
1.00
4.40
0.06
5.82
1.32
89.62
0
1
4
67
DENSE
16384
4096
4.00
3.03
0.02
7.51
2.47
83.17
1
0
0
68
DENSE
16384
4096
4.00
3.21
0.03
6.16
1.92
86.98
8
0
0
70
DENSE
12288
4096
3.00
3.53
0.05
6.47
1.84
81.16
0
0
0
71
DENSE
4096
4096
1.00
3.08
0.05
3.77
1.22
86.07
0
0
5
73
DENSE
16384
4096
4.00
3.10
0.02
7.64
2.47
84.37
1
0
0
74
DENSE
16384
4096
4.00
3.36
0.03
6.59
1.96
87.64
2
0
0
76
DENSE
12288
4096
3.00
2.24
0.05
4.11
1.83
76.16
0
1
0
77
DENSE
4096
4096
1.00
4.09
0.05
3.87
0.95
86.45
0
0
5
79
DENSE
16384
4096
4.00
3.05
0.02
7.65
2.50
83.58
1
0
0
80
DENSE
16384
4096
4.00
3.34
0.03
6.08
1.82
87.21
1
0
0
82
DENSE
12288
4096
3.00
2.99
0.06
5.41
1.81
77.98
0
0
0
83
DENSE
4096
4096
1.00
3.60
0.07
3.83
1.06
85.00
0
0
5
85
DENSE
16384
4096
4.00
3.18
0.03
8.06
2.53
82.56
1
0
0
86
DENSE
16384
4096
4.00
3.19
0.04
5.39
1.69
87.09
0
0
0
88
DENSE
12288
4096
3.00
3.14
0.06
5.70
1.81
78.19
0
0
0
89
DENSE
4096
4096
1.00
3.53
0.07
3.63
1.03
85.17
0
0
4
91
DENSE
16384
4096
4.00
3.01
0.03
7.48
2.48
81.88
1
0
0
92
DENSE
16384
4096
4.00
3.13
0.04
5.14
1.64
86.13
0
0
0
94
DENSE
12288
4096
3.00
2.33
0.08
4.14
1.78
79.60
0
6
0
95
DENSE
4096
4096
1.00
3.77
0.08
3.58
0.95
87.07
0
0
4
97
DENSE
16384
4096
4.00
3.11
0.03
7.81
2.51
82.12
0
0
0
98
DENSE
16384
4096
4.00
3.23
0.04
5.32
1.65
86.65
0
0
0
100
DENSE
12288
4096
3.00
2.98
0.04
5.36
1.80
80.48
0
0
0
101
DENSE
4096
4096
1.00
3.22
0.05
3.38
1.05
85.66
0
0
4
103
DENSE
16384
4096
4.00
3.07
0.02
7.66
2.50
82.60
0
0
0
104
DENSE
16384
4096
4.00
3.03
0.04
4.91
1.62
87.09
0
0
0
106
DENSE
12288
4096
3.00
3.20
0.04
5.96
1.86
82.54
0
0
0
107
DENSE
4096
4096
1.00
3.69
0.06
3.75
1.02
87.42
0
0
5
109
DENSE
16384
4096
4.00
2.91
0.02
7.32
2.51
81.99
0
0
0
110
DENSE
16384
4096
4.00
3.52
0.04
5.74
1.63
88.18
0
0
0
112
DENSE
12288
4096
3.00
3.14
0.04
5.76
1.83
83.61
0
0
0
113
DENSE
4096
4096
1.00
3.08
0.04
3.16
1.03
85.77
0
0
3
115
DENSE
16384
4096
4.00
2.97
0.02
7.42
2.50
82.29
0
0
0
116
DENSE
16384
4096
4.00
3.38
0.03
5.83
1.72
88.42
0
0
0
118
DENSE
12288
4096
3.00
3.49
0.04
6.01
1.72
85.36
0
0
0
119
DENSE
4096
4096
1.00
3.15
0.07
3.15
1.00
86.71
0
0
4
121
DENSE
16384
4096
4.00
3.05
0.02
7.58
2.49
83.15
0
0
0
122
DENSE
16384
4096
4.00
3.52
0.03
6.14
1.75
88.71
0
0
0
124
DENSE
12288
4096
3.00
3.62
0.04
6.37
1.76
85.84
0
0
0
125
DENSE
4096
4096
1.00
7.43
0.06
8.30
1.12
87.98
0
0
4
127
DENSE
16384
4096
4.00
3.06
0.03
7.62
2.49
83.35
0
0
0
128
DENSE
16384
4096
4.00
3.82
0.03
6.83
1.79
89.59
0
0
0
130
DENSE
12288
4096
3.00
3.64
0.03
6.34
1.74
87.79
0
0
0
131
DENSE
4096
4096
1.00
8.09
0.04
9.60
1.19
92.00
0
0
4
133
DENSE
16384
4096
4.00
3.10
0.02
7.73
2.49
83.67
0
0
0
134
DENSE
16384
4096
4.00
4.28
0.03
7.93
1.85
89.70
0
0
0
136
DENSE
12288
4096
3.00
3.68
0.02
6.29
1.71
88.25
0
0
0
137
DENSE
4096
4096
1.00
5.40
0.02
7.24
1.34
90.19
0
0
4
139
DENSE
16384
4096
4.00
3.17
0.02
7.92
2.50
84.15
0
0
0
140
DENSE
16384
4096
4.00
4.64
0.03
8.61
1.85
90.05
0
0
0
142
DENSE
12288
4096
3.00
4.07
0.02
6.86
1.68
89.95
0
0
0
143
DENSE
4096
4096
1.00
5.61
0.04
7.69
1.37
91.42
0
0
3
145
DENSE
16384
4096
4.00
3.14
0.01
7.85
2.50
84.10
0
0
0
146
DENSE
16384
4096
4.00
5.00
0.03
9.02
1.80
91.38
1
0
0
148
DENSE
12288
4096
3.00
4.15
0.02
7.10
1.71
89.36
0
0
0
149
DENSE
4096
4096
1.00
4.84
0.03
7.71
1.59
89.14
0
0
3
151
DENSE
16384
4096
4.00
3.11
0.01
7.80
2.51
83.95
0
0
0
152
DENSE
16384
4096
4.00
4.52
0.04
7.91
1.75
92.17
0
1
0
154
DENSE
12288
4096
3.00
4.27
0.02
7.44
1.74
88.92
0
0
0
155
DENSE
4096
4096
1.00
3.43
0.07
6.65
1.94
85.85
14
0
4
157
DENSE
16384
4096
4.00
3.08
0.01
7.76
2.52
83.65
0
0
0
158
DENSE
16384
4096
4.00
4.73
0.01
8.20
1.73
92.84
1
0
0
160
DENSE
12288
4096
3.00
3.67
0.03
6.45
1.76
88.09
0
0
0
161
DENSE
4096
4096
1.00
3.26
0.06
7.12
2.18
80.87
21
0
4
163
DENSE
16384
4096
4.00
3.08
0.01
7.83
2.54
83.80
0
0
0
164
DENSE
16384
4096
4.00
4.30
0.02
8.15
1.89
92.06
2
0
0
166
DENSE
12288
4096
3.00
3.63
0.01
6.65
1.83
87.77
0
0
0
167
DENSE
4096
4096
1.00
2.02
0.08
5.16
2.56
75.15
49
0
2
169
DENSE
16384
4096
4.00
3.08
0.01
7.98
2.59
83.75
0
0
0
170
DENSE
16384
4096
4.00
4.11
0.03
11.72
2.85
86.90
3
0
0
172
DENSE
12288
4096
3.00
3.50
0.01
7.27
2.08
86.72
0
0
0
173
DENSE
4096
4096
1.00
1.99
0.07
5.03
2.53
70.15
63
0
4
175
DENSE
16384
4096
4.00
3.14
0.01
8.11
2.58
84.37
0
0
0
176
DENSE
16384
4096
4.00
4.26
0.02
10.91
2.56
88.91
5
0
0
178
DENSE
12288
4096
3.00
3.32
0.01
6.73
2.03
85.98
0
0
0
179
DENSE
4096
4096
1.00
2.79
0.04
7.58
2.72
70.58
12
0
3
181
DENSE
16384
4096
4.00
3.34
0.02
8.47
2.54
85.25
0
0
0
182
DENSE
16384
4096
4.00
4.30
0.03
11.51
2.68
87.88
1
0
0
bigscience-bloom-7b1 Layer Plots
Layer 4
Layer=4 | N=12288 | M=4096 | Q=3.00 | alpha=3.07 | D_ks=0.02 | alpha-hat=6.29 | num traps=0
Layer 5
Layer=5 | N=4096 | M=4096 | Q=1.00 | alpha=3.11 | D_ks=0.02 | alpha-hat=6.23 | num traps=4
Layer 7
Layer=7 | N=16384 | M=4096 | Q=4.00 | alpha=2.37 | D_ks=0.05 | alpha-hat=6.06 | num traps=1
Layer 8
Layer=8 | N=16384 | M=4096 | Q=4.00 | alpha=5.91 | D_ks=0.05 | alpha-hat=14.97 | num traps=5
Layer 10
Layer=10 | N=12288 | M=4096 | Q=3.00 | alpha=3.51 | D_ks=0.03 | alpha-hat=7.16 | num traps=0
Layer 11
Layer=11 | N=4096 | M=4096 | Q=1.00 | alpha=3.51 | D_ks=0.03 | alpha-hat=7.73 | num traps=21
Layer 13
Layer=13 | N=16384 | M=4096 | Q=4.00 | alpha=3.38 | D_ks=0.06 | alpha-hat=8.62 | num traps=1
Layer 14
Layer=14 | N=16384 | M=4096 | Q=4.00 | alpha=4.65 | D_ks=0.04 | alpha-hat=11.27 | num traps=4
Layer 16
Layer=16 | N=12288 | M=4096 | Q=3.00 | alpha=4.24 | D_ks=0.01 | alpha-hat=7.85 | num traps=0
Layer 17
Layer=17 | N=4096 | M=4096 | Q=1.00 | alpha=4.04 | D_ks=0.04 | alpha-hat=9.47 | num traps=36
Layer 19
Layer=19 | N=16384 | M=4096 | Q=4.00 | alpha=2.76 | D_ks=0.03 | alpha-hat=6.68 | num traps=1
Layer 20
Layer=20 | N=16384 | M=4096 | Q=4.00 | alpha=3.84 | D_ks=0.03 | alpha-hat=9.28 | num traps=2
Layer 22
Layer=22 | N=12288 | M=4096 | Q=3.00 | alpha=3.61 | D_ks=0.02 | alpha-hat=7.03 | num traps=0
Layer 23
Layer=23 | N=4096 | M=4096 | Q=1.00 | alpha=3.73 | D_ks=0.02 | alpha-hat=8.33 | num traps=25
Layer 25
Layer=25 | N=16384 | M=4096 | Q=4.00 | alpha=3.22 | D_ks=0.01 | alpha-hat=7.29 | num traps=0
Layer 26
Layer=26 | N=16384 | M=4096 | Q=4.00 | alpha=3.58 | D_ks=0.01 | alpha-hat=8.44 | num traps=3
Layer 28
Layer=28 | N=12288 | M=4096 | Q=3.00 | alpha=3.06 | D_ks=0.04 | alpha-hat=6.04 | num traps=0
Layer 29
Layer=29 | N=4096 | M=4096 | Q=1.00 | alpha=3.70 | D_ks=0.03 | alpha-hat=7.59 | num traps=14
Layer 31
Layer=31 | N=16384 | M=4096 | Q=4.00 | alpha=3.01 | D_ks=0.01 | alpha-hat=6.41 | num traps=0
Layer 32
Layer=32 | N=16384 | M=4096 | Q=4.00 | alpha=3.53 | D_ks=0.01 | alpha-hat=8.18 | num traps=3
Layer 34
Layer=34 | N=12288 | M=4096 | Q=3.00 | alpha=3.10 | D_ks=0.04 | alpha-hat=6.05 | num traps=0
Layer 35
Layer=35 | N=4096 | M=4096 | Q=1.00 | alpha=3.66 | D_ks=0.02 | alpha-hat=6.95 | num traps=3
Layer 37
Layer=37 | N=16384 | M=4096 | Q=4.00 | alpha=2.98 | D_ks=0.01 | alpha-hat=6.31 | num traps=0
Layer 38
Layer=38 | N=16384 | M=4096 | Q=4.00 | alpha=3.57 | D_ks=0.01 | alpha-hat=8.23 | num traps=0
Layer 40
Layer=40 | N=12288 | M=4096 | Q=3.00 | alpha=3.34 | D_ks=0.03 | alpha-hat=6.39 | num traps=0
Layer 41
Layer=41 | N=4096 | M=4096 | Q=1.00 | alpha=3.96 | D_ks=0.03 | alpha-hat=7.51 | num traps=0
Layer 43
Layer=43 | N=16384 | M=4096 | Q=4.00 | alpha=3.02 | D_ks=0.01 | alpha-hat=6.60 | num traps=0
Layer 44
Layer=44 | N=16384 | M=4096 | Q=4.00 | alpha=3.58 | D_ks=0.01 | alpha-hat=7.56 | num traps=3
Layer 46
Layer=46 | N=12288 | M=4096 | Q=3.00 | alpha=3.04 | D_ks=0.02 | alpha-hat=5.86 | num traps=0
Layer 47
Layer=47 | N=4096 | M=4096 | Q=1.00 | alpha=3.70 | D_ks=0.04 | alpha-hat=6.17 | num traps=0
Layer 49
Layer=49 | N=16384 | M=4096 | Q=4.00 | alpha=3.08 | D_ks=0.01 | alpha-hat=8.34 | num traps=0
Layer 50
Layer=50 | N=16384 | M=4096 | Q=4.00 | alpha=3.77 | D_ks=0.01 | alpha-hat=8.49 | num traps=16
Layer 52
Layer=52 | N=12288 | M=4096 | Q=3.00 | alpha=3.18 | D_ks=0.03 | alpha-hat=6.25 | num traps=0
Layer 53
Layer=53 | N=4096 | M=4096 | Q=1.00 | alpha=3.42 | D_ks=0.08 | alpha-hat=5.29 | num traps=0
Layer 55
Layer=55 | N=16384 | M=4096 | Q=4.00 | alpha=3.19 | D_ks=0.01 | alpha-hat=8.29 | num traps=0
Layer 56
Layer=56 | N=16384 | M=4096 | Q=4.00 | alpha=3.82 | D_ks=0.02 | alpha-hat=9.33 | num traps=22
Layer 58
Layer=58 | N=12288 | M=4096 | Q=3.00 | alpha=3.31 | D_ks=0.05 | alpha-hat=5.97 | num traps=0
Layer 59
Layer=59 | N=4096 | M=4096 | Q=1.00 | alpha=4.05 | D_ks=0.03 | alpha-hat=5.75 | num traps=0
Layer 61
Layer=61 | N=16384 | M=4096 | Q=4.00 | alpha=3.09 | D_ks=0.01 | alpha-hat=8.00 | num traps=0
Layer 62
Layer=62 | N=16384 | M=4096 | Q=4.00 | alpha=3.42 | D_ks=0.03 | alpha-hat=7.52 | num traps=14
Layer 64
Layer=64 | N=12288 | M=4096 | Q=3.00 | alpha=2.85 | D_ks=0.05 | alpha-hat=5.18 | num traps=0
Layer 65
Layer=65 | N=4096 | M=4096 | Q=1.00 | alpha=4.40 | D_ks=0.06 | alpha-hat=5.82 | num traps=0
Layer 67
Layer=67 | N=16384 | M=4096 | Q=4.00 | alpha=3.03 | D_ks=0.02 | alpha-hat=7.51 | num traps=1
Layer 68
Layer=68 | N=16384 | M=4096 | Q=4.00 | alpha=3.21 | D_ks=0.03 | alpha-hat=6.16 | num traps=8
Layer 70
Layer=70 | N=12288 | M=4096 | Q=3.00 | alpha=3.53 | D_ks=0.05 | alpha-hat=6.47 | num traps=0
Layer 71
Layer=71 | N=4096 | M=4096 | Q=1.00 | alpha=3.08 | D_ks=0.05 | alpha-hat=3.77 | num traps=0
Layer 73
Layer=73 | N=16384 | M=4096 | Q=4.00 | alpha=3.10 | D_ks=0.02 | alpha-hat=7.64 | num traps=1
Layer 74
Layer=74 | N=16384 | M=4096 | Q=4.00 | alpha=3.36 | D_ks=0.03 | alpha-hat=6.59 | num traps=2
Layer 76
Layer=76 | N=12288 | M=4096 | Q=3.00 | alpha=2.24 | D_ks=0.05 | alpha-hat=4.11 | num traps=0
Layer 77
Layer=77 | N=4096 | M=4096 | Q=1.00 | alpha=4.09 | D_ks=0.05 | alpha-hat=3.87 | num traps=0
Layer 79
Layer=79 | N=16384 | M=4096 | Q=4.00 | alpha=3.05 | D_ks=0.02 | alpha-hat=7.65 | num traps=1
Layer 80
Layer=80 | N=16384 | M=4096 | Q=4.00 | alpha=3.34 | D_ks=0.03 | alpha-hat=6.08 | num traps=1
Layer 82
Layer=82 | N=12288 | M=4096 | Q=3.00 | alpha=2.99 | D_ks=0.06 | alpha-hat=5.41 | num traps=0
Layer 83
Layer=83 | N=4096 | M=4096 | Q=1.00 | alpha=3.60 | D_ks=0.07 | alpha-hat=3.83 | num traps=0
Layer 85
Layer=85 | N=16384 | M=4096 | Q=4.00 | alpha=3.18 | D_ks=0.03 | alpha-hat=8.06 | num traps=1
Layer 86
Layer=86 | N=16384 | M=4096 | Q=4.00 | alpha=3.19 | D_ks=0.04 | alpha-hat=5.39 | num traps=0
Layer 88
Layer=88 | N=12288 | M=4096 | Q=3.00 | alpha=3.14 | D_ks=0.06 | alpha-hat=5.70 | num traps=0
Layer 89
Layer=89 | N=4096 | M=4096 | Q=1.00 | alpha=3.53 | D_ks=0.07 | alpha-hat=3.63 | num traps=0
Layer 91
Layer=91 | N=16384 | M=4096 | Q=4.00 | alpha=3.01 | D_ks=0.03 | alpha-hat=7.48 | num traps=1
Layer 92
Layer=92 | N=16384 | M=4096 | Q=4.00 | alpha=3.13 | D_ks=0.04 | alpha-hat=5.14 | num traps=0
Layer 94
Layer=94 | N=12288 | M=4096 | Q=3.00 | alpha=2.33 | D_ks=0.08 | alpha-hat=4.14 | num traps=0
Layer 95
Layer=95 | N=4096 | M=4096 | Q=1.00 | alpha=3.77 | D_ks=0.08 | alpha-hat=3.58 | num traps=0
Layer 97
Layer=97 | N=16384 | M=4096 | Q=4.00 | alpha=3.11 | D_ks=0.03 | alpha-hat=7.81 | num traps=0
Layer 98
Layer=98 | N=16384 | M=4096 | Q=4.00 | alpha=3.23 | D_ks=0.04 | alpha-hat=5.32 | num traps=0
Layer 100
Layer=100 | N=12288 | M=4096 | Q=3.00 | alpha=2.98 | D_ks=0.04 | alpha-hat=5.36 | num traps=0
Layer 101
Layer=101 | N=4096 | M=4096 | Q=1.00 | alpha=3.22 | D_ks=0.05 | alpha-hat=3.38 | num traps=0
Layer 103
Layer=103 | N=16384 | M=4096 | Q=4.00 | alpha=3.07 | D_ks=0.02 | alpha-hat=7.66 | num traps=0
Layer 104
Layer=104 | N=16384 | M=4096 | Q=4.00 | alpha=3.03 | D_ks=0.04 | alpha-hat=4.91 | num traps=0
Layer 106
Layer=106 | N=12288 | M=4096 | Q=3.00 | alpha=3.20 | D_ks=0.04 | alpha-hat=5.96 | num traps=0
Layer 107
Layer=107 | N=4096 | M=4096 | Q=1.00 | alpha=3.69 | D_ks=0.06 | alpha-hat=3.75 | num traps=0
Layer 109
Layer=109 | N=16384 | M=4096 | Q=4.00 | alpha=2.91 | D_ks=0.02 | alpha-hat=7.32 | num traps=0
Layer 110
Layer=110 | N=16384 | M=4096 | Q=4.00 | alpha=3.52 | D_ks=0.04 | alpha-hat=5.74 | num traps=0
Layer 112
Layer=112 | N=12288 | M=4096 | Q=3.00 | alpha=3.14 | D_ks=0.04 | alpha-hat=5.76 | num traps=0
Layer 113
Layer=113 | N=4096 | M=4096 | Q=1.00 | alpha=3.08 | D_ks=0.04 | alpha-hat=3.16 | num traps=0
Layer 115
Layer=115 | N=16384 | M=4096 | Q=4.00 | alpha=2.97 | D_ks=0.02 | alpha-hat=7.42 | num traps=0
Layer 116
Layer=116 | N=16384 | M=4096 | Q=4.00 | alpha=3.38 | D_ks=0.03 | alpha-hat=5.83 | num traps=0
Layer 118
Layer=118 | N=12288 | M=4096 | Q=3.00 | alpha=3.49 | D_ks=0.04 | alpha-hat=6.01 | num traps=0
Layer 119
Layer=119 | N=4096 | M=4096 | Q=1.00 | alpha=3.15 | D_ks=0.07 | alpha-hat=3.15 | num traps=0
Layer 121
Layer=121 | N=16384 | M=4096 | Q=4.00 | alpha=3.05 | D_ks=0.02 | alpha-hat=7.58 | num traps=0
Layer 122
Layer=122 | N=16384 | M=4096 | Q=4.00 | alpha=3.52 | D_ks=0.03 | alpha-hat=6.14 | num traps=0
Layer 124
Layer=124 | N=12288 | M=4096 | Q=3.00 | alpha=3.62 | D_ks=0.04 | alpha-hat=6.37 | num traps=0
Layer 125
Layer=125 | N=4096 | M=4096 | Q=1.00 | alpha=7.43 | D_ks=0.06 | alpha-hat=8.30 | num traps=0
Layer 127
Layer=127 | N=16384 | M=4096 | Q=4.00 | alpha=3.06 | D_ks=0.03 | alpha-hat=7.62 | num traps=0
Layer 128
Layer=128 | N=16384 | M=4096 | Q=4.00 | alpha=3.82 | D_ks=0.03 | alpha-hat=6.83 | num traps=0
Layer 130
Layer=130 | N=12288 | M=4096 | Q=3.00 | alpha=3.64 | D_ks=0.03 | alpha-hat=6.34 | num traps=0
Layer 131
Layer=131 | N=4096 | M=4096 | Q=1.00 | alpha=8.09 | D_ks=0.04 | alpha-hat=9.60 | num traps=0
Layer 133
Layer=133 | N=16384 | M=4096 | Q=4.00 | alpha=3.10 | D_ks=0.02 | alpha-hat=7.73 | num traps=0
Layer 134
Layer=134 | N=16384 | M=4096 | Q=4.00 | alpha=4.28 | D_ks=0.03 | alpha-hat=7.93 | num traps=0
Layer 136
Layer=136 | N=12288 | M=4096 | Q=3.00 | alpha=3.68 | D_ks=0.02 | alpha-hat=6.29 | num traps=0
Layer 137
Layer=137 | N=4096 | M=4096 | Q=1.00 | alpha=5.40 | D_ks=0.02 | alpha-hat=7.24 | num traps=0
Layer 139
Layer=139 | N=16384 | M=4096 | Q=4.00 | alpha=3.17 | D_ks=0.02 | alpha-hat=7.92 | num traps=0
Layer 140
Layer=140 | N=16384 | M=4096 | Q=4.00 | alpha=4.64 | D_ks=0.03 | alpha-hat=8.61 | num traps=0
Layer 142
Layer=142 | N=12288 | M=4096 | Q=3.00 | alpha=4.07 | D_ks=0.02 | alpha-hat=6.86 | num traps=0
Layer 143
Layer=143 | N=4096 | M=4096 | Q=1.00 | alpha=5.61 | D_ks=0.04 | alpha-hat=7.69 | num traps=0
Layer 145
Layer=145 | N=16384 | M=4096 | Q=4.00 | alpha=3.14 | D_ks=0.01 | alpha-hat=7.85 | num traps=0
Layer 146
Layer=146 | N=16384 | M=4096 | Q=4.00 | alpha=5.00 | D_ks=0.03 | alpha-hat=9.02 | num traps=1
Layer 148
Layer=148 | N=12288 | M=4096 | Q=3.00 | alpha=4.15 | D_ks=0.02 | alpha-hat=7.10 | num traps=0
Layer 149
Layer=149 | N=4096 | M=4096 | Q=1.00 | alpha=4.84 | D_ks=0.03 | alpha-hat=7.71 | num traps=0
Layer 151
Layer=151 | N=16384 | M=4096 | Q=4.00 | alpha=3.11 | D_ks=0.01 | alpha-hat=7.80 | num traps=0
Layer 152
Layer=152 | N=16384 | M=4096 | Q=4.00 | alpha=4.52 | D_ks=0.04 | alpha-hat=7.91 | num traps=0
Layer 154
Layer=154 | N=12288 | M=4096 | Q=3.00 | alpha=4.27 | D_ks=0.02 | alpha-hat=7.44 | num traps=0
Layer 155
Layer=155 | N=4096 | M=4096 | Q=1.00 | alpha=3.43 | D_ks=0.07 | alpha-hat=6.65 | num traps=14
Layer 157
Layer=157 | N=16384 | M=4096 | Q=4.00 | alpha=3.08 | D_ks=0.01 | alpha-hat=7.76 | num traps=0
Layer 158
Layer=158 | N=16384 | M=4096 | Q=4.00 | alpha=4.73 | D_ks=0.01 | alpha-hat=8.20 | num traps=1
Layer 160
Layer=160 | N=12288 | M=4096 | Q=3.00 | alpha=3.67 | D_ks=0.03 | alpha-hat=6.45 | num traps=0
Layer 161
Layer=161 | N=4096 | M=4096 | Q=1.00 | alpha=3.26 | D_ks=0.06 | alpha-hat=7.12 | num traps=21
Layer 163
Layer=163 | N=16384 | M=4096 | Q=4.00 | alpha=3.08 | D_ks=0.01 | alpha-hat=7.83 | num traps=0
Layer 164
Layer=164 | N=16384 | M=4096 | Q=4.00 | alpha=4.30 | D_ks=0.02 | alpha-hat=8.15 | num traps=2
Layer 166
Layer=166 | N=12288 | M=4096 | Q=3.00 | alpha=3.63 | D_ks=0.01 | alpha-hat=6.65 | num traps=0
Layer 167
Layer=167 | N=4096 | M=4096 | Q=1.00 | alpha=2.02 | D_ks=0.08 | alpha-hat=5.16 | num traps=49
Layer 169
Layer=169 | N=16384 | M=4096 | Q=4.00 | alpha=3.08 | D_ks=0.01 | alpha-hat=7.98 | num traps=0
Layer 170
Layer=170 | N=16384 | M=4096 | Q=4.00 | alpha=4.11 | D_ks=0.03 | alpha-hat=11.72 | num traps=3
Layer 172
Layer=172 | N=12288 | M=4096 | Q=3.00 | alpha=3.50 | D_ks=0.01 | alpha-hat=7.27 | num traps=0
Layer 173
Layer=173 | N=4096 | M=4096 | Q=1.00 | alpha=1.99 | D_ks=0.07 | alpha-hat=5.03 | num traps=63
Layer 175
Layer=175 | N=16384 | M=4096 | Q=4.00 | alpha=3.14 | D_ks=0.01 | alpha-hat=8.11 | num traps=0
Layer 176
Layer=176 | N=16384 | M=4096 | Q=4.00 | alpha=4.26 | D_ks=0.02 | alpha-hat=10.91 | num traps=5
Layer 178
Layer=178 | N=12288 | M=4096 | Q=3.00 | alpha=3.32 | D_ks=0.01 | alpha-hat=6.73 | num traps=0
Layer 179
Layer=179 | N=4096 | M=4096 | Q=1.00 | alpha=2.79 | D_ks=0.04 | alpha-hat=7.58 | num traps=12
Layer 181
Layer=181 | N=16384 | M=4096 | Q=4.00 | alpha=3.34 | D_ks=0.02 | alpha-hat=8.47 | num traps=0
Layer 182
Layer=182 | N=16384 | M=4096 | Q=4.00 | alpha=4.30 | D_ks=0.03 | alpha-hat=11.51 | num traps=1
bigscience-bloom-7b1 Model Summary Plots
bigscience-bloom-7b1 Model Selected Details
| layer_type | N | M | Q | alpha | D | alpha-hat | log_SN | % Rand | num_traps | num_fingers | rank_loss | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| layer_id | ||||||||||||
| 4 | DENSE | 12288 | 4096 | 3.00 | 3.07 | 0.02 | 6.29 | 2.05 | 77.85 | 0 | 0 | 0 |
| 5 | DENSE | 4096 | 4096 | 1.00 | 3.11 | 0.02 | 6.23 | 2.00 | 82.04 | 4 | 0 | 6 |
| 7 | DENSE | 16384 | 4096 | 4.00 | 2.37 | 0.05 | 6.06 | 2.56 | 87.05 | 1 | 0 | 0 |
| 8 | DENSE | 16384 | 4096 | 4.00 | 5.91 | 0.05 | 14.97 | 2.53 | 87.80 | 5 | 0 | 0 |
| 10 | DENSE | 12288 | 4096 | 3.00 | 3.51 | 0.03 | 7.16 | 2.04 | 83.05 | 0 | 0 | 0 |
| 11 | DENSE | 4096 | 4096 | 1.00 | 3.51 | 0.03 | 7.73 | 2.20 | 81.85 | 21 | 0 | 5 |
| 13 | DENSE | 16384 | 4096 | 4.00 | 3.38 | 0.06 | 8.62 | 2.55 | 82.88 | 1 | 0 | 0 |
| 14 | DENSE | 16384 | 4096 | 4.00 | 4.65 | 0.04 | 11.27 | 2.42 | 87.00 | 4 | 0 | 0 |
| 16 | DENSE | 12288 | 4096 | 3.00 | 4.24 | 0.01 | 7.85 | 1.85 | 86.04 | 0 | 0 | 0 |
| 17 | DENSE | 4096 | 4096 | 1.00 | 4.04 | 0.04 | 9.47 | 2.34 | 81.76 | 36 | 0 | 5 |
| 19 | DENSE | 16384 | 4096 | 4.00 | 2.76 | 0.03 | 6.68 | 2.42 | 83.29 | 1 | 0 | 0 |
| 20 | DENSE | 16384 | 4096 | 4.00 | 3.84 | 0.03 | 9.28 | 2.41 | 81.74 | 2 | 0 | 0 |
| 22 | DENSE | 12288 | 4096 | 3.00 | 3.61 | 0.02 | 7.03 | 1.95 | 82.41 | 0 | 0 | 0 |
| 23 | DENSE | 4096 | 4096 | 1.00 | 3.73 | 0.02 | 8.33 | 2.23 | 81.40 | 25 | 0 | 4 |
| 25 | DENSE | 16384 | 4096 | 4.00 | 3.22 | 0.01 | 7.29 | 2.26 | 78.85 | 0 | 0 | 0 |
| 26 | DENSE | 16384 | 4096 | 4.00 | 3.58 | 0.01 | 8.44 | 2.36 | 78.39 | 3 | 0 | 0 |
| 28 | DENSE | 12288 | 4096 | 3.00 | 3.06 | 0.04 | 6.04 | 1.97 | 80.35 | 0 | 0 | 0 |
| 29 | DENSE | 4096 | 4096 | 1.00 | 3.70 | 0.03 | 7.59 | 2.05 | 83.82 | 14 | 0 | 4 |
| 31 | DENSE | 16384 | 4096 | 4.00 | 3.01 | 0.01 | 6.41 | 2.13 | 76.53 | 0 | 0 | 0 |
| 32 | DENSE | 16384 | 4096 | 4.00 | 3.53 | 0.01 | 8.18 | 2.32 | 77.26 | 3 | 0 | 0 |
| 34 | DENSE | 12288 | 4096 | 3.00 | 3.10 | 0.04 | 6.05 | 1.95 | 80.06 | 0 | 0 | 0 |
| 35 | DENSE | 4096 | 4096 | 1.00 | 3.66 | 0.02 | 6.95 | 1.90 | 84.69 | 3 | 0 | 4 |
| 37 | DENSE | 16384 | 4096 | 4.00 | 2.98 | 0.01 | 6.31 | 2.12 | 76.94 | 0 | 0 | 0 |
| 38 | DENSE | 16384 | 4096 | 4.00 | 3.57 | 0.01 | 8.23 | 2.31 | 78.58 | 0 | 0 | 0 |
| 40 | DENSE | 12288 | 4096 | 3.00 | 3.34 | 0.03 | 6.39 | 1.92 | 81.50 | 0 | 0 | 0 |
| 41 | DENSE | 4096 | 4096 | 1.00 | 3.96 | 0.03 | 7.51 | 1.89 | 84.98 | 0 | 0 | 5 |
| 43 | DENSE | 16384 | 4096 | 4.00 | 3.02 | 0.01 | 6.60 | 2.18 | 79.46 | 0 | 0 | 0 |
| 44 | DENSE | 16384 | 4096 | 4.00 | 3.58 | 0.01 | 7.56 | 2.11 | 81.94 | 3 | 0 | 0 |
| 46 | DENSE | 12288 | 4096 | 3.00 | 3.04 | 0.02 | 5.86 | 1.93 | 80.56 | 0 | 0 | 0 |
| 47 | DENSE | 4096 | 4096 | 1.00 | 3.70 | 0.04 | 6.17 | 1.67 | 87.70 | 0 | 0 | 3 |
| 49 | DENSE | 16384 | 4096 | 4.00 | 3.08 | 0.01 | 8.34 | 2.71 | 80.48 | 0 | 0 | 0 |
| 50 | DENSE | 16384 | 4096 | 4.00 | 3.77 | 0.01 | 8.49 | 2.25 | 87.37 | 16 | 0 | 0 |
| 52 | DENSE | 12288 | 4096 | 3.00 | 3.18 | 0.03 | 6.25 | 1.97 | 81.34 | 0 | 0 | 0 |
| 53 | DENSE | 4096 | 4096 | 1.00 | 3.42 | 0.08 | 5.29 | 1.55 | 88.72 | 0 | 4 | 5 |
| 55 | DENSE | 16384 | 4096 | 4.00 | 3.19 | 0.01 | 8.29 | 2.60 | 83.01 | 0 | 0 | 0 |
| 56 | DENSE | 16384 | 4096 | 4.00 | 3.82 | 0.02 | 9.33 | 2.44 | 88.67 | 22 | 0 | 0 |
| 58 | DENSE | 12288 | 4096 | 3.00 | 3.31 | 0.05 | 5.97 | 1.80 | 82.18 | 0 | 0 | 0 |
| 59 | DENSE | 4096 | 4096 | 1.00 | 4.05 | 0.03 | 5.75 | 1.42 | 89.19 | 0 | 0 | 5 |
| 61 | DENSE | 16384 | 4096 | 4.00 | 3.09 | 0.01 | 8.00 | 2.59 | 82.52 | 0 | 0 | 0 |
| 62 | DENSE | 16384 | 4096 | 4.00 | 3.42 | 0.03 | 7.52 | 2.20 | 87.91 | 14 | 0 | 0 |
| 64 | DENSE | 12288 | 4096 | 3.00 | 2.85 | 0.05 | 5.18 | 1.82 | 82.34 | 0 | 0 | 0 |
| 65 | DENSE | 4096 | 4096 | 1.00 | 4.40 | 0.06 | 5.82 | 1.32 | 89.62 | 0 | 1 | 4 |
| 67 | DENSE | 16384 | 4096 | 4.00 | 3.03 | 0.02 | 7.51 | 2.47 | 83.17 | 1 | 0 | 0 |
| 68 | DENSE | 16384 | 4096 | 4.00 | 3.21 | 0.03 | 6.16 | 1.92 | 86.98 | 8 | 0 | 0 |
| 70 | DENSE | 12288 | 4096 | 3.00 | 3.53 | 0.05 | 6.47 | 1.84 | 81.16 | 0 | 0 | 0 |
| 71 | DENSE | 4096 | 4096 | 1.00 | 3.08 | 0.05 | 3.77 | 1.22 | 86.07 | 0 | 0 | 5 |
| 73 | DENSE | 16384 | 4096 | 4.00 | 3.10 | 0.02 | 7.64 | 2.47 | 84.37 | 1 | 0 | 0 |
| 74 | DENSE | 16384 | 4096 | 4.00 | 3.36 | 0.03 | 6.59 | 1.96 | 87.64 | 2 | 0 | 0 |
| 76 | DENSE | 12288 | 4096 | 3.00 | 2.24 | 0.05 | 4.11 | 1.83 | 76.16 | 0 | 1 | 0 |
| 77 | DENSE | 4096 | 4096 | 1.00 | 4.09 | 0.05 | 3.87 | 0.95 | 86.45 | 0 | 0 | 5 |
| 79 | DENSE | 16384 | 4096 | 4.00 | 3.05 | 0.02 | 7.65 | 2.50 | 83.58 | 1 | 0 | 0 |
| 80 | DENSE | 16384 | 4096 | 4.00 | 3.34 | 0.03 | 6.08 | 1.82 | 87.21 | 1 | 0 | 0 |
| 82 | DENSE | 12288 | 4096 | 3.00 | 2.99 | 0.06 | 5.41 | 1.81 | 77.98 | 0 | 0 | 0 |
| 83 | DENSE | 4096 | 4096 | 1.00 | 3.60 | 0.07 | 3.83 | 1.06 | 85.00 | 0 | 0 | 5 |
| 85 | DENSE | 16384 | 4096 | 4.00 | 3.18 | 0.03 | 8.06 | 2.53 | 82.56 | 1 | 0 | 0 |
| 86 | DENSE | 16384 | 4096 | 4.00 | 3.19 | 0.04 | 5.39 | 1.69 | 87.09 | 0 | 0 | 0 |
| 88 | DENSE | 12288 | 4096 | 3.00 | 3.14 | 0.06 | 5.70 | 1.81 | 78.19 | 0 | 0 | 0 |
| 89 | DENSE | 4096 | 4096 | 1.00 | 3.53 | 0.07 | 3.63 | 1.03 | 85.17 | 0 | 0 | 4 |
| 91 | DENSE | 16384 | 4096 | 4.00 | 3.01 | 0.03 | 7.48 | 2.48 | 81.88 | 1 | 0 | 0 |
| 92 | DENSE | 16384 | 4096 | 4.00 | 3.13 | 0.04 | 5.14 | 1.64 | 86.13 | 0 | 0 | 0 |
| 94 | DENSE | 12288 | 4096 | 3.00 | 2.33 | 0.08 | 4.14 | 1.78 | 79.60 | 0 | 6 | 0 |
| 95 | DENSE | 4096 | 4096 | 1.00 | 3.77 | 0.08 | 3.58 | 0.95 | 87.07 | 0 | 0 | 4 |
| 97 | DENSE | 16384 | 4096 | 4.00 | 3.11 | 0.03 | 7.81 | 2.51 | 82.12 | 0 | 0 | 0 |
| 98 | DENSE | 16384 | 4096 | 4.00 | 3.23 | 0.04 | 5.32 | 1.65 | 86.65 | 0 | 0 | 0 |
| 100 | DENSE | 12288 | 4096 | 3.00 | 2.98 | 0.04 | 5.36 | 1.80 | 80.48 | 0 | 0 | 0 |
| 101 | DENSE | 4096 | 4096 | 1.00 | 3.22 | 0.05 | 3.38 | 1.05 | 85.66 | 0 | 0 | 4 |
| 103 | DENSE | 16384 | 4096 | 4.00 | 3.07 | 0.02 | 7.66 | 2.50 | 82.60 | 0 | 0 | 0 |
| 104 | DENSE | 16384 | 4096 | 4.00 | 3.03 | 0.04 | 4.91 | 1.62 | 87.09 | 0 | 0 | 0 |
| 106 | DENSE | 12288 | 4096 | 3.00 | 3.20 | 0.04 | 5.96 | 1.86 | 82.54 | 0 | 0 | 0 |
| 107 | DENSE | 4096 | 4096 | 1.00 | 3.69 | 0.06 | 3.75 | 1.02 | 87.42 | 0 | 0 | 5 |
| 109 | DENSE | 16384 | 4096 | 4.00 | 2.91 | 0.02 | 7.32 | 2.51 | 81.99 | 0 | 0 | 0 |
| 110 | DENSE | 16384 | 4096 | 4.00 | 3.52 | 0.04 | 5.74 | 1.63 | 88.18 | 0 | 0 | 0 |
| 112 | DENSE | 12288 | 4096 | 3.00 | 3.14 | 0.04 | 5.76 | 1.83 | 83.61 | 0 | 0 | 0 |
| 113 | DENSE | 4096 | 4096 | 1.00 | 3.08 | 0.04 | 3.16 | 1.03 | 85.77 | 0 | 0 | 3 |
| 115 | DENSE | 16384 | 4096 | 4.00 | 2.97 | 0.02 | 7.42 | 2.50 | 82.29 | 0 | 0 | 0 |
| 116 | DENSE | 16384 | 4096 | 4.00 | 3.38 | 0.03 | 5.83 | 1.72 | 88.42 | 0 | 0 | 0 |
| 118 | DENSE | 12288 | 4096 | 3.00 | 3.49 | 0.04 | 6.01 | 1.72 | 85.36 | 0 | 0 | 0 |
| 119 | DENSE | 4096 | 4096 | 1.00 | 3.15 | 0.07 | 3.15 | 1.00 | 86.71 | 0 | 0 | 4 |
| 121 | DENSE | 16384 | 4096 | 4.00 | 3.05 | 0.02 | 7.58 | 2.49 | 83.15 | 0 | 0 | 0 |
| 122 | DENSE | 16384 | 4096 | 4.00 | 3.52 | 0.03 | 6.14 | 1.75 | 88.71 | 0 | 0 | 0 |
| 124 | DENSE | 12288 | 4096 | 3.00 | 3.62 | 0.04 | 6.37 | 1.76 | 85.84 | 0 | 0 | 0 |
| 125 | DENSE | 4096 | 4096 | 1.00 | 7.43 | 0.06 | 8.30 | 1.12 | 87.98 | 0 | 0 | 4 |
| 127 | DENSE | 16384 | 4096 | 4.00 | 3.06 | 0.03 | 7.62 | 2.49 | 83.35 | 0 | 0 | 0 |
| 128 | DENSE | 16384 | 4096 | 4.00 | 3.82 | 0.03 | 6.83 | 1.79 | 89.59 | 0 | 0 | 0 |
| 130 | DENSE | 12288 | 4096 | 3.00 | 3.64 | 0.03 | 6.34 | 1.74 | 87.79 | 0 | 0 | 0 |
| 131 | DENSE | 4096 | 4096 | 1.00 | 8.09 | 0.04 | 9.60 | 1.19 | 92.00 | 0 | 0 | 4 |
| 133 | DENSE | 16384 | 4096 | 4.00 | 3.10 | 0.02 | 7.73 | 2.49 | 83.67 | 0 | 0 | 0 |
| 134 | DENSE | 16384 | 4096 | 4.00 | 4.28 | 0.03 | 7.93 | 1.85 | 89.70 | 0 | 0 | 0 |
| 136 | DENSE | 12288 | 4096 | 3.00 | 3.68 | 0.02 | 6.29 | 1.71 | 88.25 | 0 | 0 | 0 |
| 137 | DENSE | 4096 | 4096 | 1.00 | 5.40 | 0.02 | 7.24 | 1.34 | 90.19 | 0 | 0 | 4 |
| 139 | DENSE | 16384 | 4096 | 4.00 | 3.17 | 0.02 | 7.92 | 2.50 | 84.15 | 0 | 0 | 0 |
| 140 | DENSE | 16384 | 4096 | 4.00 | 4.64 | 0.03 | 8.61 | 1.85 | 90.05 | 0 | 0 | 0 |
| 142 | DENSE | 12288 | 4096 | 3.00 | 4.07 | 0.02 | 6.86 | 1.68 | 89.95 | 0 | 0 | 0 |
| 143 | DENSE | 4096 | 4096 | 1.00 | 5.61 | 0.04 | 7.69 | 1.37 | 91.42 | 0 | 0 | 3 |
| 145 | DENSE | 16384 | 4096 | 4.00 | 3.14 | 0.01 | 7.85 | 2.50 | 84.10 | 0 | 0 | 0 |
| 146 | DENSE | 16384 | 4096 | 4.00 | 5.00 | 0.03 | 9.02 | 1.80 | 91.38 | 1 | 0 | 0 |
| 148 | DENSE | 12288 | 4096 | 3.00 | 4.15 | 0.02 | 7.10 | 1.71 | 89.36 | 0 | 0 | 0 |
| 149 | DENSE | 4096 | 4096 | 1.00 | 4.84 | 0.03 | 7.71 | 1.59 | 89.14 | 0 | 0 | 3 |
| 151 | DENSE | 16384 | 4096 | 4.00 | 3.11 | 0.01 | 7.80 | 2.51 | 83.95 | 0 | 0 | 0 |
| 152 | DENSE | 16384 | 4096 | 4.00 | 4.52 | 0.04 | 7.91 | 1.75 | 92.17 | 0 | 1 | 0 |
| 154 | DENSE | 12288 | 4096 | 3.00 | 4.27 | 0.02 | 7.44 | 1.74 | 88.92 | 0 | 0 | 0 |
| 155 | DENSE | 4096 | 4096 | 1.00 | 3.43 | 0.07 | 6.65 | 1.94 | 85.85 | 14 | 0 | 4 |
| 157 | DENSE | 16384 | 4096 | 4.00 | 3.08 | 0.01 | 7.76 | 2.52 | 83.65 | 0 | 0 | 0 |
| 158 | DENSE | 16384 | 4096 | 4.00 | 4.73 | 0.01 | 8.20 | 1.73 | 92.84 | 1 | 0 | 0 |
| 160 | DENSE | 12288 | 4096 | 3.00 | 3.67 | 0.03 | 6.45 | 1.76 | 88.09 | 0 | 0 | 0 |
| 161 | DENSE | 4096 | 4096 | 1.00 | 3.26 | 0.06 | 7.12 | 2.18 | 80.87 | 21 | 0 | 4 |
| 163 | DENSE | 16384 | 4096 | 4.00 | 3.08 | 0.01 | 7.83 | 2.54 | 83.80 | 0 | 0 | 0 |
| 164 | DENSE | 16384 | 4096 | 4.00 | 4.30 | 0.02 | 8.15 | 1.89 | 92.06 | 2 | 0 | 0 |
| 166 | DENSE | 12288 | 4096 | 3.00 | 3.63 | 0.01 | 6.65 | 1.83 | 87.77 | 0 | 0 | 0 |
| 167 | DENSE | 4096 | 4096 | 1.00 | 2.02 | 0.08 | 5.16 | 2.56 | 75.15 | 49 | 0 | 2 |
| 169 | DENSE | 16384 | 4096 | 4.00 | 3.08 | 0.01 | 7.98 | 2.59 | 83.75 | 0 | 0 | 0 |
| 170 | DENSE | 16384 | 4096 | 4.00 | 4.11 | 0.03 | 11.72 | 2.85 | 86.90 | 3 | 0 | 0 |
| 172 | DENSE | 12288 | 4096 | 3.00 | 3.50 | 0.01 | 7.27 | 2.08 | 86.72 | 0 | 0 | 0 |
| 173 | DENSE | 4096 | 4096 | 1.00 | 1.99 | 0.07 | 5.03 | 2.53 | 70.15 | 63 | 0 | 4 |
| 175 | DENSE | 16384 | 4096 | 4.00 | 3.14 | 0.01 | 8.11 | 2.58 | 84.37 | 0 | 0 | 0 |
| 176 | DENSE | 16384 | 4096 | 4.00 | 4.26 | 0.02 | 10.91 | 2.56 | 88.91 | 5 | 0 | 0 |
| 178 | DENSE | 12288 | 4096 | 3.00 | 3.32 | 0.01 | 6.73 | 2.03 | 85.98 | 0 | 0 | 0 |
| 179 | DENSE | 4096 | 4096 | 1.00 | 2.79 | 0.04 | 7.58 | 2.72 | 70.58 | 12 | 0 | 3 |
| 181 | DENSE | 16384 | 4096 | 4.00 | 3.34 | 0.02 | 8.47 | 2.54 | 85.25 | 0 | 0 | 0 |
| 182 | DENSE | 16384 | 4096 | 4.00 | 4.30 | 0.03 | 11.51 | 2.68 | 87.88 | 1 | 0 | 0 |
bigscience-bloom-7b1 Layer Plots
Layer 4
Layer=4 | N=12288 | M=4096 | Q=3.00 | alpha=3.07 | D_ks=0.02 | alpha-hat=6.29 | num traps=0
Layer 5
Layer=5 | N=4096 | M=4096 | Q=1.00 | alpha=3.11 | D_ks=0.02 | alpha-hat=6.23 | num traps=4
Layer 7
Layer=7 | N=16384 | M=4096 | Q=4.00 | alpha=2.37 | D_ks=0.05 | alpha-hat=6.06 | num traps=1
Layer 8
Layer=8 | N=16384 | M=4096 | Q=4.00 | alpha=5.91 | D_ks=0.05 | alpha-hat=14.97 | num traps=5
Layer 10
Layer=10 | N=12288 | M=4096 | Q=3.00 | alpha=3.51 | D_ks=0.03 | alpha-hat=7.16 | num traps=0
Layer 11
Layer=11 | N=4096 | M=4096 | Q=1.00 | alpha=3.51 | D_ks=0.03 | alpha-hat=7.73 | num traps=21
Layer 13
Layer=13 | N=16384 | M=4096 | Q=4.00 | alpha=3.38 | D_ks=0.06 | alpha-hat=8.62 | num traps=1
Layer 14
Layer=14 | N=16384 | M=4096 | Q=4.00 | alpha=4.65 | D_ks=0.04 | alpha-hat=11.27 | num traps=4
Layer 16
Layer=16 | N=12288 | M=4096 | Q=3.00 | alpha=4.24 | D_ks=0.01 | alpha-hat=7.85 | num traps=0
Layer 17
Layer=17 | N=4096 | M=4096 | Q=1.00 | alpha=4.04 | D_ks=0.04 | alpha-hat=9.47 | num traps=36
Layer 19
Layer=19 | N=16384 | M=4096 | Q=4.00 | alpha=2.76 | D_ks=0.03 | alpha-hat=6.68 | num traps=1
Layer 20
Layer=20 | N=16384 | M=4096 | Q=4.00 | alpha=3.84 | D_ks=0.03 | alpha-hat=9.28 | num traps=2
Layer 22
Layer=22 | N=12288 | M=4096 | Q=3.00 | alpha=3.61 | D_ks=0.02 | alpha-hat=7.03 | num traps=0
Layer 23
Layer=23 | N=4096 | M=4096 | Q=1.00 | alpha=3.73 | D_ks=0.02 | alpha-hat=8.33 | num traps=25
Layer 25
Layer=25 | N=16384 | M=4096 | Q=4.00 | alpha=3.22 | D_ks=0.01 | alpha-hat=7.29 | num traps=0
Layer 26
Layer=26 | N=16384 | M=4096 | Q=4.00 | alpha=3.58 | D_ks=0.01 | alpha-hat=8.44 | num traps=3
Layer 28
Layer=28 | N=12288 | M=4096 | Q=3.00 | alpha=3.06 | D_ks=0.04 | alpha-hat=6.04 | num traps=0
Layer 29
Layer=29 | N=4096 | M=4096 | Q=1.00 | alpha=3.70 | D_ks=0.03 | alpha-hat=7.59 | num traps=14
Layer 31
Layer=31 | N=16384 | M=4096 | Q=4.00 | alpha=3.01 | D_ks=0.01 | alpha-hat=6.41 | num traps=0
Layer 32
Layer=32 | N=16384 | M=4096 | Q=4.00 | alpha=3.53 | D_ks=0.01 | alpha-hat=8.18 | num traps=3
Layer 34
Layer=34 | N=12288 | M=4096 | Q=3.00 | alpha=3.10 | D_ks=0.04 | alpha-hat=6.05 | num traps=0
Layer 35
Layer=35 | N=4096 | M=4096 | Q=1.00 | alpha=3.66 | D_ks=0.02 | alpha-hat=6.95 | num traps=3
Layer 37
Layer=37 | N=16384 | M=4096 | Q=4.00 | alpha=2.98 | D_ks=0.01 | alpha-hat=6.31 | num traps=0
Layer 38
Layer=38 | N=16384 | M=4096 | Q=4.00 | alpha=3.57 | D_ks=0.01 | alpha-hat=8.23 | num traps=0
Layer 40
Layer=40 | N=12288 | M=4096 | Q=3.00 | alpha=3.34 | D_ks=0.03 | alpha-hat=6.39 | num traps=0
Layer 41
Layer=41 | N=4096 | M=4096 | Q=1.00 | alpha=3.96 | D_ks=0.03 | alpha-hat=7.51 | num traps=0
Layer 43
Layer=43 | N=16384 | M=4096 | Q=4.00 | alpha=3.02 | D_ks=0.01 | alpha-hat=6.60 | num traps=0
Layer 44
Layer=44 | N=16384 | M=4096 | Q=4.00 | alpha=3.58 | D_ks=0.01 | alpha-hat=7.56 | num traps=3
Layer 46
Layer=46 | N=12288 | M=4096 | Q=3.00 | alpha=3.04 | D_ks=0.02 | alpha-hat=5.86 | num traps=0
Layer 47
Layer=47 | N=4096 | M=4096 | Q=1.00 | alpha=3.70 | D_ks=0.04 | alpha-hat=6.17 | num traps=0
Layer 49
Layer=49 | N=16384 | M=4096 | Q=4.00 | alpha=3.08 | D_ks=0.01 | alpha-hat=8.34 | num traps=0
Layer 50
Layer=50 | N=16384 | M=4096 | Q=4.00 | alpha=3.77 | D_ks=0.01 | alpha-hat=8.49 | num traps=16
Layer 52
Layer=52 | N=12288 | M=4096 | Q=3.00 | alpha=3.18 | D_ks=0.03 | alpha-hat=6.25 | num traps=0
Layer 53
Layer=53 | N=4096 | M=4096 | Q=1.00 | alpha=3.42 | D_ks=0.08 | alpha-hat=5.29 | num traps=0
Layer 55
Layer=55 | N=16384 | M=4096 | Q=4.00 | alpha=3.19 | D_ks=0.01 | alpha-hat=8.29 | num traps=0
Layer 56
Layer=56 | N=16384 | M=4096 | Q=4.00 | alpha=3.82 | D_ks=0.02 | alpha-hat=9.33 | num traps=22
Layer 58
Layer=58 | N=12288 | M=4096 | Q=3.00 | alpha=3.31 | D_ks=0.05 | alpha-hat=5.97 | num traps=0
Layer 59
Layer=59 | N=4096 | M=4096 | Q=1.00 | alpha=4.05 | D_ks=0.03 | alpha-hat=5.75 | num traps=0
Layer 61
Layer=61 | N=16384 | M=4096 | Q=4.00 | alpha=3.09 | D_ks=0.01 | alpha-hat=8.00 | num traps=0
Layer 62
Layer=62 | N=16384 | M=4096 | Q=4.00 | alpha=3.42 | D_ks=0.03 | alpha-hat=7.52 | num traps=14
Layer 64
Layer=64 | N=12288 | M=4096 | Q=3.00 | alpha=2.85 | D_ks=0.05 | alpha-hat=5.18 | num traps=0
Layer 65
Layer=65 | N=4096 | M=4096 | Q=1.00 | alpha=4.40 | D_ks=0.06 | alpha-hat=5.82 | num traps=0
Layer 67
Layer=67 | N=16384 | M=4096 | Q=4.00 | alpha=3.03 | D_ks=0.02 | alpha-hat=7.51 | num traps=1
Layer 68
Layer=68 | N=16384 | M=4096 | Q=4.00 | alpha=3.21 | D_ks=0.03 | alpha-hat=6.16 | num traps=8
Layer 70
Layer=70 | N=12288 | M=4096 | Q=3.00 | alpha=3.53 | D_ks=0.05 | alpha-hat=6.47 | num traps=0
Layer 71
Layer=71 | N=4096 | M=4096 | Q=1.00 | alpha=3.08 | D_ks=0.05 | alpha-hat=3.77 | num traps=0
Layer 73
Layer=73 | N=16384 | M=4096 | Q=4.00 | alpha=3.10 | D_ks=0.02 | alpha-hat=7.64 | num traps=1
Layer 74
Layer=74 | N=16384 | M=4096 | Q=4.00 | alpha=3.36 | D_ks=0.03 | alpha-hat=6.59 | num traps=2
Layer 76
Layer=76 | N=12288 | M=4096 | Q=3.00 | alpha=2.24 | D_ks=0.05 | alpha-hat=4.11 | num traps=0
Layer 77
Layer=77 | N=4096 | M=4096 | Q=1.00 | alpha=4.09 | D_ks=0.05 | alpha-hat=3.87 | num traps=0
Layer 79
Layer=79 | N=16384 | M=4096 | Q=4.00 | alpha=3.05 | D_ks=0.02 | alpha-hat=7.65 | num traps=1
Layer 80
Layer=80 | N=16384 | M=4096 | Q=4.00 | alpha=3.34 | D_ks=0.03 | alpha-hat=6.08 | num traps=1
Layer 82
Layer=82 | N=12288 | M=4096 | Q=3.00 | alpha=2.99 | D_ks=0.06 | alpha-hat=5.41 | num traps=0
Layer 83
Layer=83 | N=4096 | M=4096 | Q=1.00 | alpha=3.60 | D_ks=0.07 | alpha-hat=3.83 | num traps=0
Layer 85
Layer=85 | N=16384 | M=4096 | Q=4.00 | alpha=3.18 | D_ks=0.03 | alpha-hat=8.06 | num traps=1
Layer 86
Layer=86 | N=16384 | M=4096 | Q=4.00 | alpha=3.19 | D_ks=0.04 | alpha-hat=5.39 | num traps=0
Layer 88
Layer=88 | N=12288 | M=4096 | Q=3.00 | alpha=3.14 | D_ks=0.06 | alpha-hat=5.70 | num traps=0
Layer 89
Layer=89 | N=4096 | M=4096 | Q=1.00 | alpha=3.53 | D_ks=0.07 | alpha-hat=3.63 | num traps=0
Layer 91
Layer=91 | N=16384 | M=4096 | Q=4.00 | alpha=3.01 | D_ks=0.03 | alpha-hat=7.48 | num traps=1
Layer 92
Layer=92 | N=16384 | M=4096 | Q=4.00 | alpha=3.13 | D_ks=0.04 | alpha-hat=5.14 | num traps=0
Layer 94
Layer=94 | N=12288 | M=4096 | Q=3.00 | alpha=2.33 | D_ks=0.08 | alpha-hat=4.14 | num traps=0
Layer 95
Layer=95 | N=4096 | M=4096 | Q=1.00 | alpha=3.77 | D_ks=0.08 | alpha-hat=3.58 | num traps=0
Layer 97
Layer=97 | N=16384 | M=4096 | Q=4.00 | alpha=3.11 | D_ks=0.03 | alpha-hat=7.81 | num traps=0
Layer 98
Layer=98 | N=16384 | M=4096 | Q=4.00 | alpha=3.23 | D_ks=0.04 | alpha-hat=5.32 | num traps=0
Layer 100
Layer=100 | N=12288 | M=4096 | Q=3.00 | alpha=2.98 | D_ks=0.04 | alpha-hat=5.36 | num traps=0
Layer 101
Layer=101 | N=4096 | M=4096 | Q=1.00 | alpha=3.22 | D_ks=0.05 | alpha-hat=3.38 | num traps=0
Layer 103
Layer=103 | N=16384 | M=4096 | Q=4.00 | alpha=3.07 | D_ks=0.02 | alpha-hat=7.66 | num traps=0
Layer 104
Layer=104 | N=16384 | M=4096 | Q=4.00 | alpha=3.03 | D_ks=0.04 | alpha-hat=4.91 | num traps=0
Layer 106
Layer=106 | N=12288 | M=4096 | Q=3.00 | alpha=3.20 | D_ks=0.04 | alpha-hat=5.96 | num traps=0
Layer 107
Layer=107 | N=4096 | M=4096 | Q=1.00 | alpha=3.69 | D_ks=0.06 | alpha-hat=3.75 | num traps=0
Layer 109
Layer=109 | N=16384 | M=4096 | Q=4.00 | alpha=2.91 | D_ks=0.02 | alpha-hat=7.32 | num traps=0
Layer 110
Layer=110 | N=16384 | M=4096 | Q=4.00 | alpha=3.52 | D_ks=0.04 | alpha-hat=5.74 | num traps=0
Layer 112
Layer=112 | N=12288 | M=4096 | Q=3.00 | alpha=3.14 | D_ks=0.04 | alpha-hat=5.76 | num traps=0
Layer 113
Layer=113 | N=4096 | M=4096 | Q=1.00 | alpha=3.08 | D_ks=0.04 | alpha-hat=3.16 | num traps=0
Layer 115
Layer=115 | N=16384 | M=4096 | Q=4.00 | alpha=2.97 | D_ks=0.02 | alpha-hat=7.42 | num traps=0
Layer 116
Layer=116 | N=16384 | M=4096 | Q=4.00 | alpha=3.38 | D_ks=0.03 | alpha-hat=5.83 | num traps=0
Layer 118
Layer=118 | N=12288 | M=4096 | Q=3.00 | alpha=3.49 | D_ks=0.04 | alpha-hat=6.01 | num traps=0
Layer 119
Layer=119 | N=4096 | M=4096 | Q=1.00 | alpha=3.15 | D_ks=0.07 | alpha-hat=3.15 | num traps=0
Layer 121
Layer=121 | N=16384 | M=4096 | Q=4.00 | alpha=3.05 | D_ks=0.02 | alpha-hat=7.58 | num traps=0
Layer 122
Layer=122 | N=16384 | M=4096 | Q=4.00 | alpha=3.52 | D_ks=0.03 | alpha-hat=6.14 | num traps=0
Layer 124
Layer=124 | N=12288 | M=4096 | Q=3.00 | alpha=3.62 | D_ks=0.04 | alpha-hat=6.37 | num traps=0
Layer 125
Layer=125 | N=4096 | M=4096 | Q=1.00 | alpha=7.43 | D_ks=0.06 | alpha-hat=8.30 | num traps=0
Layer 127
Layer=127 | N=16384 | M=4096 | Q=4.00 | alpha=3.06 | D_ks=0.03 | alpha-hat=7.62 | num traps=0
Layer 128
Layer=128 | N=16384 | M=4096 | Q=4.00 | alpha=3.82 | D_ks=0.03 | alpha-hat=6.83 | num traps=0
Layer 130
Layer=130 | N=12288 | M=4096 | Q=3.00 | alpha=3.64 | D_ks=0.03 | alpha-hat=6.34 | num traps=0
Layer 131
Layer=131 | N=4096 | M=4096 | Q=1.00 | alpha=8.09 | D_ks=0.04 | alpha-hat=9.60 | num traps=0
Layer 133
Layer=133 | N=16384 | M=4096 | Q=4.00 | alpha=3.10 | D_ks=0.02 | alpha-hat=7.73 | num traps=0
Layer 134
Layer=134 | N=16384 | M=4096 | Q=4.00 | alpha=4.28 | D_ks=0.03 | alpha-hat=7.93 | num traps=0
Layer 136
Layer=136 | N=12288 | M=4096 | Q=3.00 | alpha=3.68 | D_ks=0.02 | alpha-hat=6.29 | num traps=0
Layer 137
Layer=137 | N=4096 | M=4096 | Q=1.00 | alpha=5.40 | D_ks=0.02 | alpha-hat=7.24 | num traps=0
Layer 139
Layer=139 | N=16384 | M=4096 | Q=4.00 | alpha=3.17 | D_ks=0.02 | alpha-hat=7.92 | num traps=0
Layer 140
Layer=140 | N=16384 | M=4096 | Q=4.00 | alpha=4.64 | D_ks=0.03 | alpha-hat=8.61 | num traps=0
Layer 142
Layer=142 | N=12288 | M=4096 | Q=3.00 | alpha=4.07 | D_ks=0.02 | alpha-hat=6.86 | num traps=0
Layer 143
Layer=143 | N=4096 | M=4096 | Q=1.00 | alpha=5.61 | D_ks=0.04 | alpha-hat=7.69 | num traps=0
Layer 145
Layer=145 | N=16384 | M=4096 | Q=4.00 | alpha=3.14 | D_ks=0.01 | alpha-hat=7.85 | num traps=0
Layer 146
Layer=146 | N=16384 | M=4096 | Q=4.00 | alpha=5.00 | D_ks=0.03 | alpha-hat=9.02 | num traps=1
Layer 148
Layer=148 | N=12288 | M=4096 | Q=3.00 | alpha=4.15 | D_ks=0.02 | alpha-hat=7.10 | num traps=0
Layer 149
Layer=149 | N=4096 | M=4096 | Q=1.00 | alpha=4.84 | D_ks=0.03 | alpha-hat=7.71 | num traps=0
Layer 151
Layer=151 | N=16384 | M=4096 | Q=4.00 | alpha=3.11 | D_ks=0.01 | alpha-hat=7.80 | num traps=0
Layer 152
Layer=152 | N=16384 | M=4096 | Q=4.00 | alpha=4.52 | D_ks=0.04 | alpha-hat=7.91 | num traps=0
Layer 154
Layer=154 | N=12288 | M=4096 | Q=3.00 | alpha=4.27 | D_ks=0.02 | alpha-hat=7.44 | num traps=0
Layer 155
Layer=155 | N=4096 | M=4096 | Q=1.00 | alpha=3.43 | D_ks=0.07 | alpha-hat=6.65 | num traps=14
Layer 157
Layer=157 | N=16384 | M=4096 | Q=4.00 | alpha=3.08 | D_ks=0.01 | alpha-hat=7.76 | num traps=0
Layer 158
Layer=158 | N=16384 | M=4096 | Q=4.00 | alpha=4.73 | D_ks=0.01 | alpha-hat=8.20 | num traps=1
Layer 160
Layer=160 | N=12288 | M=4096 | Q=3.00 | alpha=3.67 | D_ks=0.03 | alpha-hat=6.45 | num traps=0
Layer 161
Layer=161 | N=4096 | M=4096 | Q=1.00 | alpha=3.26 | D_ks=0.06 | alpha-hat=7.12 | num traps=21
Layer 163
Layer=163 | N=16384 | M=4096 | Q=4.00 | alpha=3.08 | D_ks=0.01 | alpha-hat=7.83 | num traps=0
Layer 164
Layer=164 | N=16384 | M=4096 | Q=4.00 | alpha=4.30 | D_ks=0.02 | alpha-hat=8.15 | num traps=2
Layer 166
Layer=166 | N=12288 | M=4096 | Q=3.00 | alpha=3.63 | D_ks=0.01 | alpha-hat=6.65 | num traps=0
Layer 167
Layer=167 | N=4096 | M=4096 | Q=1.00 | alpha=2.02 | D_ks=0.08 | alpha-hat=5.16 | num traps=49
Layer 169
Layer=169 | N=16384 | M=4096 | Q=4.00 | alpha=3.08 | D_ks=0.01 | alpha-hat=7.98 | num traps=0
Layer 170
Layer=170 | N=16384 | M=4096 | Q=4.00 | alpha=4.11 | D_ks=0.03 | alpha-hat=11.72 | num traps=3
Layer 172
Layer=172 | N=12288 | M=4096 | Q=3.00 | alpha=3.50 | D_ks=0.01 | alpha-hat=7.27 | num traps=0
Layer 173
Layer=173 | N=4096 | M=4096 | Q=1.00 | alpha=1.99 | D_ks=0.07 | alpha-hat=5.03 | num traps=63
Layer 175
Layer=175 | N=16384 | M=4096 | Q=4.00 | alpha=3.14 | D_ks=0.01 | alpha-hat=8.11 | num traps=0
Layer 176
Layer=176 | N=16384 | M=4096 | Q=4.00 | alpha=4.26 | D_ks=0.02 | alpha-hat=10.91 | num traps=5
Layer 178
Layer=178 | N=12288 | M=4096 | Q=3.00 | alpha=3.32 | D_ks=0.01 | alpha-hat=6.73 | num traps=0
Layer 179
Layer=179 | N=4096 | M=4096 | Q=1.00 | alpha=2.79 | D_ks=0.04 | alpha-hat=7.58 | num traps=12
Layer 181
Layer=181 | N=16384 | M=4096 | Q=4.00 | alpha=3.34 | D_ks=0.02 | alpha-hat=8.47 | num traps=0
Layer 182
Layer=182 | N=16384 | M=4096 | Q=4.00 | alpha=4.30 | D_ks=0.03 | alpha-hat=11.51 | num traps=1