RedPajama-Chat-3b-v1


Find this model in the RedPajama model summary

RedPajama-Chat-3b-v1 Model Summary Plots

RedPajama-Chat-3b-v1 Model Selected Details
  layer_type N M Q alpha D alpha-hat log_SN rank_loss
layer_id                  
4 DENSE 7680 2560 3.00 2.11 0.04 6.75 3.20 0
5 DENSE 2560 2560 1.00 3.42 0.05 4.17 1.22 7
6 DENSE 10240 2560 4.00 5.12 0.07 11.15 2.18 0
7 DENSE 10240 2560 4.00 7.96 0.06 14.75 1.85 0
10 DENSE 7680 2560 3.00 3.70 0.02 7.60 2.05 0
11 DENSE 2560 2560 1.00 4.83 0.02 7.00 1.45 2
12 DENSE 10240 2560 4.00 3.07 0.02 6.85 2.23 0
13 DENSE 10240 2560 4.00 4.05 0.04 6.20 1.53 0
16 DENSE 7680 2560 3.00 2.53 0.04 4.84 1.91 0
17 DENSE 2560 2560 1.00 4.03 0.03 6.63 1.64 3
18 DENSE 10240 2560 4.00 3.21 0.01 8.42 2.63 0
19 DENSE 10240 2560 4.00 3.80 0.01 7.56 1.99 0
22 DENSE 7680 2560 3.00 3.28 0.02 5.93 1.81 0
23 DENSE 2560 2560 1.00 3.96 0.04 6.77 1.71 4
24 DENSE 10240 2560 4.00 3.40 0.02 7.27 2.14 0
25 DENSE 10240 2560 4.00 4.07 0.01 7.03 1.73 0
28 DENSE 7680 2560 3.00 3.54 0.01 6.76 1.91 0
29 DENSE 2560 2560 1.00 3.62 0.04 5.70 1.57 2
30 DENSE 10240 2560 4.00 3.36 0.02 7.25 2.16 0
31 DENSE 10240 2560 4.00 3.96 0.02 6.84 1.73 0
34 DENSE 7680 2560 3.00 3.57 0.01 6.92 1.94 0
35 DENSE 2560 2560 1.00 3.69 0.04 5.82 1.58 3
36 DENSE 10240 2560 4.00 3.47 0.02 7.46 2.15 0
37 DENSE 10240 2560 4.00 3.94 0.02 6.30 1.60 0
40 DENSE 7680 2560 3.00 3.80 0.02 7.28 1.92 0
41 DENSE 2560 2560 1.00 5.01 0.04 7.88 1.57 2
42 DENSE 10240 2560 4.00 3.45 0.01 7.43 2.16 0
43 DENSE 10240 2560 4.00 3.95 0.04 6.26 1.58 0
46 DENSE 7680 2560 3.00 3.60 0.01 6.85 1.90 0
47 DENSE 2560 2560 1.00 4.47 0.04 6.94 1.55 3
48 DENSE 10240 2560 4.00 3.42 0.02 7.36 2.15 0
49 DENSE 10240 2560 4.00 3.74 0.04 6.11 1.64 0
52 DENSE 7680 2560 3.00 3.65 0.02 6.98 1.91 0
53 DENSE 2560 2560 1.00 4.64 0.04 7.73 1.67 3
54 DENSE 10240 2560 4.00 3.42 0.03 7.34 2.15 0
55 DENSE 10240 2560 4.00 3.88 0.04 6.33 1.63 0
58 DENSE 7680 2560 3.00 3.42 0.02 6.44 1.88 0
59 DENSE 2560 2560 1.00 3.79 0.04 5.89 1.55 2
60 DENSE 10240 2560 4.00 3.43 0.03 7.32 2.13 0
61 DENSE 10240 2560 4.00 3.93 0.04 6.64 1.69 0
64 DENSE 7680 2560 3.00 3.30 0.02 6.14 1.86 0
65 DENSE 2560 2560 1.00 4.05 0.03 6.52 1.61 2
66 DENSE 10240 2560 4.00 3.43 0.03 7.25 2.11 0
67 DENSE 10240 2560 4.00 5.76 0.04 9.30 1.62 0
70 DENSE 7680 2560 3.00 3.37 0.02 6.31 1.87 0
71 DENSE 2560 2560 1.00 4.30 0.05 6.91 1.61 3
72 DENSE 10240 2560 4.00 3.41 0.03 7.11 2.08 0
73 DENSE 10240 2560 4.00 3.73 0.04 6.06 1.62 0
76 DENSE 7680 2560 3.00 3.21 0.02 6.11 1.90 0
77 DENSE 2560 2560 1.00 4.51 0.05 6.86 1.52 3
78 DENSE 10240 2560 4.00 3.87 0.03 8.18 2.11 0
79 DENSE 10240 2560 4.00 3.67 0.04 6.15 1.68 0
82 DENSE 7680 2560 3.00 3.24 0.03 6.03 1.86 0
83 DENSE 2560 2560 1.00 4.42 0.04 7.27 1.64 2
84 DENSE 10240 2560 4.00 3.93 0.03 8.17 2.08 0
85 DENSE 10240 2560 4.00 3.75 0.03 6.24 1.66 0
88 DENSE 7680 2560 3.00 3.07 0.03 5.77 1.88 0
89 DENSE 2560 2560 1.00 4.97 0.06 8.07 1.63 4
90 DENSE 10240 2560 4.00 3.54 0.04 7.32 2.07 0
91 DENSE 10240 2560 4.00 3.98 0.03 6.58 1.65 0
94 DENSE 7680 2560 3.00 3.14 0.02 5.83 1.86 0
95 DENSE 2560 2560 1.00 4.66 0.06 7.32 1.57 2
96 DENSE 10240 2560 4.00 3.80 0.03 7.94 2.09 0
97 DENSE 10240 2560 4.00 4.87 0.04 8.14 1.67 0
100 DENSE 7680 2560 3.00 3.16 0.01 5.83 1.85 0
101 DENSE 2560 2560 1.00 3.76 0.04 6.29 1.67 3
102 DENSE 10240 2560 4.00 3.74 0.03 7.90 2.11 0
103 DENSE 10240 2560 4.00 4.36 0.03 6.96 1.60 0
106 DENSE 7680 2560 3.00 3.49 0.02 6.45 1.85 0
107 DENSE 2560 2560 1.00 3.88 0.06 6.40 1.65 3
108 DENSE 10240 2560 4.00 4.46 0.03 9.39 2.11 0
109 DENSE 10240 2560 4.00 4.63 0.02 7.79 1.68 0
112 DENSE 7680 2560 3.00 3.54 0.01 6.70 1.89 0
113 DENSE 2560 2560 1.00 5.26 0.04 9.09 1.73 3
114 DENSE 10240 2560 4.00 4.37 0.03 9.26 2.12 0
115 DENSE 10240 2560 4.00 4.90 0.01 8.35 1.70 0
118 DENSE 7680 2560 3.00 3.69 0.01 7.00 1.90 0
119 DENSE 2560 2560 1.00 4.86 0.05 8.03 1.65 2
120 DENSE 10240 2560 4.00 4.00 0.03 8.60 2.15 0
121 DENSE 10240 2560 4.00 5.28 0.01 8.83 1.67 0
124 DENSE 7680 2560 3.00 4.08 0.03 8.03 1.97 0
125 DENSE 2560 2560 1.00 6.43 0.07 11.12 1.73 2
126 DENSE 10240 2560 4.00 3.60 0.02 7.92 2.20 0
127 DENSE 10240 2560 4.00 6.55 0.02 10.53 1.61 0
130 DENSE 7680 2560 3.00 4.27 0.04 8.31 1.94 0
131 DENSE 2560 2560 1.00 6.44 0.07 10.21 1.59 2
132 DENSE 10240 2560 4.00 3.99 0.03 8.77 2.20 0
133 DENSE 10240 2560 4.00 6.79 0.02 10.14 1.49 0
136 DENSE 7680 2560 3.00 3.88 0.03 7.66 1.97 0
137 DENSE 2560 2560 1.00 6.87 0.06 10.98 1.60 1
138 DENSE 10240 2560 4.00 4.06 0.04 8.94 2.20 0
139 DENSE 10240 2560 4.00 7.03 0.02 9.65 1.37 0
142 DENSE 7680 2560 3.00 4.23 0.03 8.46 2.00 0
143 DENSE 2560 2560 1.00 7.12 0.06 10.30 1.45 2
144 DENSE 10240 2560 4.00 4.04 0.04 8.82 2.18 0
145 DENSE 10240 2560 4.00 7.05 0.03 10.36 1.47 0
148 DENSE 7680 2560 3.00 3.15 0.01 6.31 2.00 0
149 DENSE 2560 2560 1.00 7.62 0.08 10.86 1.43 2
150 DENSE 10240 2560 4.00 3.87 0.04 8.48 2.19 0
151 DENSE 10240 2560 4.00 6.60 0.04 9.37 1.42 0
154 DENSE 7680 2560 3.00 3.75 0.01 7.51 2.00 0
155 DENSE 2560 2560 1.00 6.43 0.04 9.46 1.47 2
156 DENSE 10240 2560 4.00 4.08 0.04 8.92 2.19 0
157 DENSE 10240 2560 4.00 7.80 0.02 10.15 1.30 0
160 DENSE 7680 2560 3.00 3.87 0.01 7.69 1.99 0
161 DENSE 2560 2560 1.00 7.42 0.03 9.38 1.26 2
162 DENSE 10240 2560 4.00 4.17 0.04 9.07 2.17 0
163 DENSE 10240 2560 4.00 7.67 0.02 10.29 1.34 0
166 DENSE 7680 2560 3.00 3.65 0.02 7.33 2.01 0
167 DENSE 2560 2560 1.00 6.65 0.02 9.30 1.40 2
168 DENSE 10240 2560 4.00 3.99 0.02 8.69 2.18 0
169 DENSE 10240 2560 4.00 8.42 0.02 12.05 1.43 0
172 DENSE 7680 2560 3.00 3.94 0.02 7.88 2.00 0
173 DENSE 2560 2560 1.00 5.79 0.02 7.39 1.28 2
174 DENSE 10240 2560 4.00 4.20 0.02 9.06 2.16 0
175 DENSE 10240 2560 4.00 6.96 0.03 12.16 1.75 0
178 DENSE 7680 2560 3.00 3.37 0.02 6.72 2.00 0
179 DENSE 2560 2560 1.00 7.40 0.05 9.36 1.27 2
180 DENSE 10240 2560 4.00 4.45 0.01 9.42 2.12 0
181 DENSE 10240 2560 4.00 5.96 0.03 12.78 2.15 0
184 DENSE 7680 2560 3.00 3.46 0.03 10.96 3.17 0
185 DENSE 2560 2560 1.00 4.99 0.03 8.29 1.66 2
186 DENSE 10240 2560 4.00 4.32 0.02 8.97 2.08 0
187 DENSE 10240 2560 4.00 4.94 0.02 10.66 2.16 0
190 DENSE 7680 2560 3.00 2.70 0.05 8.53 3.16 0
191 DENSE 2560 2560 1.00 3.91 0.06 8.56 2.19 2
192 DENSE 10240 2560 4.00 3.16 0.03 6.63 2.10 0
193 DENSE 10240 2560 4.00 4.02 0.04 8.67 2.16 0