You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+15-12Lines changed: 15 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -97,6 +97,19 @@ watcher.get_ESD()
97
97
watcher.distances(model_1, model_2)
98
98
```
99
99
100
+
## PEFT / LORA models
101
+
102
+
To analyze an PEFT / LORA fine-tuned model, specify the peft option.
103
+
104
+
- peft = True: Analyes the base_model, the delta, and the combined layer weight matrices
105
+
- peft = 'lora_only': Musch faster, only analyzes the delta
106
+
107
+
i.e.
108
+
```
109
+
details = watcher.analyze(peft=True)
110
+
```
111
+
112
+
100
113
## Ploting and Fitting the Empirical Spectral Density (ESD)
101
114
102
115
WW creates plots for each layer weight matrix to observe how well the power law fits work
@@ -147,17 +160,6 @@ All of these attempt to measure how on-random and/or non-heavy-tailed the layer
147
160
- (Truncated) PL quality of fit `D` : <imgsrc="https://render.githubusercontent.com/render/math?math=\D"> (the Kolmogorov Smirnov Distance metric)
148
161
149
162
150
-
#### PEFT / LORA models
151
-
152
-
To analyze an PEFT / LORA fine-tuned model, specify the peft option.
153
-
154
-
- peft = True: Analyes the base_model, the delta, and the combined layer weight matrices
155
-
- peft = 'lora_only': Musch faster, only analyzes the delta
156
-
157
-
i.e.
158
-
```
159
-
details = watcher.analyze(peft=True)
160
-
```
161
163
162
164
163
165
(advanced usage)
@@ -210,7 +212,8 @@ The summary statistics can be used to gauge the test error of a series of pre/tr
210
212
- average `alpha` can be used to compare one or more DNN models with different hyperparemeter settings **θ**, when depth is not a driving factor (i.e transformer models)
211
213
- average `log_spectral_norm` is useful to compare models of different depths **L** at a coarse grain level
212
214
- average `alpha_weighted` and `log_alpha_norm` are suitable for DNNs of differing hyperparemeters **θ** and depths **L** simultaneously. (i.e CV models like VGG and ResNet)
0 commit comments