diff --git a/spectral_scheduling.md b/spectral_scheduling.md new file mode 100644 index 000000000..5ac180ea9 --- /dev/null +++ b/spectral_scheduling.md @@ -0,0 +1,8 @@ +# Spectral Scheduling + +labels: experimental + +1. learn a singular basis over the data +2. treat the eigenvalues as relative sampling weight, sample a basis dimension/component each batch +3. sample a batch of data, using that datum's value in the sampled component as their relative likelihood (threshold this) +4. incorporate a temperature term so we can sample disproportionately from the important components in early training, progressively increasing the weight of the tail probabilities.