Distillation — katta va kuchli modelning bilimini kichikroq modelga o‘tkazish uchun teacher-student trainingidan foydalanadigan model compression usuli. To‘liq nomi Knowledge Distillation. Teacher model yumshoq probability, hidden representation yoki generated output orqali student modelga qo‘shimcha signal beradi.
Maqsad student modelni faqat hard label bilan o‘qitishdan ko‘ra teacher behavioriga yaqinlashtirishdir.
Teacher model
Teacher odatda:
- kattaroq;
- chuqurroq;
- ensemble;
- qimmat inference;
- yuqori sifatli
model bo‘ladi.
Teacher training paytida fixed qolishi mumkin.
Ba’zi usullarda teacher va student birgalikda yangilanadi.
Student model
Student deployment uchun mos kichik model.
U:
maqsadida tanlanadi.
Student architecture teacher bilan aynan bir xil bo‘lishi shart emas.
Hard label
Oddiy classification trainingida label:
mushuk = 1
it = 0
kabi one-hot ko‘rinishda bo‘ladi.
Bu label boshqa classlar orasidagi o‘xshashlikni ko‘rsatmaydi.
Teacher probabilitylari boyroq signal beradi.
Soft target
Teacher outputi:
mushuk 0.70
yo‘lbars 0.20
it 0.08
boshqa 0.02
ko‘rinishida bo‘lishi mumkin.
Student faqat “mushuk” labelini emas, mushuk va yo‘lbars yaqinligini ham o‘rganadi.
Bu dark knowledge deb ataladigan qo‘shimcha ma’lumotga yaqin tushuncha.
Temperature
Softmax temperature probability distributionni yumshatadi.
Yuqori temperature classlar orasidagi kichik farqlarni ko‘rinadigan qiladi.
Teacher va student logitslari bir xil temperature bilan solishtiriladi.
Inference’da odatiy temperature ishlatiladi.
Distillation loss
Student loss odatda ikki qismdan tuziladi:
- hard label loss;
- teacher soft target loss.
Og‘irliklar taskga qarab tanlanadi.
Faqat teacherga taqlid qilish teacher xatolarini ham studentga o‘tkazishi mumkin.
Logit distillation
Student teacherning final logits yoki probabilitylarini taqlid qiladi.
Bu eng sodda va keng tarqalgan yondashuv.
Teacher va student output classlari mos bo‘lishi kerak.
Generativ modelda token distribution distill qilinishi mumkin.
Feature distillation
Student teacherning intermediate layer representationlariga yaqinlashadi.
Layer dimensionlari farq qilsa projection ishlatiladi.
Bu faqat final outputdan ko‘ra ko‘proq ichki signal beradi.
Qaysi layerlarni moslashtirish architecture’ga bog‘liq.
Attention distillation
Transformer student teacher attention maplari yoki relationshiplarini taqlid qilishi mumkin.
Barcha headlarni aynan birma-bir moslashtirish shart emas.
Attention distribution, token relation yoki aggregated map distill qilinadi.
Sequence-level distillation
Translation yoki generative taskda teacher avval output sequence yaratadi.
Student original human label o‘rniga yoki u bilan birga teacher generated sequence’da o‘qitiladi.
Teacher outputi training targetni soddalashtirishi mumkin.
Biroq hallucination va uslub studentga ko‘chadi.
Offline distillation
Teacher oldindan barcha training sample uchun output yaratadi.
Natijalar datasetga saqlanadi.
Afzalligi — teacher har epochda qayta ishlamaydi.
Kamchiligi — storage katta va augmentation o‘zgarsa teacher output yangilanmaydi.
Online distillation
Teacher training vaqtida real-time output beradi.
Bu dynamic augmentation va yangi sample bilan mos.
Hisoblash qimmatroq, chunki teacher va student forward pass bajaradi.
Self-distillation
Bir modelning oldingi checkpointi, chuqur layeri yoki ensemble’si teacher bo‘lishi mumkin.
Student architecture teacher bilan bir xil ham bo‘lishi mumkin.
Self-distillation ba’zan regularization va generalizationni yaxshilaydi.
Data-free distillation
Original training data mavjud bo‘lmasa synthetic yoki generated input orqali teacher behaviori olinadi.
Bu privacy, licensing yoki data yo‘qolishi holatlarida qiziq.
Generated input real distributionni yetarli qamramasa student sifati past bo‘ladi.
LLM distillation
Katta til modeli kichik model uchun:
- instruction-response;
- reasoningga oid synthetic data;
- preference;
- tool call;
- structured output
yaratishi mumkin.
Student supervised fine-tuning orqali bu outputlardan o‘rganadi.
Synthetic data filtrlanadi va teacher xatolari tekshiriladi.
Distillation va fine-tuning
Fine-tuning pretrained modelni yangi taskga moslashtiradi.
Distillation teacher behaviorini studentga o‘tkazadi.
Student bir vaqtning o‘zida task dataset va teacher outputida fine-tuning qilinishi mumkin.
Ikki jarayon bir-birini to‘ldiradi.
Distillation va quantization
Distillation model parameter sonini yoki architecture hajmini kamaytiradi.
Quantization har parameterning bit aniqligini kamaytiradi.
Kichik distilled model keyin quantize qilinishi mumkin.
Bu edge deploymentda katta tejam beradi.
Cheklovlar
Student capacity juda kichik bo‘lsa teacher qobiliyatini to‘liq o‘rgana olmaydi.
Teacher:
- bias;
- xato;
- calibration muammosi;
- zararli behavior
ni ham o‘tkazishi mumkin.
Student teacherdan yaxshiroq bo‘lishi mumkin, ammo bu avtomatik emas.
Evaluation
Student teacher va baseline bilan solishtiriladi:
Compression darajasi bilan sifat yo‘qotilishi birga ko‘rsatiladi.
Intermediate matching
Teacher va student layer soni farqli bo‘lsa qaysi qatlamlar moslashtirilishi mapping bilan belgilanadi. Masalan, teacherning har ikkinchi layeri studentning navbatdagi layeriga teacher signalini beradi. Representation dimensioni boshqa bo‘lsa trainable projection qo‘shiladi. Barcha layerni majburan moslashtirish studentning o‘z architecture afzalligini cheklashi mumkin.
Calibration
Distilled student probabilitylari teacherga qaraganda ortiqcha ishonchli yoki ehtiyotkor bo‘lishi mumkin. Calibration setda confidence va real aniqlik solishtiriladi. Classification threshold, reject option va downstream risk qarori student uchun qayta tanlanadi.
Deployment
Studentning haqiqiy foydasi target qurilmada o‘lchanadi. Parameter soni kichik bo‘lsa ham unsupported operator, yomon memory layout yoki kichik batch sabab latency kutilganidek kamaymasligi mumkin.
Bog‘liq tushunchalar
Knowledge Distillation, Teacher model, Student model, Soft target, Temperature, Logit distillation, Feature distillation, Model compression, Quantization, Fine-tuning