Reinforcement — agent environment bilan interaction qilib, actionlari uchun reward olib, uzoq muddatli umumiy mukofotni oshiradigan policy’ni o‘rganish yondashuvi. To‘liq soha nomi Reinforcement Learning. U o‘yin, robototexnika, control, recommendation, resource allocation va alignmentda ishlatiladi.
Supervised learning tayyor to‘g‘ri labeldan o‘rganadi. Reinforcement learningda agent action oqibatini kuzatib, qaysi behavior foydali ekanini tajriba orqali biladi.
Agent
Agent state yoki observation asosida action tanlaydi.
- robot;
- game player;
- software process;
- recommendation policy;
- language model
bo‘lishi mumkin.
U environmentni to‘liq yoki qisman kuzatadi.
Environment
Environment agentdan tashqaridagi tizim.
Agent action yuboradi.
Environment yangi state, observation va reward qaytaradi.
Real muhit qimmat yoki xavfli bo‘lsa simulator ishlatilishi mumkin.
State
State kelajakni bashorat qilish uchun yetarli environment holatini ifodalaydi.
Amalda agent ko‘pincha faqat observation oladi.
Observation to‘liq state bo‘lmasa muammo partially observable hisoblanadi.
Memory oldingi observationlarni saqlashga yordam beradi.
Action
Agent tanlay oladigan qaror.
Action space:
- discrete;
- continuous;
- hybrid;
- structured.
Game’da tugma, robotda motor kuchi, modelda token action bo‘lishi mumkin.
Invalid action mask bilan cheklanadi.
Reward
Reward actiondan keyingi sonli feedback.
Masalan:
- g‘alaba +1;
- crash -100;
- click +1;
- energiya sarfi uchun penalty.
Reward dizayni agent o‘rganadigan behaviorni belgilaydi.
Noto‘g‘ri reward kutilmagan exploitga olib kelishi mumkin.
Return
Agent faqat joriy rewardni emas, kelajak rewardlar yig‘indisini maksimal qilishga intiladi.
Discount factor uzoq kelajak rewardiga qancha weight berilishini belgilaydi.
gamma 1ga yaqin bo‘lsa uzoq horizon muhimroq.
Policy
Policy state’dan action distributioniga mapping.
Deterministic policy bitta action beradi.
Stochastic policy probability distributiondan tanlaydi.
Policy neural network bilan ifodalanishi mumkin.
Value function
State’dan kutiladigan kelajak returnni baholaydi.
V(s) state qiymati.
Q(s,a) state-action juftligi qiymati.
Agent yaxshi actionni value orqali tanlashi mumkin.
Markov Decision Process
RL muammosi ko‘pincha MDP bilan ifodalanadi:
- state;
- action;
- transition;
- reward;
- discount.
Markov xususiyati keyingi state uchun joriy state yetarli ekanini bildiradi.
Real tizimda bu taxmin taxminiy bo‘lishi mumkin.
Exploration
Agent yangi actionlarni sinaydi.
Aks holda erta topilgan, ammo optimal bo‘lmagan behaviorga yopishib qoladi.
Usullar:
- epsilon-greedy;
- entropy bonus;
- noise;
- upper confidence;
- curiosity.
Exploration real tizimda xavfli actionlarni sinamasligi kerak.
Exploitation
Agent hozir eng yaxshi deb bilgan actionni tanlaydi.
Productionda exploitation ko‘proq, kontrolli exploration esa cheklangan bo‘lishi mumkin.
Exploration va exploitation orasida muvozanat kerak.
Q-learning
Model-free value-based algorithm.
Update target reward va keyingi state’dagi maksimal Q qiymatiga tayanadi.
Tabular Q-learning kichik state-action space’da ishlaydi.
Deep Q-Network Q functionni neural network bilan yaqinlashtiradi.
Policy gradient
Policy parameterlari expected return gradienti bilan yangilanadi.
Continuous action va stochastic policy uchun mos.
Gradient variance yuqori bo‘lishi mumkin.
Baseline va advantage variance’ni kamaytiradi.
Actor-Critic
Actor policy’ni, critic value yoki advantage’ni o‘rganadi.
Critic actorga training signal beradi.
PPO, A2C, SAC kabi ko‘p algorithm actor-critic oilasiga kiradi.
On-policy va off-policy
On-policy algorithm ayni policy yaratgan data’da o‘rganadi.
Off-policy eski yoki boshqa policy data’sidan foydalanishi mumkin.
Replay buffer sample efficiency’ni oshiradi.
Distribution farqi correction talab qilishi mumkin.
Model-based RL
Agent transition modelini biladi yoki o‘rganadi.
Kelajakni simulyatsiya qilib plan tuzadi.
Model xatosi uzoq rolloutda yig‘iladi.
Real interactionni kamaytirishi mumkin.
Sample efficiency
Real robot yoki foydalanuvchi bilan interaction qimmat.
Agent kam experience bilan o‘rganishi kerak.
Offline RL oldindan yig‘ilgan datasetdan policy o‘rganadi.
Dataset qamramagan actionlarda uncertainty yuqori.
Reward hacking
Agent reward formulasidagi bo‘shliqdan foydalanib, maqsadga zid, ammo yuqori score oladigan behavior topishi mumkin.
Reward, constraint, monitoring va human review birga ishlaydi.
“Metric maqsadga aylanganda” uning sifati pasayishi mumkin.
Safety
RL agent exploration vaqtida zararli action qilishi mumkin.
Himoya:
- safe action space;
- simulator;
- constraint;
- human approval;
- fallback controller;
- emergency stop;
- offline evaluation.
High-stakes tizimda uncontrolled learning productionda bajarilmaydi.
Episode
Agent environment bilan ma’lum boshlanishdan terminal holatgacha interaction qiladi. Bu ketma-ketlik episode deb ataladi. Har episode uzunligi har xil bo‘lishi mumkin. Game tugashi, robot vazifani bajarishi yoki timeout terminal holat yaratadi.
Trajectory
State, action, reward va keyingi state’lar ketma-ketligi trajectory hisoblanadi. Policy gradient va offline RL training datasetlari trajectorylardan tuziladi. Logda action tanlangan paytdagi policy probability ham saqlanishi mumkin.
Credit assignment
Yakuniy rewardga qaysi oldingi actionlar sabab bo‘lganini aniqlash qiyin. Discount, value function va temporal-difference update rewardni qadamlar bo‘ylab taqsimlashga yordam beradi.
Simulator gap
Simulatorda o‘rgangan policy real dunyoda sensor, physics va noise farqi sabab yomon ishlashi mumkin. Domain randomization va controlled real testing qo‘llanadi.
Baseline policy
Yangi policy oddiy heuristic yoki oldingi production controller bilan solishtiriladi. Simulationdagi yuqori reward real xavfsizlik va cost talablarini almashtirmaydi.
Bog‘liq tushunchalar
Reinforcement Learning, Agent, Environment, State, Action, Reward, Policy, Value function, Q-learning, Actor-Critic, Exploration