Haron forgets everythingHaron sab bhool jaata hai
One morning you ask Haron: "What's the weather in Pune right now?" Haron freezes. It doesn't know what "now" means, because it has no clock. It can't look out of a window. It doesn't even remember that you talked yesterday. Every time you talk to Haron, it is as if it just woke up.
Ek subah aap Haron se poochte ho: "Pune mein abhi mausam kaisa hai?" Haron atak jaata hai. Use "abhi" ka matlab nahi pata, kyunki uske paas ghadi nahi hai. Woh khidki se bahar nahi dekh sakta. Use yeh bhi yaad nahi ki kal aapne usse baat ki thi. Har baar jab aap Haron se baat karte ho, woh aisa hai jaise abhi-abhi neend se utha ho.
So we give Haron a team. Watch them arrive one by one.
Isliye hum Haron ko ek team dete hain. Dekho, ek-ek karke sab aate hain.
Because Haron forgets everything, the team must hand over the whole diary every single time: the rules, the tools, the full chat so far. On message 40 of a chat, you pay again for messages 1 to 39. Most of this book is about doing that smartly.
Kyunki Haron sab bhool jaata hai, team ko har baar poori diary deni padti hai: niyam, tools, ab tak ki poori chat. Chat ke 40th message par aap message 1 se 39 tak ka paisa phir se dete ho. Yeh poori book isi kaam ko smart tareeke se karne ke baare mein hai.
Deep dive: model vs harnessGehraai mein: model vs harnessadvanced
An LLM is a stateless function: it takes a list of token IDs and returns a score (logit) for every token in its vocabulary. It keeps no state between calls and has no tools, clock, identity or budget. Everything agent-like lives in the harness: prompt assembly, tool execution, memory, the control loop, output validation, policy, budgeting and telemetry. When something fails, first decide which side it belongs to. An invented API name is model behaviour, and the harness handles it by validating tool calls. An endless loop is a harness bug: a missing step limit.
LLM ek stateless function hai: token IDs ki list leta hai aur vocabulary ke har token ke liye ek score (logit) deta hai. Do calls ke beech kuch yaad nahi rakhta, aur uske paas na tools hain, na ghadi, na pehchaan, na budget. Agent jaisa sab kuch harness mein hota hai: prompt jodna, tools chalana, memory, loop, output check, policy, budget aur logging. Jab kuch fail ho, pehle socho galti kis taraf ki hai. Model ne nakli API naam bana diya toh woh model ka behaviour hai, aur harness tool call validate karke use pakadta hai. Agent kabhi khatam hi na ho toh woh harness ka bug hai: step limit lagana bhool gaye.
Haron can't read lettersHaron letters nahi padh sakta
Here is a secret: Haron has never seen a single letter. Before a sentence reaches Haron, a helper chops it into pieces, the way a cook chops vegetables before cooking. Each piece is called a token. Common words stay whole. Rare words, like "Haron" or "Bengaluru", get cut into smaller bits.
Ek raaz ki baat: Haron ne aaj tak ek bhi letter nahi dekha. Sentence Haron tak pahunchne se pehle ek helper use tukdon mein kaat deta hai, jaise khaana banane se pehle sabzi kaatte hain. Har tukde ko token kehte hain. Common words poore rehte hain. "Haron" ya "Bengaluru" jaise rare words chhote tukdon mein kat jaate hain.
The last step is the magic one. Each number becomes a long list of numbers called a vector. Think of it as the token's home address on a meaning map. Words with similar meanings live in the same neighbourhood: "king" lives near "queen", "chai" lives near "coffee", and "banana" lives far away from "laptop".
Aakhri step sabse jaadui hai. Har number ek lambi numbers ki list ban jaata hai, jise vector kehte hain. Ise token ka "meaning map" par ghar ka pata samjho. Milte-julte matlab wale words ek hi mohalle mein rehte hain: "king" "queen" ke paas, "chai" "coffee" ke paas, aur "banana" "laptop" se bahut door.
Why Haron can't count the r's in "strawberry"Haron "strawberry" mein r kyun nahi gin paata
Haron sees "straw" + "berry" as two pieces, not ten letters. So counting letters, spelling words backwards, or adding long numbers digit by digit is surprisingly hard for it. That isn't stupidity; Haron simply never sees the letters. The fix belongs to the team: give Haron a calculator or code tool for these jobs.
Haron ke liye "strawberry" das letters nahi, balki do tukde hain: "straw" + "berry". Isliye letters ginna, word ulta likhna, ya lambe numbers jodna uske liye ajeeb tarah se mushkil hai. Yeh bewakoofi nahi hai; Haron letters dekhta hi nahi. Iska ilaaj team ke paas hai: aise kaamon ke liye Haron ko calculator ya code tool do.
Hindi costs more than EnglishHindi, English se mehengi padti hai
Companies pay per token. Tokenizers learned mostly from English, so English words become few pieces. Hindi, Tamil or Kannada text often breaks into many more pieces for the same meaning. Same message, bigger bill. A company serving Indian customers must measure this on real messages, never guess.
Companies har token ka paisa deti hain. Tokenizer zyaadatar English se seekhe hain, isliye English words kam tukdon mein bante hain. Wahi baat Hindi, Tamil ya Kannada mein likho toh aksar kai guna zyaada tukde bante hain. Same message, bada bill. Indian customers wali company ko yeh asli messages par naapna chahiye, andaaza nahi lagana chahiye.
Deep dive: BPE, vocabularies, positionsGehraai mein: BPE, vocabulary, positionadvanced
Most tokenizers use byte-pair encoding (BPE). BPE starts from raw bytes and repeatedly merges the most frequent neighbouring pair until it reaches a target vocabulary size: 128,256 tokens for Llama 3, about 200k for OpenAI's o200k_base. A rough English rule is 1 token ≈ 4 characters ≈ ¾ word. Each token ID selects one row of the embedding matrix (|V| × dmodel, with dmodel = 4,096 in Llama-3-8B).
Zyaadatar tokenizer byte-pair encoding (BPE) use karte hain. BPE bytes se shuru karta hai aur sabse zyaada saath aane wale jode ko baar-baar jodta rehta hai, jab tak vocabulary target size tak na pahunch jaaye: Llama 3 mein 128,256 tokens, OpenAI o200k_base mein lagbhag 200k. English ke liye mota hisaab: 1 token ≈ 4 characters ≈ ¾ word. Har token ID embedding matrix (|V| × dmodel; Llama-3-8B mein dmodel = 4,096) ki ek row chunta hai.
Position: attention by itself has no sense of word order. RoPE (rotary position embedding) rotates query and key vectors by an angle that grows with position, so their dot product depends on the distance between two tokens. Stretching these rotations (NTK scaling, YaRN) is how a model trained on 8k tokens gets served at 128k.
Position: attention khud word ka order nahi jaanta. RoPE query aur key vectors ko position ke hisaab se ghumata hai, taaki unka dot product do tokens ki doori par depend kare. In rotations ko stretch karke (NTK scaling, YaRN) 8k par train hua model 128k tak chalaya jaata hai.
Harness rule: count tokens with each provider's own tokenizer or token-count API. Never use len(text)/4 for budgets.
Harness niyam: tokens har provider ke apne tokenizer ya token-count API se gino. Budget ke liye kabhi len(text)/4 mat lagao.
Who is "it"? Attention"it" kaun hai? Attention
Haron reads: "The cat sat on the mat because it was tired." Who was tired, the cat or the mat? You know instantly. Haron has to work it out, and the way it does this is called attention.
Haron padhta hai: "The cat sat on the mat because it was tired." (Billi chatai par baithi kyunki woh thaki thi.) Thaka kaun tha, billi ya chatai? Aapko turant pata chal gaya. Haron ko yeh nikaalna padta hai, aur jis tareeke se woh nikaalta hai use attention kehte hain.
Picture a classroom. The word "it" stands up holding a question card: "Who am I talking about?" Every earlier word holds up a name tag. The better a name tag matches the question, the more "it" listens to that word, and that word hands over its message.
Ek classroom socho. Word "it" khada hota hai, haath mein ek sawaal card leke: "Main kiski baat kar raha hoon?" Pehle ke har word ke haath mein ek name tag hai. Jiska name tag sawaal se jitna zyaada match kare, "it" us word ki utni zyaada sunta hai, aur woh word apna message de deta hai.
Lost in the middleBeech mein kho jaana
Give Haron a 100-page document and ask about one fact. Haron finds it more easily when the fact is near the start or the end, and more often misses it when it sits in the middle (Liu et al., 2023). It's like a concert: you remember the first and last songs best. So the team puts important rules at the top and the current question at the bottom.
Haron ko 100 page ka document do aur ek fact poocho. Fact shuru ya aakhir mein ho toh Haron use aasaani se dhoondh leta hai, beech mein ho toh aksar miss kar deta hai (Liu et al., 2023). Concert jaisa hai: pehla aur aakhri gaana sabse achha yaad rehta hai. Isliye team zaroori niyam upar rakhti hai aur abhi ka sawaal sabse neeche.
Deep dive: the attention formulaGehraai mein: attention ka formulaadvanced
Q·Kᵀ scores every (question, name-tag) pair. Dividing by √dk keeps the scores from blowing up as vectors get longer. M is the causal mask: it sets every score for a later word to −∞, so a word can never peek ahead. Softmax turns each row into percentages that add up to 100%, and those percentages blend the values. Multi-head attention runs many of these in parallel (64 heads in Llama-3-70B), each learning a different kind of question. GQA lets several query heads share one key/value head (64 query heads, 8 KV heads), which cuts memory by 8×. Models also park extra attention on the first tokens, called attention sinks (Xiao et al., 2023).
Q·Kᵀ har (sawaal, name-tag) jode ko score deta hai. √dk se divide karne par lambe vectors ke saath scores bahut bade nahi hote. M causal mask hai: aage wale words ka score −∞ kar deta hai, taaki koi word aage jhaank na sake. Softmax har row ko percentages mein badalta hai jinka total 100% hota hai, aur unhi percentages se values mix hoti hain. Multi-head attention aise kai kaam saath-saath karta hai (Llama-3-70B mein 64 heads), har head alag tarah ka sawaal seekhta hai. GQA mein kai query heads ek hi key/value head share karte hain (64 query, 8 KV), jisse memory 8 guna kam lagti hai. Model pehle tokens par extra attention daal dete hain; ise attention sinks kehte hain (Xiao et al., 2023).
A building with many floorsKai manzilon wali building
Haron's brain is like a tall building. Each word takes the lift up, floor by floor. On every floor, the word does one round of attention (talking to other words) and one round of thinking alone. By the top floor, the word understands the whole sentence, and the roof shouts out a guess for the next word. Small models have about 30 floors, big ones 80 or more. Each floor is called a layer.
Haron ka dimaag ek oonchi building jaisa hai. Har word lift mein baithkar ek-ek manzil upar jaata hai. Har manzil par word ek baar attention karta hai (baaki words se baat) aur ek baar akele sochta hai. Top floor tak pahunchte-pahunchte word poora sentence samajh jaata hai, aur chhat se agle word ka guess bola jaata hai. Chhote models mein lagbhag 30 manzil hoti hain, bade models mein 80 ya usse zyaada. Har manzil ko layer kehte hain.
Deep dive: residual stream, MLP, mixture of expertsGehraai mein: residual stream, MLP, mixture of expertsadvanced
Each layer is x = x + Attention(RMSNorm(x)) followed by x = x + MLP(RMSNorm(x)). The running vector x is the residual stream (the lift), and every sub-layer adds to it. The MLP holds most of the parameters and much of the stored factual knowledge. In mixture-of-experts models, a router sends each token to only a few expert MLPs: 2 of 8 in Mixtral 8×7B, and about 37B of 671B parameters active per token in DeepSeek-V3. You pay compute only for the active parameters, but all of them must fit in GPU memory. At the top, the LM head maps the final vector to one raw score per vocabulary token. These scores are the logits, the true output of the model.
Har layer mein pehle x = x + Attention(RMSNorm(x)) hota hai, phir x = x + MLP(RMSNorm(x)). Chalta hua vector x residual stream hai (yahi lift hai), aur har sub-layer usme kuch jodti hai. Zyaadatar parameters aur facts ka bada hissa MLP mein hota hai. Mixture-of-experts models mein ek router har token ko sirf kuch expert MLPs ke paas bhejta hai: Mixtral 8×7B mein 8 mein se 2, aur DeepSeek-V3 mein har token par 671B mein se lagbhag 37B parameters active hote hain. Compute ka kharch sirf active parameters ka hota hai, par saare parameters GPU memory mein fit hone chahiye. Top par LM head aakhri vector ko har vocab token ke ek raw score mein badalta hai. Yeh scores logits hain, model ka asli output.
Reading is fast, writing is slowPadhna tez, likhna dheema
Think of an exam. You read the whole question paper in one go, quickly. But you write the answer one word at a time, and before each new word you glance at everything you've written so far. Haron works exactly like this.
Exam socho. Poora question paper aap ek baar mein, jaldi se padh lete ho. Lekin jawab ek-ek word karke likhte ho, aur har naye word se pehle ab tak ka likha hua dekhte ho. Haron bilkul aise hi kaam karta hai.
Words Haron writes (output tokens) are slower and cost about 3 to 5 times more than words it reads (input tokens). A good team keeps Haron's answers short, sets a word limit (max_tokens), and shows words on screen as they come (streaming) so the user isn't left waiting.
Haron jo words likhta hai (output tokens) woh slow hote hain aur jo words padhta hai (input tokens) unse lagbhag 3 se 5 guna mehenge padte hain. Achhi team Haron ke jawab chhote rakhti hai, word limit (max_tokens) lagaati hai, aur words aate hi screen par dikhaati hai (streaming) taaki user ko intezaar na karna pade.
Deep dive: prefill, decode, TTFT, TPOTGehraai mein: prefill, decode, TTFT, TPOTadvanced
Prefill (reading) processes all prompt tokens in parallel as large matrix multiplies. It is compute-bound, and attention cost grows as O(n²). Decode (writing) makes one forward pass per new token. Each pass does little arithmetic but must stream all the weights and the KV cache from GPU memory, so it is memory-bandwidth-bound. TTFT is time to first token, and TPOT is time per output token.
Prefill (padhna) saare prompt tokens ek saath bade matrix multiply se process karta hai. Yeh compute-bound hai, aur attention ka kharch O(n²) ki tarah badhta hai. Decode (likhna) har naye token ke liye ek forward pass karta hai. Har pass mein hisaab kam hota hai, lekin GPU memory se saare weights aur KV cache padhne padte hain, isliye yeh memory-bandwidth-bound hai. TTFT = pehla token aane ka samay, TPOT = har output token ka samay.
Haron's sticky notesHaron ke sticky notes
Imagine writing an essay where, before every new word, you must re-read the whole essay from the first line. Word 100 would mean reading 99 words again. Exhausting! So Haron keeps sticky notes: a short summary of each word's name tag and message, written once and reused forever. Each new word only needs its own note. These notes are the KV cache (K for name tags, V for messages).
Socho aap essay likh rahe ho, aur har naye word se pehle poora essay pehli line se dobara padhna pade. 100th word ke liye 99 words phir se padhne padenge. Thak jaaoge! Isliye Haron sticky notes rakhta hai: har word ke name tag aur message ka chhota note, ek baar likha aur hamesha kaam aaya. Naye word ko bas apna note banana hota hai. Inhi notes ko KV cache kehte hain (K = name tags, V = messages).
The train of notes: prompt cachingNotes ki train: prompt caching
Now picture each request to Haron as a train. The front coaches are the rules and tools, then a big policy book, and the last coach is the customer's question. If the next customer's train has the same front coaches, Haron reuses its old notes for them, and that part costs about 90% less. But the reuse stops at the first coach that is different. Change one tiny thing in the front coach, and every coach behind it has to be read from scratch.
Ab Haron ki har request ko ek train socho. Aage ke dibbe niyam aur tools hain, phir ek badi policy book, aur aakhri dibba customer ka sawaal hai. Agar agle customer ki train ke aage ke dibbe same hain, toh Haron unke purane notes dobara use kar leta hai, aur woh hissa lagbhag 90% sasta padta hai. Lekin jo pehla dibba alag hua, wahin se reuse ruk jaata hai. Aage ke dibbe mein ek chhoti si cheez badli, toh uske peeche ka har dibba shuru se padhna padega.
The golden train ruleTrain ka golden niyam
Line up every request from never-changing to always-changing: rules, then tools in a fixed order, then documents, then the chat history, then the newest question. Things that change every time, like today's date, the user's name or a request ID, go in the last coach, never the first. This one rule can cut a company's bill by half or more.
Har request ko kabhi-na-badalne-wale se hamesha-badalne-wale ki line mein lagao: niyam, phir tools (fixed order mein), phir documents, phir chat history, phir sabse naya sawaal. Har baar badalne wali cheezein, jaise aaj ki date, user ka naam ya request ID, aakhri dibbe mein jaati hain, pehle mein kabhi nahi. Sirf is ek niyam se company ka bill aadha ya usse bhi kam ho sakta hai.
Deep dive: KV-cache memory mathsGehraai mein: KV-cache memory ka hisaabadvanced
| Model | per tokenper token | 128k context | × 32 users× 32 users |
|---|---|---|---|
| Llama-3.1-8B | 128 KB | 15.6 GB | 500 GB |
| Llama-3.1-70B | 320 KB | 39.1 GB | 1.2 TB |
| Same 70B, no GQAWahi 70B, bina GQA | 2.5 MB | 312.5 GB | 9.8 TB |
Numbers are bf16, from code/ch01/kv_cache.py. For a self-hosted model, the notes often take more GPU memory than the model itself: the 70B weights are about 140 GB. That is why vLLM uses PagedAttention (Kwon et al., 2023), and why a per-team context limit is a capacity decision. On APIs, a cache read costs a fraction of the normal input price (10% on Anthropic).
Numbers bf16 mein hain, code/ch01/kv_cache.py se. Khud host kiye model mein aksar notes model se bhi zyaada GPU memory le lete hain: 70B ke weights lagbhag 140 GB ke hain. Isi wajah se vLLM PagedAttention use karta hai (Kwon et al., 2023), aur har team ki context limit ek capacity ka faisla hai. APIs par cache read normal input price ka chhota hissa padta hai (Anthropic par 10%).
The lucky wheelLucky wheel
At the top floor, Haron doesn't give one answer. It gives chances: "Paris" a big chance, "Lyon" a small one, "banana" almost none. To pick a word, the team spins a lucky wheel, like at a fair, where each word's slice is as big as its chance.
Top floor par Haron ek jawab nahi deta. Woh chances deta hai: "Paris" ka bada chance, "Lyon" ka chhota, "banana" ka lagbhag zero. Word chunne ke liye team ek lucky wheel ghumaati hai, mele jaisa, jisme har word ka hissa uske chance jitna bada hota hai.
Two knobs change the wheel. Temperature is how "fair" the wheel is: cold makes the big slice swallow everything, and hot lets small slices grow. Top-p removes the tiny joke slices from the wheel. Try it. Question: "The capital of France is…"
Wheel ko do knob badalte hain. Temperature batata hai wheel kitna "fair" hai: thanda ho toh bada hissa sab kuch kha jaata hai, garam ho toh chhote hisse bade ho jaate hain. Top-p wheel se chhote-chhote mazaak wale hisse hata deta hai. Khud try karo. Sawaal: "The capital of France is…"
A cold wheel doesn't make Haron rightThanda wheel Haron ko sahi nahi banata
If Haron is confidently wrong, and the wrong answer has the biggest slice, a cold wheel picks that wrong answer every single time. Temperature controls variety, not truth. Making Haron correct is the team's job: give it real documents, tools, and checks (Chapters 3 and 8). And a fully cold wheel has its own bug: Haron can get stuck repeating the same phrase in a loop.
Agar Haron pakke yakeen se galat hai, aur galat jawab ka hissa sabse bada hai, toh thanda wheel har baar wahi galat jawab chunega. Temperature variety control karta hai, sach nahi. Haron ko sahi banana team ka kaam hai: asli documents, tools aur checks do (Chapter 3 aur 8). Aur bilkul thande wheel ka apna bug hai: Haron ek hi line baar-baar dohraane ke loop mein atak sakta hai.
"Same question, same answer" is not guaranteed"Same sawaal, same jawab" ki guarantee nahi
Auditors often ask: "If we ask again, will we get the same answer?" Even with temperature 0, the answer can change slightly, because GPU maths depends on how many other requests run alongside, and providers update models. So a company's team must save every request and every answer with the exact model version and settings. The saved log is the proof, not a re-run.
Auditors aksar poochte hain: "Dobara poochenge toh wahi jawab milega?" Temperature 0 par bhi jawab thoda badal sakta hai, kyunki GPU ka hisaab is baat par depend karta hai ki saath mein kitni aur requests chal rahi hain, aur providers models update karte rehte hain. Isliye company ki team ko har request aur har jawab save karna chahiye, exact model version aur settings ke saath. Proof saved log hai, dobara chalana nahi.
Deep dive: the maths of sampling, and which settings to useGehraai mein: sampling ka maths, aur kaunsi settings kabadvanced
Pipeline used by vLLM and llama.cpp: logits → ÷T → softmax → top-k → top-p → min-p → renormalise → draw. Top-k keeps the k most likely tokens. Top-p (nucleus) keeps the smallest set whose probabilities add up to p (Holtzman et al., 2019). Min-p keeps tokens with at least min_p × the top token's probability, and works better at high T (Nguyen et al., 2024).
vLLM aur llama.cpp ki pipeline: logits → ÷T → softmax → top-k → top-p → min-p → renormalise → draw. Top-k sabse likely k tokens rakhta hai. Top-p (nucleus) sabse chhota set rakhta hai jiski probability ka total p ho (Holtzman et al., 2019). Min-p sirf woh tokens rakhta hai jinki probability top token ki probability × min_p se kam na ho, aur high T par behtar chalta hai (Nguyen et al., 2024).
Hosted APIs are moving away from these knobs. Since 21 Oct 2026, Anthropic's Messages API rejects temperature, top_p and top_k on every model; you steer depth and spend with effort instead. The knobs still matter when you self-host (vLLM, llama.cpp) and on providers that keep them, so a provider-agnostic harness must send them only where the provider accepts them (Chapter 5).
Hosted APIs in knobs se door ja rahe hain. 21 Oct 2026 se Anthropic ki Messages API har model par temperature, top_p aur top_k reject karti hai; gehraai aur kharch effort se control hota hai. Self-host (vLLM, llama.cpp) aur jo providers inhe rakhte hain wahan yeh knobs ab bhi kaam ke hain, isliye provider-agnostic harness inhe sirf wahi bheje jahan provider accept kare (Chapter 5).
| TaskKaam | T | TruncationKaat-chhaant |
|---|---|---|
| Tool calls, JSON, classificationTool calls, JSON, classification | 0–0.2 | none |
| CodeCode | 0–0.4 | top-p 0.95 |
| Reasoning modelsReasoning models | default | provider default (often fixed)provider ka default (aksar fixed) |
| Writing, brainstormingLikhna, brainstorming | 0.7–1.0 | top-p 0.9–0.95 / min-p 0.05 |
| Majority voting (n samples)Majority voting (n samples) | 0.7 | top-p 0.95 |
What we learnedHumne kya seekha
- I only guess the next word. My team (the harness) does everything else.
- I forget everything, so my team gives me the whole diary every time, and pays for it.
- I read tokens, not letters. Hindi costs more tokens than English.
- I use attention to work out which words matter. I remember the start and end of long texts best.
- I read fast but write slowly, one word at a time. Writing costs more.
- My sticky notes (KV cache) save work. Keep the front of the train the same to reuse them.
- My lucky wheel controls variety, not truth. Always keep a log.
- Main sirf agla word guess karta hoon. Baaki sab meri team (harness) karti hai.
- Main sab bhool jaata hoon, isliye team mujhe har baar poori diary deti hai, aur uska paisa lagta hai.
- Main letters nahi, tokens padhta hoon. Hindi mein English se zyaada tokens lagte hain.
- Kaunse words zaroori hain, yeh main attention se samajhta hoon. Lambe text ka shuru aur aakhir mujhe sabse achha yaad rehta hai.
- Main tez padhta hoon par dheere likhta hoon, ek-ek word karke. Likhna mehenga hai.
- Mere sticky notes (KV cache) mehnat bachaate hain. Unhe dobara use karne ke liye train ka aage ka hissa same rakho.
- Mera lucky wheel variety control karta hai, sach nahi. Hamesha log rakho.
Word dictionaryShabd-kosh
Try it yourselfKhud karke dekho
Open any tokenizer playground (for example OpenAI's online tokenizer). Type the same sentence in English and in Hindi. Which one makes more tokens? How many times more? easy
Koi bhi tokenizer playground kholo (jaise OpenAI ka online tokenizer). Ek hi sentence English aur Hindi mein likho. Kisme zyaada tokens bane? Kitne guna? easy
On the lucky wheel, set temperature to 0 and spin 10 times. Then set it to 2.5 and spin 10 times. Write down what changed and why. easy
Lucky wheel par temperature 0 karke 10 baar ghumao. Phir 2.5 karke 10 baar ghumao. Kya badla aur kyun, likho. easy
Your company's system prompt starts with "Today is 24 Oct, 10:42 AM. You are a helpful assistant…". Using the train rule, explain what's wrong and how to fix it. harness
Aapki company ka system prompt shuru hota hai "Today is 24 Oct, 10:42 AM. You are a helpful assistant…" se. Train wale niyam se samjhao isme kya galat hai aur kaise theek karein. harness
Run code/ch01/kv_cache.py, then change it to show how many 32k-token chats fit on 8 GPUs of 80 GB each with Llama-3.1-70B. coder
code/ch01/kv_cache.py chalao, phir use badlo taaki dikhe ki Llama-3.1-70B ke saath 80 GB wale 8 GPUs par 32k-token ki kitni chats fit hongi. coder
For coders: build Haron's attention in PythonCoders ke liye: Python mein Haron ka attention banaocode/ch01/
Three small scripts, needing only numpy. The outputs below are real runs.
Teen chhote scripts, sirf numpy chahiye. Neeche ke outputs asli runs ke hain.
scores = q @ k.transpose(0, 2, 1) / np.sqrt(self.dh) # (h, T_new, T_all) # Causal mask: query i (absolute position T_all-T_new+i) may only see keys <= its position. offset = T_all - T_new mask = np.triu(np.ones((T_new, T_all), dtype=bool), k=offset + 1) scores = np.where(mask, -np.inf, scores) w = softmax(scores, axis=-1) out = (w @ v).transpose(1, 0, 2).reshape(T_new, -1) @ self.Wo
Head 0 attention weights (rows = query, cols = key): [[1. 0. 0. 0. 0. 0. ] [0.48 0.52 0. 0. 0. 0. ] [0.01 0.95 0.04 0. 0. 0. ] [0.26 0.35 0.1 0.3 0. 0. ] [0.37 0.05 0.17 0.28 0.12 0. ] [0.29 0.2 0.24 0.16 0.07 0.04]] Prefill output == incremental (KV-cached) output: True Cached K shape (heads, tokens, d_head): (4, 6, 16)
True proves the sticky notes are exact: writing word by word with notes gives the same result as reading everything at once.
True saabit karta hai ki sticky notes bilkul sahi hain: notes ke saath word-by-word likhne ka result wahi hai jo sab ek saath padhne ka.
{'temperature': 0} H=0.00 bits | Paris=1.000
{'temperature': 0.7} H=0.31 bits | Paris=0.962, Lyon=0.015, the=0.010, France=0.006
{'temperature': 1.0} H=0.92 bits | Paris=0.858, Lyon=0.047, the=0.035, France=0.026
{'temperature': 1.5} H=1.80 bits | Paris=0.654, Lyon=0.095, the=0.077, France=0.063
{'temperature': 1.0, 'top_p': 0.9} H=0.30 bits | Paris=0.948, Lyon=0.052
{'temperature': 1.5, 'min_p': 0.05} H=1.66 bits | Paris=0.669, Lyon=0.097, the=0.079, France=0.065
ReferencesReferences
- Vaswani et al. (2017). Attention Is All You Need.
- Su et al. (2021). RoFormer: Rotary Position Embedding. · Ainslie et al. (2023). GQA.
- Holtzman et al. (2019). The Curious Case of Neural Text Degeneration. · Nguyen et al. (2024). Min-p Sampling.
- Liu et al. (2023). Lost in the Middle. · Xiao et al. (2023). Attention Sinks.
- Kwon et al. (2023). PagedAttention (vLLM). · Jiang et al. (2024). Mixtral. · DeepSeek-AI (2024). DeepSeek-V3.
- He & Thinking Machines Lab (2025). Defeating Nondeterminism in LLM Inference.