Term
Continued Pretraining
Continued pretraining is the further training of an already pretrained language model on large amounts of domain text — before any classic fine-tuning begins.
Continued pretraining — explained in more detail
When a general-purpose LLM should perform significantly better in a narrow domain (law, medicine, mechanical engineering, an underrepresented language), classic SFT with a few thousand question-answer pairs is often not enough. The model simply does not know the specialist vocabulary, syntax and conventions well enough. Continued pretraining (sometimes called domain-adaptive pretraining, DAPT) intervenes earlier: the base model is trained further with the same procedure as in the original pretraining — next-token prediction on raw text — on a large domain corpus. Typical data volumes: hundreds of millions to billions of tokens. SFT and possibly preference optimisation follow afterwards.
Example / Practical context
Classic cases: BloombergGPT (finance), Med-PaLM (medicine), Code Llama (code-specific extension of Llama), or specialised German-language models such as SauerkrautLM, which post-train an English-dominated base model on large German corpora. Tackling this in-house generally requires several hundred GB of domain text and corresponding GPU resources — significantly more demanding than SFT with LoRA.
Distinction from related concepts
Pretraining starts from randomly initialised weights and takes weeks to months across many GPUs. Continued pretraining takes an already trained model and adapts it on new text — hours to days, much cheaper. SFT follows afterwards with supervised examples. Risk in continued pretraining: catastrophic forgetting — the model can lose general knowledge or instruction-following ability if the domain data is too narrow or the training runs too long.
Entdecke mehr
AI Workflows by Keyword: How We Make Recurring Routines Enforceable
A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.
GlossarDPO (Direct Preference Optimization)
DPO is a fine-tuning method that aligns language models directly to human preference pairs — without a separate reward model and without reinforcement learning.
LexikonFine-Tuning Explained — When, How, With What
When fine-tuning pays off, which methods exist (SFT, DPO, LoRA, QLoRA), and what AMD vs. NVIDIA hardware actually means in practice.