# items

url: https://www.theregister.com/2025/08/24/llama_cpp_hands_on/
title: How to run LLMs on PC at home using Llama.cpp • The Register
added_at: 2025-08-28 03:31:23
thumb_url: https://regmedia.co.uk/2025/08/21/llamacpp_cli.jpg?x=648&y=436&infer_y=1
content_html:
<p>Hands on Training large language models (LLMs) may require millions or even billion of dollars of infrastructure, but the fruits of that labor are often more accessible than you might think. Many recent releases, including Alibaba's Qwen 3 and OpenAI's gpt-oss, can run on even modest PC hardware.</p>
<p>If you really want to learn about how LLMs work, running one locally is essential. It also gives you unlimited access to a chatbot, without paying extra for priority access or sending your data to the cloud. While there are simpler tools, activating <a href="https://github.com/ggml-org/llama.cpp" rel="nofollow" target="_blank">Llama.cpp</a> at the command line provides the best performance and most options, including the ability to assign the workload to the CPU or GPU and the capability of quantizing (aka compressing) models for faster output.</p>
<p>Under the hood, many of the most popular frameworks for running models locally on your PC or Mac, including Ollama, Jan, or LM Studio are really wrappers built atop Llama.cpp's open source foundation with the goal of abstracting away complexity and improving the user experience.</p>
<p>While these niceties make running local models less daunting for newcomers, they often leave something to be desired with regard to performance and features.</p>
<p>As of this writing, Ollama still doesn't support Llama.cpp's Vulkan back end, which offers broader compatibility and often higher generation performance, particularly for AMD GPUs and APUs. And while LM Studio does support Vulkan, it lacks support for Intel's SYCL runtime and GGUF model creation.</p>
<p>In this hands-on guide, we'll explore Llama.cpp, including how to build and install the app, deploy and serve LLMs across GPUs and CPUs, generate quantized models, maximize performance, and enable tool calling.</p>

<h3>Prerequisites:</h3>
<p>Llama.cpp will run on just about anything, including a Raspberry Pi. However, for the best experience possible, we recommend a machine with at least 16GB of system memory.</p>
<p>While not required, a dedicated GPU from Intel, AMD, or Nvidia will vastly improve performance. If you do have access to one you'll want to make sure you have the latest drivers for them installed on your system before proceeding.</p>

<p>For most users, installing Llama.cpp is about as easy as downloading a ZIP file.</p>
<p>While Llama.cpp may be available from package managers like <code>apt</code>, <code>snap</code>, or <code>WinGet</code>, it is updated very frequently, sometimes multiple times a day, so it's best to grab the latest precompiled binaries from the official GitHub <a href="https://github.com/ggml-org/llama.cpp" rel="nofollow" target="_blank">page</a>.</p>
<p>Binaries are available for a variety of accelerators and frameworks for macOS, Windows, and Ubuntu on both Arm64 and x86-64 based host CPUs.</p>
<p>Here's a quick cheat sheet if you're uncertain which to grab:</p>
<ul>
<li><strong>Nvidia:</strong> CUDA</li>
<li><strong>Intel Arc / Xe:</strong> Sycl</li>
<li><strong>AMD:</strong> Vulkan or HIP</li>
<li><strong>Qualcomm:</strong> OpenCL-Adreno</li>
<li><strong>Apple M-series:</strong> macOS-Arm64</li>
</ul>
<p>Or, if you don't have a supported GPU, grab the appropriate "CPU" build for your operating system and processor architecture. Note that integrated GPUs can be rather hit-and-miss with Llama.cpp and, due to memory bandwidth constraints, may not result in higher performance than CPU-based inference even if you can get them working.</p>
<p>Once you have downloaded Llama.cpp, unzip the folder to your home directory for easy access.</p>
<p>If you can't find a prebuilt binary for your preferred flavor of Linux or accelerator we'll cover how to build Llama.cpp from source a little later. We promise it's easier than it sounds.</p>

<h3>macOS users:</h3>
<p>While we recommend that Windows and Linux users grab the precompiled binaries from GitHub, platform security measures in macOS make running unsigned code a bit of a headache. Because of this, we recommend that macOS users employ the <code>brew</code> package manager to install Llama.cpp. Just be aware that it may not be the latest version available.</p>
<p>A guide to setting up the Homebrew package manager can be found <a href="https://brew.sh/" rel="nofollow" target="_blank">here</a>. Once you have Homebrew installed, you can get Llama.cpp by running:</p>
<pre>brew install llama.cpp
</pre>
<h3>Deploying your first model</h3>
<p>Unlike other apps such as LM Studio or Ollama, Llama.cpp is a command-line utility. To access it, you'll need to open the terminal and navigate to the folder we just downloaded. Note that, on Linux, the binaries will be located under the <code>build/bin</code> directory.</p>
<pre>cd folder_name_here
</pre>
<p>We can then run the following command to download and run a 4-bit quantized version of Qwen3-8B within a command-line chat interface on our device. For this model, we recommend at at least 8GB of system memory or a GPU with at least 6GB of VRAM.</p>
<pre>./llama-cli -hfr bartowski/Qwen_Qwen3-8B-GGUF:Q4_K_M
</pre>
<p>If you installed Llama.cpp using <code>brew</code>, you can leave off the <code>./</code> before <code>llama-cli</code>.</p>
<p>Once the model is downloaded, it should only take a few seconds to spin up and you'll be presented with a rudimentary command-line chat interface.</p>
<a href="https://regmedia.co.uk/2025/08/21/llamacpp_cli.jpg" rel="noopener noreferrer" target="_blank"><img alt="The easiest way to interact with Llama.cpp is through the CLI" loading="lazy" referrerpolicy="no-referrer" src="https://regmedia.co.uk/2025/08/21/llamacpp_cli.jpg?x=648&amp;y=436&amp;infer_y=1" title="The easiest way to interact with Llama.cpp is through the CLI"/></a>
<p>The easiest way to interact with Llama.cpp is through the CLI - Click to enlarge</p>

<p>Unless you happen to be running M-series silicon, Llama.cpp is going to load the model into system memory and run it on the CPU by default. If you've got a GPU with enough memory, you probably don't want to do that since DDR is usually a lot slower than GDDR.</p>
<p>To use the GPU, we need to specify how many layers you'd like to offload onto it by appending the <code>-ngl</code> flag. In this case, Qwen3-8B has 37 layers, but if you're not sure, setting <code>-ngl</code> to something like 999 will guarantee the model runs entirely from the GPU. And, yes, you can adjust this to split the model between the system and GPU memory if you don't have enough. We'll dive deeper into that, including some advanced approaches, a little later in the story.</p>
<pre>./llama-cli -hfr bartowski/Qwen_Qwen3-8B-GGUF:Q4_K_M -ngl 37
</pre>
<h3>Dealing with multiple devices</h3>
<p>Llama.cpp will attempt to use all available GPUs which may cause problems if you've got both a dedicated graphics card and an iGPU on board. In our testing with an AMD W7900 on Windows using the HIP binaries, we encountered an error because the model attempted to offload some layers to our CPU's integrated graphics.</p>
<p>To get around this, we can specify which GPUs to run Llama.cpp on using the &lt;device&gt;--device&lt;/device&gt; flag. We can list all available devices by running the following:</p>
<pre>./llama-cli --list-devices
</pre>
<p>You should see an output similar to this one:</p>
<pre>Available devices:
  ROCm0: AMD Radeon RX 7900 XT (20464 MiB, 20314 MiB free
  ROCm1: AMD Radeon(TM) Graphics (12498 MiB, 12347 MiB free)
</pre>
<p>Note that, depending on whether you're using the HIP, Vulkan, CUDA, or OpenCL, backend device names are going to be different. For example if you're using CUDA, you might see CUDA0 and CUDA1.</p>
<p>We can now launch Llama.cpp using our preferred GPU by running</p>
<pre>./llama-cli -hfr bartowski/Qwen_Qwen3-8B-GGUF:Q4_K_M -ngl 37 --device ROCm0
</pre>
<h3>Serving up your model:</h3>
<p>As great as the CLI-based chat interface is, it's not necessarily the most convenient way to interact with Llama.cpp. Instead, you might want to connect it up to a graphical user interface (GUI) instead.</p>
<p>Thankfully, Llama.cpp includes an API server that can be connected to any app that supports OpenAI-compatible endpoints, like Jan or Open WebUI. If you just want a basic GUI, you don't need to do anything special, just launch the model with <code>llama-server</code> instead.</p>
<pre>./llama-server -hfr bartowski/Qwen_Qwen3-8B-GGUF:Q4_K_M -ngl 37
</pre>
<p>After a few moments, you should be able to open the web GUI by navigating to <code>http://localhost:8080</code> in your web browser.</p>
<a href="https://regmedia.co.uk/2025/08/21/llamacpp_webui.jpg" rel="noopener noreferrer" target="_blank"><img alt="By default launching a model with llama-server will start a basic web interface for chatting with models" loading="lazy" referrerpolicy="no-referrer" src="https://regmedia.co.uk/2025/08/21/llamacpp_webui.jpg?x=648&amp;y=428&amp;infer_y=1" title="By default launching a model with llama-server will start a basic web interface for chatting with models"/></a>
<p>By default, launching a model with llama-server will start a basic web interface for chatting with models - Click to enlarge</p>

<p>If you want to access the server from another device, you'll need to expose the server to the rest of your network, by setting the <code>--host</code> address to <code>0.0.0.0</code>, and, if you want to use a different port, you'd append the <code>--port</code> flag. If you're going to make your server available to strangers on the Internet or a large network, we also recommend setting an <code>--api-key</code> flag.</p>
<pre>./llama-server -hfr bartowski/Qwen_Qwen3-8B-GGUF:Q4_K_M -ngl 37 --host 0.0.0.0 --port 8000 --api-key top-secret
</pre>
<p>Now the API will be available at:</p>
<p><strong>API address:</strong> http://ServerIP:8080/v1</p>
<p>The API key should be passed to the URL as a bearer token if you're writing your own tool. Some other tools such as OpenWeb UI have a field where you can enter the key and the client application takes care of the rest.</p>
<p><strong>Editor's note:</strong> For most home users, this should be relatively safe as you should be deploying the model from behind your router's firewall. However, if you're running Llama.cpp in the cloud, you'll definitely want to lock down your firewall first.</p>
<h3>Where to find models</h3>
<p>Llama.cpp works with most models quantized using the GGUF format. These models can be found on a variety of model repos, with Hugging Face being among the most popular.</p>
<p>If you're looking for a particular model, it's worth checking profiles like <a href="https://huggingface.co/bartowski" rel="nofollow" target="_blank">Bartowski</a>, <a href="https://huggingface.co/unsloth" rel="nofollow" target="_blank">Unsloth</a>, and <a href="https://huggingface.co/ggml-org" rel="nofollow" target="_blank">GGML-Org</a> as they're usually among the first to have GGUF quants of new models.</p>
<p>If you're using Hugging Face, downloading them can be done directly from Llama.cpp. In fact, this is how we pulled down Qwen3-8B in the earlier step and requires specifying the model repo and the specific quantization level you'd prefer.</p>
<p>For example <code>-hfr bartowski/Qwen_Qwen3-8B-GGUF:Q8_0</code> would pull down an 8-bit quantized version of the model while <code>-hfr bartowski/Qwen_Qwen3-8B-GGUF:IQ3_XS</code> would download a 3-bit i-quant.</p>
<p>Generally speaking, smaller quants require fewer resources to run, but also tend to be lower quality.</p>
<h3>Quantizing your own models</h3>
<p>If the model you're looking for isn't already available as a GGUF, you may be able to create your own. Llama.cpp provides tools for converting models to GGUF format and then quantizing them from 16-bits to a lower precision (usually 8-2 bits) so they can run on lesser hardware.</p>
<p>To do this, you'll need to clone the Llama.cpp repo and install a recent version of Python. If you're running Windows, we actually find it's easier to do this step in Windows Subsystem for Linux (WSL) rather than trying to wrangle Python packages natively. If you need help setting up WSL, Microsoft's set up guide can be found <a href="https://learn.microsoft.com/en-us/windows/wsl/install" rel="nofollow" target="_blank">here</a>.</p>
<pre>git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
</pre>
<p>Next, we'll need to create a Python virtual environment and install our dependencies. If you don't already have <code>python3-pip</code> and <code>python3-venv</code> installed you'll want to grab those first.</p>
<pre>sudo apt install python3-pip python3-venv
</pre>
<p>Then create a virtual environment and activate by running:</p>
<pre>python3 -m venv llama-cpp
source llama-cpp/bin/activate
</pre>
<p>With that out of the way we can install the Python dependencies with:</p>
<pre>pip install -r requirements.txt
</pre>
<p>From there, we can use the <code>convert_hf_to_gguf.py</code> script to convert a safe tensors model, in this case Microsoft's Phi4, to a 16-bit GGUF file.</p>
<pre>python3.12 convert_hf_to_gguf.py --remote microsoft/phi-4 --outfile phi4-14b-FP16.gguf
</pre>
<p>Unless you've got a multi-gig interconnection it will take a few minutes to download. At native precision, Phi 4 is nearly 30GB. If you run into an error trying to download a model like Llama or Gemma, you likely need to request permission on Hugging Face first and sign in using the <code>huggingface-cli</code></p>
<pre>huggingface-cli log-in
</pre>
<p>From here, we can quantize the model to the desired bit-width. We'll use <code>Q4_K_M</code> quantization since it cuts the model size by nearly three quarters without sacrificing too much quality. You can find a full list of available quants <a href="https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md" rel="nofollow" target="_blank">here</a>. Or by running <code>llama-quantize --help</code>.</p>
<pre>./llama-quantize phi4-14b-FP16.gguf phi4-14b-Q4_K_M.gguf q4_k_m
</pre>
<p>To test the model, we can launch <code>llama-cli</code> but rather than using <code>-hfr</code> to select a Hugging Face repo, we'll use <code>-m</code> instead and point it at our newly quantized model.</p>
<pre>llama-cli -m phi4-14b-Q4_K_M.gguf -ngl 99
</pre>
<p>If you'd like to learn more about quantization, we've got an entire guide dedicated to model compression including how to measure and minimize quality losses, which you can find <a href="https://www.theregister.com/2024/07/14/quantization_llm_feature/" rel="nofollow" target="_blank">here</a>.</p>
<p>
<strong>Building from source</strong>
</p>
<p>On the off chance that the Llama.cpp doesn't offer a precompiled binary for your hardware or operating system, you'll need to build the app from source.</p>
<p>The Llama.cpp dev team maintains comprehensive documentation on how to build from source on every operating system and compute runtime, be it CUDA, HIP, SYCL, CANN, MUSA, or something else. Regardless of which you're building for, you'll want to make sure you have the latest drivers and runtimes installed and configured first.</p>
<p>For this demonstration, we'll be building Llama.cpp for both an 8GB Raspberry Pi 5 and an x86-based Linux box with an Nvidia GPU since there are no pre-compiled binaries for either. For our host operating system, we'll be using Ubuntu Server (25.04 for the RPI and 24.04 for the PC).</p>
<p>To get started, we'll install a couple of dependencies using apt.</p>
<pre>sudo apt install git cmake build-essential libcurl4-openssl-dev
</pre>
<p>Next we can clone the repo from GitHub and open the directory.</p>
<pre>git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
</pre>
<p>From there, building the Llama.cpp is pretty simple. The devs have an entire page dedicated to <a href="https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md" rel="nofollow" target="_blank">building from source</a> with instructions for everything SYCL and CUDA to HIP and even Huawei's CANN and Moore Threads MUSA runtimes.</p>
<p>
<strong>Building LLama.cpp on the RPI5:</strong>
</p>
<p>For the Raspberry Pi 5, we can use the standard build flags. Note we've added the <code>-j 4</code> flag here to parallelize the process across the RPI's four cores.</p>
<pre>cmake -B build
cmake --build build --config Release -j 4
</pre>
<p>
<strong>Building Llama.cpp for x86 and CUDA:</strong>
</p>
<p>For the x86 box, we'll need to make sure that the Nvidia drivers and CUDA tool kit are installed first by running:</p>
<pre>sudo apt install nvidia-driver-570-server nvidia-cuda-toolkit
sudo reboot
</pre>
<p>After the system has rebooted, open the llama.cpp folder and build it with CUDA support by running:</p>
<pre>cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release
</pre>
<p>Building for a different system or running into trouble? Check out Llama.cpp's <a href="https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md" rel="nofollow" target="_blank">docs</a>.</p>
<p>
<strong>Completing the installation:</strong>
</p>
<p>Regardless of which system you're building on, it could take a couple minutes to complete. Once they are finished, binaries will be saved in the <code>/build/bin/</code> folder.</p>
<p>You can run these binaries directly from this directory, or complete the installation by copying them to your <code>/usr/bin</code> directory.</p>
<pre>sudo cp /build/bin/ /usr/bin/
</pre>
<p>If everything worked correctly, we should be able to spin up Google's itty bitty <a href="https://www.theregister.com/2025/08/15/little_llm_on_the_ram/" rel="noopener noreferrer" target="_blank">new language model</a>, Gemma 3 270M, by running:</p>
<pre>llama-cli -hfr bartowski/google_gemma-3-270m-it-qat-GGUF:bf16
</pre>
<p>Would we recommend running LLMs on a Raspberry Pi? Not really, but at least now you know you can.</p>
<h3>Performance tuning</h3>
<p>So far we've covered how to download, install, run, serve, and quantize models in Llama.cpp, but we've only scratched the surface of what it's capable of.</p>
<p>Run <code>llama-cli --help</code> and you'll see just how many levers to pull and knobs to turn there really are. So, lets take a look at some of the more useful flags at our disposal.</p>
<p>In this example, we've configured Llama.cpp to run OpenAI's gpt-oss-20b model along with a few addition flags to maximize performance. Let's break them down one by one.</p>
<pre>
./llama-server -hf ggml-org/gpt-oss-20b-GGUF --jinja -fa -c 16384 -np 2 --cache-reuse 256 -ngl 99
</pre>
<p><strong><code>-fa</code></strong> — enables Flash Attention on supported platforms which can dramatically speed up prompt processing times while also reducing memory requirements. We find that this is beneficial for most setups, but it's worth trying with and without to be sure.</p>
<p><strong><code>-c 16384</code></strong> — sets the model's context window, or short-term memory, to 16,384 tokens. If left unset, Llama.cpp defaults to 4,096 tokens which minimizes memory requirements, but means the model will start forgetting details once you exceed that threshold. If you've got memory to spare, we recommend setting this as high as you can without running into out-of-memory errors, up to the model limits. For gpt-oss, that's 131,072 tokens.</p>
<p>The larger the context window, the more RAM or VRAM is required to run the model. LMCache offers a <a href="https://lmcache.ai/kv_cache_calculator.html" rel="nofollow" target="_blank">calculator</a> to help you determine how many tokens you can fit into your memory.</p>
<p><strong><code>-np 2</code></strong> — allows Llama.cpp to process up to two requests simultaneously, which can be useful in multi-user or when connecting Llama.cpp to code assistant tools like Cline or Continue, which might make multiple simultaneous requests for code competition or chat functionality. Note that the context window is divided by the number of parallel processes. In this example, each parallel process has an 8192 token context.</p>
<p><strong><code>--cache-reuse 256</code></strong> — setting this helps to avoid recomputing key-value caches speeding up prompt processing, particularly for extended multi-turn conversations. We recommend starting with 256 token chunks.</p>
<h3>Hyperparameter tuning</h3>
<p>For optimal performance and output quality, many model builders recommend setting sampling parameters, like temperature or min-p to specific values.</p>
<p>For example, Alibaba's Qwen team recommends setting temp to 0.7, top-p to 0.8, top-k to 20, and min-p to 0 when running many of its instruct models, like Qwen3-30B-A3B-Instruct-2507. Recommended hyperparameters can usually be found on the model card on repos like Hugging Face.</p>
<p>In a nutshell, these parameters influence which tokens the model selects from the probability curve. Temperature is one of the easiest to understand, as setting it lower usually results in less creative and more deterministic outputs while setting it higher will result in more adventurous results.</p>
<p>Many applications, like Open WebUI, LibreChat, and Jan allow overriding them via the API, which they (or you) can access at localhost:8080 if you are running llama-server. However, for applications that don't, it can be helpful to set these when spinning up the model in Llama.cpp.</p>
<p>For example, for Qwen 3 instruct models, we'd run something like the following. (Note this particular model requires a little over 20GB of memory so you if want to test it, you may need to swap out Qwen for a smaller model.)</p>
<pre>./llama-server -hfr bartowski/Qwen_Qwen3-30B-A3B-Instruct-2507-GGUF:Q4_K_M --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0
</pre>
<p>You can find a full list of available sampling parameters by running:</p>
<pre>./llama-cli --help
</pre>
<p>For more information on how sampling parameters impact output generation, check out Amazon Web Services' explainer <a href="https://docs.aws.amazon.com/bedrock/latest/userguide/inference-parameters.html" rel="nofollow" target="_blank">here</a>.</p>
<h3>Boosting performance with speculative decoding</h3>
<p>One of the features in Llama.cpp that you won't see in other model runners, like Ollama, is support for speculative decoding. This process can speed up token generation in highly repetitive workloads, like code generation, by using a small draft model to predict the outputs of a larger, more accurate one.</p>
<p>This approach does require a compatible draft model, usually one from the same family as your primary one. In this example we're using speculative decoding to speed up Alibaba's Qwen3-14B model using the 0.6B variant as our drafter.</p>
<pre>./llama-server -hfr Qwen/Qwen3-14B-GGUF:Q4_K_M -hfrd Qwen/Qwen3-0.6B-GGUF:Q8_0 -c 4096 -cd 4096 -ngl 99 -ngld 99 --draft-max 32 --draft-min 2 --cache-reuse 256 -fa
</pre>
<p>To test, we can ask the model to generate a block of text or code. In our testing with speculative decoding enabled, generation rates were roughly on par with running Qwen3-14B on its own. However, when we requested a minor change to that text or code, performance roughly doubled jumping from around 60 tok/s to 117 tok/s.</p>
<p>If you'd like to know more about how speculative decoding works in Llama.cpp, you can find our deep dive <a href="https://www.theregister.com/2024/12/15/speculative_decoding/" rel="noopener noreferrer" target="_blank">here</a>.</p>
<h3>Splitting big models between CPUs and GPUs</h3>
<p>One of Llama.cpp's most valuable features is its ability to split large models between CPU and GPUs. So long as you have enough memory between your DRAM and VRAM to fit the model weights (and the OS), there's a good chance you'll be able to run it.</p>
<p>As we alluded to earlier, the easiest way to do this is to slowly increase the number layers offloaded to the GPU (<code>-ngl</code>) until you get an out-of-memory error and then back off a bit.</p>
<p>For example, if you had a GPU with 20GB of vRAM and 32GB of DDR5 and wanted to run Meta's Llama 3.1 70B model at 4-bit precision, which requires a little over 42GB of memory, you might offload 40 of the layers to the GPU and run the rest on the CPU.</p>
<pre>./llama-server -hfr bartowski/Meta-Llama-3.1-70B-Instruct-GGUF -ngl 40
</pre>
<p>While the model runs, performance won't be great — in our testing we got about 2 tok/s.</p>
<p>However, thanks to a relatively small number of active parameters in mixture of expert (MoE) models such as gpt-oss, its actually possible to get decent performance, even when running far larger models.</p>
<p>By taking advantage of Llama.cpp's MoE expert offload features, we were able to get OpenAI's 120 billion parameter gpt-oss model running at a rather-respectable 20 tok/s on a system with a 20GB GPU and 64GB of DDR4 3200 MT/s memory.</p>
<pre>./llama-server -hf ggml-org/gpt-oss-120b-GGU -fa -c 32768 --jinja -ngl 999 --n-cpu-moe 26
</pre>
<p>In this case, we set <code>-ngl</code> to <code>999</code> and used the <code>--n-cpu-moe</code> parameter to offload progressively more expert layers to the CPU until Llama.cpp stopped throwing out-of-memory errors.</p>
<h3>Tool calling</h3>
<p>If your workload requires it, Llama.cpp can also parse tool calls from OpenAI compatible API Endpoints like Open WebUI or Cline. You'd need tools if you want to bring in an outside feature such as a clock, a calculator, or to check the status of Proxmox cluster.</p>
<a href="https://regmedia.co.uk/2024/08/22/proxmox_report_tool.jpg" rel="noopener noreferrer" target="_blank"><img alt="We previously built a tool to generate status reports from Proxmox VE." loading="lazy" referrerpolicy="no-referrer" src="https://regmedia.co.uk/2024/08/22/proxmox_report_tool.jpg?x=442&amp;y=273&amp;infer_y=1" title="We previously built a tool to generate status reports from Proxmox VE."/></a>
<p>We previously built a tool to generate status reports from Proxmox VE.</p>

<p>Enabling tool calling varies from model to model. For most popular models, including gpt-oss, nothing special is required. Simply append the <code>--jinja</code> flag and you're off to the races.</p>
<pre>./llama-server -hf ggml-org/gpt-oss-20b-GGUF --jinja
</pre>
<p>Others models, like DeepSeek R1, may require setting a chat-template manually when launching the model. For example:</p>
<pre>./llama-server --jinja -fa -hf bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF:Q4_K_M \
    --chat-template-file models/templates/llama-cpp-deepseek-r1.jinja
</pre>
<p>Tool calling is a whole can of worms unto itself, so, if you're interested in learning more, checking out our functional calling and Model Context Protocol deep dives <a href="https://www.theregister.com/2024/08/26/ai_llm_tool_calling/" rel="noopener noreferrer" target="_blank">here</a> and <a href="https://www.theregister.com/2025/04/21/mcp_guide/" rel="noopener noreferrer" target="_blank">here</a>.</p>
<h3>Summing up</h3>
<p>While Llama.cpp may be one of the most comprehensive model runners out there — we've only discussed a fraction of all the app entails — we understand it can quite daunting for those dipping their toes into local LLMs for the first time. This is one of the reasons why it's taken us so long to do a hands-on of the app, and why we think that simpler apps like <a href="https://www.theregister.com/2025/08/07/run_openai_gpt_oss_locally/" rel="noopener noreferrer" target="_blank">Ollama</a> and <a href="https://lmstudio.ai/" rel="nofollow" target="_blank">LM Studio</a> are still valuable.</p>
<p>So now that you've gotten your head wrapped around Llama.cpp you might be wondering how LLMs are <a href="https://www.theregister.com/2025/04/22/llm_production_guide/" rel="noopener noreferrer" target="_blank">actually deployed</a> in production, or maybe how to get started with <a href="https://www.theregister.com/2024/06/29/image_gen_guide/" rel="noopener noreferrer" target="_blank">image generation</a>. We've got guides for both. ®</p>

content_text:
Hands on Training large language models (LLMs) may require millions or even billion of dollars of infrastructure, but the fruits of that labor are often more accessible than you might think. Many recent releases, including Alibaba's Qwen 3 and OpenAI's gpt-oss, can run on even modest PC hardware.

If you really want to learn about how LLMs work, running one locally is essential. It also gives you unlimited access to a chatbot, without paying extra for priority access or sending your data to the cloud. While there are simpler tools, activating

Llama.cpp

at the command line provides the best performance and most options, including the ability to assign the workload to the CPU or GPU and the capability of quantizing (aka compressing) models for faster output.

Under the hood, many of the most popular frameworks for running models locally on your PC or Mac, including Ollama, Jan, or LM Studio are really wrappers built atop Llama.cpp's open source foundation with the goal of abstracting away complexity and improving the user experience.

While these niceties make running local models less daunting for newcomers, they often leave something to be desired with regard to performance and features.

As of this writing, Ollama still doesn't support Llama.cpp's Vulkan back end, which offers broader compatibility and often higher generation performance, particularly for AMD GPUs and APUs. And while LM Studio does support Vulkan, it lacks support for Intel's SYCL runtime and GGUF model creation.

In this hands-on guide, we'll explore Llama.cpp, including how to build and install the app, deploy and serve LLMs across GPUs and CPUs, generate quantized models, maximize performance, and enable tool calling.

Prerequisites:

Llama.cpp will run on just about anything, including a Raspberry Pi. However, for the best experience possible, we recommend a machine with at least 16GB of system memory.

While not required, a dedicated GPU from Intel, AMD, or Nvidia will vastly improve performance. If you do have access to one you'll want to make sure you have the latest drivers for them installed on your system before proceeding.

For most users, installing Llama.cpp is about as easy as downloading a ZIP file.

While Llama.cpp may be available from package managers like

apt

,

snap

, or

WinGet

, it is updated very frequently, sometimes multiple times a day, so it's best to grab the latest precompiled binaries from the official GitHub

page

.

Binaries are available for a variety of accelerators and frameworks for macOS, Windows, and Ubuntu on both Arm64 and x86-64 based host CPUs.

Here's a quick cheat sheet if you're uncertain which to grab:

Nvidia:

CUDA

Intel Arc / Xe:

Sycl

AMD:

Vulkan or HIP

Qualcomm:

OpenCL-Adreno

Apple M-series:

macOS-Arm64

Or, if you don't have a supported GPU, grab the appropriate "CPU" build for your operating system and processor architecture. Note that integrated GPUs can be rather hit-and-miss with Llama.cpp and, due to memory bandwidth constraints, may not result in higher performance than CPU-based inference even if you can get them working.

Once you have downloaded Llama.cpp, unzip the folder to your home directory for easy access.

If you can't find a prebuilt binary for your preferred flavor of Linux or accelerator we'll cover how to build Llama.cpp from source a little later. We promise it's easier than it sounds.

macOS users:

While we recommend that Windows and Linux users grab the precompiled binaries from GitHub, platform security measures in macOS make running unsigned code a bit of a headache. Because of this, we recommend that macOS users employ the

brew

package manager to install Llama.cpp. Just be aware that it may not be the latest version available.

A guide to setting up the Homebrew package manager can be found

here

. Once you have Homebrew installed, you can get Llama.cpp by running:

brew install llama.cpp

Deploying your first model

Unlike other apps such as LM Studio or Ollama, Llama.cpp is a command-line utility. To access it, you'll need to open the terminal and navigate to the folder we just downloaded. Note that, on Linux, the binaries will be located under the

build/bin

directory.

cd folder_name_here

We can then run the following command to download and run a 4-bit quantized version of Qwen3-8B within a command-line chat interface on our device. For this model, we recommend at at least 8GB of system memory or a GPU with at least 6GB of VRAM.

./llama-cli -hfr bartowski/Qwen_Qwen3-8B-GGUF:Q4_K_M

If you installed Llama.cpp using

brew

, you can leave off the

./

before

llama-cli

.

Once the model is downloaded, it should only take a few seconds to spin up and you'll be presented with a rudimentary command-line chat interface.

The easiest way to interact with Llama.cpp is through the CLI - Click to enlarge

Unless you happen to be running M-series silicon, Llama.cpp is going to load the model into system memory and run it on the CPU by default. If you've got a GPU with enough memory, you probably don't want to do that since DDR is usually a lot slower than GDDR.

To use the GPU, we need to specify how many layers you'd like to offload onto it by appending the

-ngl

flag. In this case, Qwen3-8B has 37 layers, but if you're not sure, setting

-ngl

to something like 999 will guarantee the model runs entirely from the GPU. And, yes, you can adjust this to split the model between the system and GPU memory if you don't have enough. We'll dive deeper into that, including some advanced approaches, a little later in the story.

./llama-cli -hfr bartowski/Qwen_Qwen3-8B-GGUF:Q4_K_M -ngl 37

Dealing with multiple devices

Llama.cpp will attempt to use all available GPUs which may cause problems if you've got both a dedicated graphics card and an iGPU on board. In our testing with an AMD W7900 on Windows using the HIP binaries, we encountered an error because the model attempted to offload some layers to our CPU's integrated graphics.

To get around this, we can specify which GPUs to run Llama.cpp on using the <device>--device</device> flag. We can list all available devices by running the following:

./llama-cli --list-devices

You should see an output similar to this one:

Available devices:

ROCm0: AMD Radeon RX 7900 XT (20464 MiB, 20314 MiB free

ROCm1: AMD Radeon(TM) Graphics (12498 MiB, 12347 MiB free)

Note that, depending on whether you're using the HIP, Vulkan, CUDA, or OpenCL, backend device names are going to be different. For example if you're using CUDA, you might see CUDA0 and CUDA1.

We can now launch Llama.cpp using our preferred GPU by running

./llama-cli -hfr bartowski/Qwen_Qwen3-8B-GGUF:Q4_K_M -ngl 37 --device ROCm0

Serving up your model:

As great as the CLI-based chat interface is, it's not necessarily the most convenient way to interact with Llama.cpp. Instead, you might want to connect it up to a graphical user interface (GUI) instead.

Thankfully, Llama.cpp includes an API server that can be connected to any app that supports OpenAI-compatible endpoints, like Jan or Open WebUI. If you just want a basic GUI, you don't need to do anything special, just launch the model with

llama-server

instead.

./llama-server -hfr bartowski/Qwen_Qwen3-8B-GGUF:Q4_K_M -ngl 37

After a few moments, you should be able to open the web GUI by navigating to

http://localhost:8080

in your web browser.

By default, launching a model with llama-server will start a basic web interface for chatting with models - Click to enlarge

If you want to access the server from another device, you'll need to expose the server to the rest of your network, by setting the

--host

address to

0.0.0.0

, and, if you want to use a different port, you'd append the

--port

flag. If you're going to make your server available to strangers on the Internet or a large network, we also recommend setting an

--api-key

flag.

./llama-server -hfr bartowski/Qwen_Qwen3-8B-GGUF:Q4_K_M -ngl 37 --host 0.0.0.0 --port 8000 --api-key top-secret

Now the API will be available at:

API address:

http://ServerIP:8080/v1

The API key should be passed to the URL as a bearer token if you're writing your own tool. Some other tools such as OpenWeb UI have a field where you can enter the key and the client application takes care of the rest.

Editor's note:

For most home users, this should be relatively safe as you should be deploying the model from behind your router's firewall. However, if you're running Llama.cpp in the cloud, you'll definitely want to lock down your firewall first.

Where to find models

Llama.cpp works with most models quantized using the GGUF format. These models can be found on a variety of model repos, with Hugging Face being among the most popular.

If you're looking for a particular model, it's worth checking profiles like

Bartowski

,

Unsloth

, and

GGML-Org

as they're usually among the first to have GGUF quants of new models.

If you're using Hugging Face, downloading them can be done directly from Llama.cpp. In fact, this is how we pulled down Qwen3-8B in the earlier step and requires specifying the model repo and the specific quantization level you'd prefer.

For example

-hfr bartowski/Qwen_Qwen3-8B-GGUF:Q8_0

would pull down an 8-bit quantized version of the model while

-hfr bartowski/Qwen_Qwen3-8B-GGUF:IQ3_XS

would download a 3-bit i-quant.

Generally speaking, smaller quants require fewer resources to run, but also tend to be lower quality.

Quantizing your own models

If the model you're looking for isn't already available as a GGUF, you may be able to create your own. Llama.cpp provides tools for converting models to GGUF format and then quantizing them from 16-bits to a lower precision (usually 8-2 bits) so they can run on lesser hardware.

To do this, you'll need to clone the Llama.cpp repo and install a recent version of Python. If you're running Windows, we actually find it's easier to do this step in Windows Subsystem for Linux (WSL) rather than trying to wrangle Python packages natively. If you need help setting up WSL, Microsoft's set up guide can be found

here

.

git clone https://github.com/ggml-org/llama.cpp.git

cd llama.cpp

Next, we'll need to create a Python virtual environment and install our dependencies. If you don't already have

python3-pip

and

python3-venv

installed you'll want to grab those first.

sudo apt install python3-pip python3-venv

Then create a virtual environment and activate by running:

python3 -m venv llama-cpp

source llama-cpp/bin/activate

With that out of the way we can install the Python dependencies with:

pip install -r requirements.txt

From there, we can use the

convert_hf_to_gguf.py

script to convert a safe tensors model, in this case Microsoft's Phi4, to a 16-bit GGUF file.

python3.12 convert_hf_to_gguf.py --remote microsoft/phi-4 --outfile phi4-14b-FP16.gguf

Unless you've got a multi-gig interconnection it will take a few minutes to download. At native precision, Phi 4 is nearly 30GB. If you run into an error trying to download a model like Llama or Gemma, you likely need to request permission on Hugging Face first and sign in using the

huggingface-cli

huggingface-cli log-in

From here, we can quantize the model to the desired bit-width. We'll use

Q4_K_M

quantization since it cuts the model size by nearly three quarters without sacrificing too much quality. You can find a full list of available quants

here

. Or by running

llama-quantize --help

.

./llama-quantize phi4-14b-FP16.gguf phi4-14b-Q4_K_M.gguf q4_k_m

To test the model, we can launch

llama-cli

but rather than using

-hfr

to select a Hugging Face repo, we'll use

-m

instead and point it at our newly quantized model.

llama-cli -m phi4-14b-Q4_K_M.gguf -ngl 99

If you'd like to learn more about quantization, we've got an entire guide dedicated to model compression including how to measure and minimize quality losses, which you can find

here

.

Building from source

On the off chance that the Llama.cpp doesn't offer a precompiled binary for your hardware or operating system, you'll need to build the app from source.

The Llama.cpp dev team maintains comprehensive documentation on how to build from source on every operating system and compute runtime, be it CUDA, HIP, SYCL, CANN, MUSA, or something else. Regardless of which you're building for, you'll want to make sure you have the latest drivers and runtimes installed and configured first.

For this demonstration, we'll be building Llama.cpp for both an 8GB Raspberry Pi 5 and an x86-based Linux box with an Nvidia GPU since there are no pre-compiled binaries for either. For our host operating system, we'll be using Ubuntu Server (25.04 for the RPI and 24.04 for the PC).

To get started, we'll install a couple of dependencies using apt.

sudo apt install git cmake build-essential libcurl4-openssl-dev

Next we can clone the repo from GitHub and open the directory.

git clone https://github.com/ggml-org/llama.cpp.git

cd llama.cpp

From there, building the Llama.cpp is pretty simple. The devs have an entire page dedicated to

building from source

with instructions for everything SYCL and CUDA to HIP and even Huawei's CANN and Moore Threads MUSA runtimes.

Building LLama.cpp on the RPI5:

For the Raspberry Pi 5, we can use the standard build flags. Note we've added the

-j 4

flag here to parallelize the process across the RPI's four cores.

cmake -B build

cmake --build build --config Release -j 4

Building Llama.cpp for x86 and CUDA:

For the x86 box, we'll need to make sure that the Nvidia drivers and CUDA tool kit are installed first by running:

sudo apt install nvidia-driver-570-server nvidia-cuda-toolkit

sudo reboot

After the system has rebooted, open the llama.cpp folder and build it with CUDA support by running:

cd llama.cpp

cmake -B build -DGGML_CUDA=ON

cmake --build build --config Release

Building for a different system or running into trouble? Check out Llama.cpp's

docs

.

Completing the installation:

Regardless of which system you're building on, it could take a couple minutes to complete. Once they are finished, binaries will be saved in the

/build/bin/

folder.

You can run these binaries directly from this directory, or complete the installation by copying them to your

/usr/bin

directory.

sudo cp /build/bin/ /usr/bin/

If everything worked correctly, we should be able to spin up Google's itty bitty

new language model

, Gemma 3 270M, by running:

llama-cli -hfr bartowski/google_gemma-3-270m-it-qat-GGUF:bf16

Would we recommend running LLMs on a Raspberry Pi? Not really, but at least now you know you can.

Performance tuning

So far we've covered how to download, install, run, serve, and quantize models in Llama.cpp, but we've only scratched the surface of what it's capable of.

Run

llama-cli --help

and you'll see just how many levers to pull and knobs to turn there really are. So, lets take a look at some of the more useful flags at our disposal.

In this example, we've configured Llama.cpp to run OpenAI's gpt-oss-20b model along with a few addition flags to maximize performance. Let's break them down one by one.

./llama-server -hf ggml-org/gpt-oss-20b-GGUF --jinja -fa -c 16384 -np 2 --cache-reuse 256 -ngl 99

-fa

— enables Flash Attention on supported platforms which can dramatically speed up prompt processing times while also reducing memory requirements. We find that this is beneficial for most setups, but it's worth trying with and without to be sure.

-c 16384

— sets the model's context window, or short-term memory, to 16,384 tokens. If left unset, Llama.cpp defaults to 4,096 tokens which minimizes memory requirements, but means the model will start forgetting details once you exceed that threshold. If you've got memory to spare, we recommend setting this as high as you can without running into out-of-memory errors, up to the model limits. For gpt-oss, that's 131,072 tokens.

The larger the context window, the more RAM or VRAM is required to run the model. LMCache offers a

calculator

to help you determine how many tokens you can fit into your memory.

-np 2

— allows Llama.cpp to process up to two requests simultaneously, which can be useful in multi-user or when connecting Llama.cpp to code assistant tools like Cline or Continue, which might make multiple simultaneous requests for code competition or chat functionality. Note that the context window is divided by the number of parallel processes. In this example, each parallel process has an 8192 token context.

--cache-reuse 256

— setting this helps to avoid recomputing key-value caches speeding up prompt processing, particularly for extended multi-turn conversations. We recommend starting with 256 token chunks.

Hyperparameter tuning

For optimal performance and output quality, many model builders recommend setting sampling parameters, like temperature or min-p to specific values.

For example, Alibaba's Qwen team recommends setting temp to 0.7, top-p to 0.8, top-k to 20, and min-p to 0 when running many of its instruct models, like Qwen3-30B-A3B-Instruct-2507. Recommended hyperparameters can usually be found on the model card on repos like Hugging Face.

In a nutshell, these parameters influence which tokens the model selects from the probability curve. Temperature is one of the easiest to understand, as setting it lower usually results in less creative and more deterministic outputs while setting it higher will result in more adventurous results.

Many applications, like Open WebUI, LibreChat, and Jan allow overriding them via the API, which they (or you) can access at localhost:8080 if you are running llama-server. However, for applications that don't, it can be helpful to set these when spinning up the model in Llama.cpp.

For example, for Qwen 3 instruct models, we'd run something like the following. (Note this particular model requires a little over 20GB of memory so you if want to test it, you may need to swap out Qwen for a smaller model.)

./llama-server -hfr bartowski/Qwen_Qwen3-30B-A3B-Instruct-2507-GGUF:Q4_K_M --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0

You can find a full list of available sampling parameters by running:

./llama-cli --help

For more information on how sampling parameters impact output generation, check out Amazon Web Services' explainer

here

.

Boosting performance with speculative decoding

One of the features in Llama.cpp that you won't see in other model runners, like Ollama, is support for speculative decoding. This process can speed up token generation in highly repetitive workloads, like code generation, by using a small draft model to predict the outputs of a larger, more accurate one.

This approach does require a compatible draft model, usually one from the same family as your primary one. In this example we're using speculative decoding to speed up Alibaba's Qwen3-14B model using the 0.6B variant as our drafter.

./llama-server -hfr Qwen/Qwen3-14B-GGUF:Q4_K_M -hfrd Qwen/Qwen3-0.6B-GGUF:Q8_0 -c 4096 -cd 4096 -ngl 99 -ngld 99 --draft-max 32 --draft-min 2 --cache-reuse 256 -fa

To test, we can ask the model to generate a block of text or code. In our testing with speculative decoding enabled, generation rates were roughly on par with running Qwen3-14B on its own. However, when we requested a minor change to that text or code, performance roughly doubled jumping from around 60 tok/s to 117 tok/s.

If you'd like to know more about how speculative decoding works in Llama.cpp, you can find our deep dive

here

.

Splitting big models between CPUs and GPUs

One of Llama.cpp's most valuable features is its ability to split large models between CPU and GPUs. So long as you have enough memory between your DRAM and VRAM to fit the model weights (and the OS), there's a good chance you'll be able to run it.

As we alluded to earlier, the easiest way to do this is to slowly increase the number layers offloaded to the GPU (

-ngl

) until you get an out-of-memory error and then back off a bit.

For example, if you had a GPU with 20GB of vRAM and 32GB of DDR5 and wanted to run Meta's Llama 3.1 70B model at 4-bit precision, which requires a little over 42GB of memory, you might offload 40 of the layers to the GPU and run the rest on the CPU.

./llama-server -hfr bartowski/Meta-Llama-3.1-70B-Instruct-GGUF -ngl 40

While the model runs, performance won't be great — in our testing we got about 2 tok/s.

However, thanks to a relatively small number of active parameters in mixture of expert (MoE) models such as gpt-oss, its actually possible to get decent performance, even when running far larger models.

By taking advantage of Llama.cpp's MoE expert offload features, we were able to get OpenAI's 120 billion parameter gpt-oss model running at a rather-respectable 20 tok/s on a system with a 20GB GPU and 64GB of DDR4 3200 MT/s memory.

./llama-server -hf ggml-org/gpt-oss-120b-GGU -fa -c 32768 --jinja -ngl 999 --n-cpu-moe 26

In this case, we set

-ngl

to

999

and used the

--n-cpu-moe

parameter to offload progressively more expert layers to the CPU until Llama.cpp stopped throwing out-of-memory errors.

Tool calling

If your workload requires it, Llama.cpp can also parse tool calls from OpenAI compatible API Endpoints like Open WebUI or Cline. You'd need tools if you want to bring in an outside feature such as a clock, a calculator, or to check the status of Proxmox cluster.

We previously built a tool to generate status reports from Proxmox VE.

Enabling tool calling varies from model to model. For most popular models, including gpt-oss, nothing special is required. Simply append the

--jinja

flag and you're off to the races.

./llama-server -hf ggml-org/gpt-oss-20b-GGUF --jinja

Others models, like DeepSeek R1, may require setting a chat-template manually when launching the model. For example:

./llama-server --jinja -fa -hf bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF:Q4_K_M \

--chat-template-file models/templates/llama-cpp-deepseek-r1.jinja

Tool calling is a whole can of worms unto itself, so, if you're interested in learning more, checking out our functional calling and Model Context Protocol deep dives

here

and

here

.

Summing up

While Llama.cpp may be one of the most comprehensive model runners out there — we've only discussed a fraction of all the app entails — we understand it can quite daunting for those dipping their toes into local LLMs for the first time. This is one of the reasons why it's taken us so long to do a hands-on of the app, and why we think that simpler apps like

Ollama

and

LM Studio

are still valuable.

So now that you've gotten your head wrapped around Llama.cpp you might be wondering how LLMs are

actually deployed

in production, or maybe how to get started with

image generation

. We've got guides for both. ®


url: https://www.reddit.com/r/Rag/comments/1n1iqy3/humanlike_rag_without_vectors/
title: Human-like RAG – without vectors : r/Rag
added_at: 2025-08-28 16:44:06
thumb_url: https://www.redditstatic.com/shreddit/assets/favicon/192x192.png
content_html:
<p>
      Great question. Here’s my take:
    </p><p>
      - PageIndex builds a tree-like hierarchy (like a smart table of contents). It follows the natural structure of a document — sections, subsections, and paragraphs — and uses reasoning to navigate down the right branch. Ideal where hierarchy matters (e.g. financial filings, research papers).
    </p><p>
      - Graph-based solutions (like GraphRAG) connect many different entities (facts, people, concepts, etc) into a network. Flexible, but can get messy — nodes can be very different kinds of things, and the structure doesn’t match how humans normally read or organize a document.
    </p><p>
      The main drawback of graph-based RAG is that it loses document hierarchy and context, while PageIndex preserves them, making reasoning and retrieval feel more natural.
    </p>

content_text:
Great question. Here’s my take:

- PageIndex builds a tree-like hierarchy (like a smart table of contents). It follows the natural structure of a document — sections, subsections, and paragraphs — and uses reasoning to navigate down the right branch. Ideal where hierarchy matters (e.g. financial filings, research papers).

- Graph-based solutions (like GraphRAG) connect many different entities (facts, people, concepts, etc) into a network. Flexible, but can get messy — nodes can be very different kinds of things, and the structure doesn’t match how humans normally read or organize a document.

The main drawback of graph-based RAG is that it loses document hierarchy and context, while PageIndex preserves them, making reasoning and retrieval feel more natural.


url: https://www.reddit.com/r/Rag/comments/1n23re5/ann_big_news_for_text_processing_chunkletpy_v140/
title: [ANN] 🚀 Big news for text processing! chunklet-py v1.4.0 is officially out! 🎉 : r/Rag
added_at: 2025-08-28 16:45:36
thumb_url: https://www.redditstatic.com/shreddit/assets/favicon/192x192.png
content_html:
<p>
      We've rebranded from 'chunklet' to 'chunklet-py' to make it easier to find our powerful text chunking library. But that's not all! This release is packed with features designed to make your workflow smoother and more efficient:
    </p><p>
      ✨ <strong>Enhanced Batch Processing:</strong> Now effortlessly chunk entire directories of <code>.txt</code> and <code>.md</code> files with <code>--input-dir</code>, and save each chunk to its own file in a specified <code>--output-dir</code>. 💡 <strong>Smarter CLI:</strong> Enjoy improved readability with newlines between chunks, clearer error messages, and a heads-up about upcoming changes with our new deprecation warning. ⚡️ <strong>Faster Startup:</strong> We've optimized <code>mpire</code> imports for quicker application launch times.
    </p><p>
      Get the latest version and streamline your text processing tasks today!
    </p><p>
<strong>Links:</strong>
</p><h1>chunklet #python #NLP #textprocessing #opensource #newrelease</h1>

content_text:
We've rebranded from 'chunklet' to 'chunklet-py' to make it easier to find our powerful text chunking library. But that's not all! This release is packed with features designed to make your workflow smoother and more efficient:

✨

Enhanced Batch Processing:

Now effortlessly chunk entire directories of

.txt

and

.md

files with

--input-dir

, and save each chunk to its own file in a specified

--output-dir

. 💡

Smarter CLI:

Enjoy improved readability with newlines between chunks, clearer error messages, and a heads-up about upcoming changes with our new deprecation warning. ⚡️

Faster Startup:

We've optimized

mpire

imports for quicker application launch times.

Get the latest version and streamline your text processing tasks today!

Links:

chunklet #python #NLP #textprocessing #opensource #newrelease


url: https://docs.unsloth.ai/basics/gemma-3-how-to-run-and-fine-tune
title: Gemma 3: How to Run & Fine-tune | Unsloth Documentation
added_at: 2025-08-28 17:29:09
content_html:
<p>This site uses cookies to deliver its service and to analyze traffic. By browsing this site, you accept the <a href="https://policies.gitbook.com/privacy/cookies" rel="noopener noreferrer" target="_blank">privacy policy</a>.</p><p>AcceptReject</p>

content_text:
This site uses cookies to deliver its service and to analyze traffic. By browsing this site, you accept the

privacy policy

.

AcceptReject

thumb_url:
https://docs.unsloth.ai/~gitbook/image?url=https%3A%2F%2F2815821428-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Forganizations%252FHpyELzcNe0topgVLGCZY%252Fsites%252Fsite_mXXTe%252Fsocialpreview%252FOeFDVbzp4cgwaId7PbvL%252Funsloth%2520docs%2520pic.png%3Falt%3Dmedia%26token%3Df34a6bcc-db15-449a-bc2d-e0e8f52fe953&width=1200&height=630&sign=3e009e0f&sv=2


title: 5 books that changed my engineering career forever
added_at: 2025-08-30 16:35:35
url:
https://open.substack.com/pub/techworldwithmilan/p/5-books-behind-my-growth-as-a-cto?r=3tujkj&utm_medium=ios

content_html:
<p>I love books. When I look at the 5000 books on my bookshelf, I see more than paper and ink; I see <strong>a personal board of mentors I never had</strong>. Over the years, as an avid reader, these books have <em>quietly</em> reshaped how I approach coding, leadership, and even thinking itself. </p><p>Early in my career, I assumed improving my skills was all about writing more code or learning the latest framework, like everyone else. But the most significant improvements in my growth came from hours spent <strong>flipping pages in some of the best books out there</strong>. </p><p>There’s something tactile and deliberate about reading, scribbling notes in the margins, dog-earing pages, that forces me to slow down and reflect. </p><p>In this newsletter issue, I want to share five books that influenced my journey from engineer to CTO the most. Each of <strong>these books contributed beyond just technical knowledge</strong>; they challenged my assumptions, introduced new mental models, and ultimately changed how I work and lead teams.</p><figure><img loading="lazy" referrerpolicy="no-referrer" src="https://substackcdn.com/image/fetch/$s_!TMUC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01c22b39-796c-4ba1-bc45-cf8039816a85_3008x2151.jpeg"/><figcaption>Books that improved my life (note: <em>note all are in the image</em>)</figcaption></figure><p>Here, I’ll explain <em>why each book mattered to me</em> and <em>how it changed my approach</em> as a technologist and leader, from timeless principles of personal effectiveness to systems architecture. </p><p>Together, these books became my off-hours crash course in becoming a better engineer, architect, and decision-maker. </p><p>So, let’s dive in.</p><p></p><hr/><p><em>JetBrains is excited to unveil the public preview of the ReSharper and AI Assistant extensions for VS Code. <a href="https://jb.gg/rs-vsc-mm-newsletter" rel="" target="_blank">Install them today</a>, try out their powerful new features, and share your feedback to help shape their evolution.</em></p><p><a href="https://jb.gg/rs-vsc-mm-newsletter" rel="" target="_blank">Try it out!</a></p><p></p><hr/><p><strong><a href="https://newsletter.techworld-with-milan.com/p/sponsorship-of-tech-world-with-milan" rel="" target="_blank">Sponsor this newsletter</a></strong></p><p><strong>Authors</strong>: <em>David Thomas, Andrew Hunt</em></p><p>One book I always recommend to developers is "<strong><a href="https://amzn.to/3GZ6u0K" rel="" target="_blank">The Pragmatic Programmer</a></strong>" by David Thomas and Andrew Hunt. Although it is not new, it builds on a foundation that is important today and will continue to be. </p><p>When it was released in 1999, it was revolutionary for me, as this book wrote about the things that make developers professionals in a highly unregulated profession. When I read it, I understand that.</p><p>Here are a few lessons that stayed with me:</p><ol><li><p><strong>Write good enough software</strong>. Striving for perfection can lead to never-ending projects. Instead, aim for "good enough" and iterate quickly. Deliver value early and often (all of this was before Scrum). Write code that works, prove it by writing tests, and ensure they are executed automatically. </p></li><li><p><strong>Use Tracer bullets</strong>. Use tracer code or prototypes to validate your understanding of a problem. This helps in clarifying requirements before fully implementing a feature. This was all before the MVP approach.</p></li><li><p><strong>Don’t blame, fix!</strong> Sometimes, you will find bad code that someone else wrote. Sometimes, you will find bad code that you wrote some time ago. Pragmatic programmers don't say, "<em>I didn't write this, so I will not fix it</em>," but do something about it and open a debate in a team about why.</p></li><li><p><strong>Don’t live with broken windows</strong>. In the book, you will learn about the "<em>Broken Windows Theory</em>," which states that if something is broken, others will break it even more in the future. As a pragmatic programmer, you should try to fix all major problems you find in the system while working on it.</p></li><li><p><strong>Keep your knowledge up to date, always</strong>. Continuous learning is essential. Stay updated with new tools, technologies, and best practices. Learn at least one programming language or tech stack every year. Read technical and non-technical books every once in a while (yes!). This investment in your skills pays off in your career growth.</p></li><li><p><strong>Be DRY</strong>. Redundant code is more complex to maintain. When you notice duplication, refactor it into a shared function or module. This will promote consistency and reduce errors.</p></li></ol><p>Learn more in the book:</p><p><em>The Pragmatic Programmer</em> taught me that good engineering isn’t just about writing code that works; it’s about building code that <strong>stays</strong> working, adaptable, and clean for years to come.</p><p>Here is <strong>the unified mind map</strong> of the whole book with chapters and tips:</p><figure><img loading="lazy" referrerpolicy="no-referrer" src="https://substackcdn.com/image/fetch/$s_!lNTh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2b7b226f-7bc5-4a6d-b848-d6d89c3f034d_5682x4160.png"/><figcaption>“The Pragmatic Programmer”, the unified mind map</figcaption></figure><blockquote><p>ℹ️<em> A similar book to “The Pragmatic Programmer” is “<strong><a href="https://amzn.to/458Ziag" rel="" target="_blank">Code Complete</a></strong>”. This book provides similar valuable tips, but is a bit longer and harder to follow. </em></p><p><em>The third book in this group is “<strong><a href="https://amzn.to/4mkP8dt" rel="" target="_blank">The Clean Coder</a></strong>”, a manifesto of professionalism in software development by Robert Martin.</em></p></blockquote><p><strong>Author</strong>: <em>Martin Kleppman</em></p><p>Even with two decades in software, I found <a href="https://martin.kleppmann.com/" rel="" target="_blank">Martin Kleppmann</a>’s <em>Designing Data-Intensive Applications (DDIA)</em> a humbling and <strong>mind-expanding</strong> read. It’s often dubbed the <em>encyclopedia of modern data systems</em>, and for good reason. </p><p>Before DDIA, terms like <em>replication</em>, <em>partitioning</em>, and <em>CAP theorem</em> were things I thought I understood. Reading this book (twice) was like getting a masterclass in how data systems work. </p><p>We can say that it is one of <strong>the most influential books in backend engineering</strong>, and I can see why. </p><p>Here are a few things I learned in the book:</p><ol><li><p>One key takeaway was learning to <strong>recognize trade-offs</strong> in any architectural decision. For example, the book discusses how specific databases favor consistency over availability (and vice versa), and how <em>there’s no free lunch</em>; you must choose based on your requirements. This helped me to understand that there is no single best choice we should use for everything. Instead, I started asking “<em>What problem are we solving, and what trade-off fits best here</em>?” </p></li><li><p><strong>Focus on fundamentals</strong>. I also appreciated how DDIA emphasized the fundamentals: storage engines, indexing, and distributed consensus, which remain unchanged even when new products come along. Concretely, after reading, I revisited our data architecture and improved how we handle data replication and consistency in our systems. For instance, we had a microservice that occasionally served stale data under high load. Applying principles from DDIA, we refined our approach to <strong>replication lag and read-replicas</strong> so that the service could scale without sacrificing the freshness of data. </p></li><li><p><strong>Evaluating databases</strong>. I also became more rigorous in assessing databases: instead of going by vendor hype, I’d map each option to the use-case (does it need transactions? horizontal scale? etc.). Then, select the proper database type and vendor.</p></li><li><p><strong>Deep dive into distributed transactions</strong>. Kleppmann’s detailed explanation of concepts like <strong>distributed transactions and log-based streaming</strong> provided insights into properly diagnosing system behavior in complex distributed systems. </p></li></ol><p>More than anything, this book taught me to think in terms of <em>reliability, scalability, and maintainability</em> as first-class design goals. It shifted my mindset that system design is about making clear-headed trade-offs, not just following trends. </p><p>The result is I now approach new architecture decisions with far more confidence and clarity. DDIA truly “bridged the gap” between what I thought I knew and how things work under the hood.</p><p>It is worth mentioning that DIA isn't light reading; it's 500 pages of dense material. People who should read it are: mid-career engineers designing systems, or anyone preparing for systems design interviews. </p><p>Still, I consider it not suitable for junior developers or those looking for quick tutorials.</p><p>Here are my notes from the book:</p><p>➡️ Read my <strong>full book review</strong>:</p><p><strong>Author:</strong> <em>John Ousterhout</em></p><p>As an engineer, I’ve seen how <strong>code complexity</strong> can become the silent killer of productivity (and projects). Prof. John Ousterhout’s <em><a href="https://amzn.to/3IwzyK8" rel="" target="_blank">A Philosophy of Software Design</a></em> gave me a vocabulary and mindset to attack this problem at the source. </p><p>The book is relatively short but packed with wisdom on how to break down systems into simpler, more manageable modules and <strong>design for readability and maintainability</strong>. Ousterhout defines complexity as anything related to code that makes it harder to modify or reason about, and then systematically shows how to reduce it. </p><p>He argues that code should be written for <strong>ease of reading</strong> by humans, not just ease of writing for the compiler. This gave me a new lens for code reviews. I now ask, <em>Will someone understand this in a year?</em> - and if not, we need to simplify.</p><p>One of the most powerful concepts I learned was the idea of <strong>“deep modules”</strong>: modules with simple interfaces but rich functionality under the hood (something contrary to the Clean Code). If an interface hides a lot of complexity, the module is doing its job well. </p><p>This was eye-opening; it flipped my approach to abstraction. Instead of pushing complexity onto callers (shallow modules), I now try to have more complexity inside modules, exposing only what’s necessary. This concept of <strong>deep vs. shallow modules</strong> fundamentally changed how I structure libraries and services, encouraging me to <em>spend more effort designing clean interfaces</em> upfront.</p><p>The book also provides a list of <em><strong>red flags</strong></em> for bad design, things like “classes that are too big” or “primitive obsession,” which I began using as an early warning system in code reviews. </p><p>I remember after reading Ousterhout’s philosophy, I went back to one of our core projects and refactored a tangle of classes into a few modules with clearer APIs. The result was that onboarding new developers became easier, and the number of bugs dropped because the code’s intent was more explicit. </p><p>Another quote from the book stayed with me: <em>“<strong>complexity is incremental</strong>”</em>, creeps in a little at a time, through small decisions. This prompted me to be more vigilant; I now tackle potential complexity at the design stage before it becomes larger. </p><p>In short, this book taught me that <strong>working code isn’t enough</strong>; if the code isn’t clean and simple, it’s a liability. That perspective has made me a more pragmatic architect who values quality over quick fixes.</p><p>➡️ Read my <strong>full book review</strong>:</p><p><strong>Author:</strong> <em>Daniel Kahneman</em></p><p><a href="https://en.wikipedia.org/wiki/Daniel_Kahneman" rel="" target="_blank">Daniel Kahneman</a> showed the world that we don’t think as rationally as we believe, and won a Nobel Prize for that (in behavioral economics).</p><p>His book <em><a href="https://amzn.to/41e3yUJ" rel="" target="_blank">Thinking, Fast and Slow</a></em> turned decades of research into a global bestseller, making his ideas accessible for everyone, not only for scientists.</p><figure><img loading="lazy" referrerpolicy="no-referrer" src="https://substackcdn.com/image/fetch/$s_!JWj7!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5024630d-38ef-47e0-bf75-5c9987c63206_1000x484.png"/><figcaption>Daniel Kahneman passed away in 2024. (Image: <a href="https://www.thekurzweillibrary.com/book-thinking-fast-and-slow" rel="" target="_blank">Kurzweil</a>)</figcaption></figure><p>This is one of the books that improved my critical thinking the most. Kahneman’s research focuses on how our minds operate in two modes: </p><ul><li><p><strong>System 1 (Fast)</strong>. Intuitive, automatic, emotional, our default setting. We use it for quick judgments and everyday decisions.</p></li><li><p><strong>System 2 (Slow)</strong>. Analytical, effortful, and deliberate, the system we use consciously for critical thinking. We use it for complex problems that require focused effort.</p></li></ul><p>As it turns out, we humans, even the best engineers, are often overconfident in our intuition and routinely trick ourselves without knowing it. </p><p>Reading this book was so revealing for me. I realized how many bugs, design flaws, or even management slip-ups in my career boiled down to cognitive biases and snap judgments.</p><p>Before, if I were debugging a problem or making a system design decision, I often trusted my gut feeling on the solution. <strong>Afterward, I started forcing myself to slow down.</strong> I’d ask: <em>What assumption am I making? Could I be jumping to conclusions?</em> </p><p>For example, if an outage happened, instead of immediately blaming the newest deployment (a System 1 reaction), I now methodically gather data and consider multiple hypotheses, a more System 2 approach. </p><p>Kahneman also introduced me to ideas like <strong>the anchoring effect</strong> and <strong>confirmation bias</strong>, which made me more willing to question my initial impressions.</p><p>One powerful concept that stuck with me was <strong>WYSIATI (What You See Is All There Is)</strong>. It describes how quickly we jump straight to conclusions, but often miss essential facts. “<em>Information that is not retrieved (even unconsciously) from memory might as well not exist</em>,” Kahneman states. Our minds search for good stories, yet they show a stubborn resistance to the quality and quantity of objective information that informs those narratives. </p><p>This phenomenon explained many frustrations I had experienced as a leader: why people made wrong (but fast) judgments, ignored contrary evidence, or stubbornly stuck to incomplete stories.</p><p>After reading Kahneman, I became intensely aware of how biases influence technical decisions and team dynamics. Perhaps most importantly, it changed <strong>how I interact with my team</strong>. I listen more carefully and I’m quicker to admit when I might be wrong, echoing Kahneman’s central lesson that <em>we’re all fallible, and that’s okay</em>. In meetings, I’ll say, “<em>This is my intuition speaking, but let’s verify it with data</em>.” </p><p>This book taught me that being a better engineer or CTO isn’t just about what you know; it’s about recognizing the <strong>blind spots in your thinking</strong> and compensating for them. It’s a mental toolkit that helped me make better decisions.</p><p>The book is <strong>an absolute must-read for everyone!</strong></p><p>➡️ Learn more about <strong>how biases and mental models shape your life</strong> and projects:</p><p><strong>Author</strong>:<em> Stephen Covey</em></p><p>Covey’s book taught me that effective engineering leadership starts with <em>personal</em> leadership. It gives a framework of seven habits, from being proactive and <strong>“beginning with the end in mind”</strong> to seeking win-win solutions and sharpening the saw (continuous improvement). </p><p>These habits might sound like common sense to most people, but reading this book was a wake-up call for me. Covey challenged me to take responsibility for my growth and to prioritize principles over urgency. </p><p>I consider it one of those books <strong>that changed my life and career </strong><em><strong>10x</strong></em><strong>.</strong></p><p>The concept of the <em><strong>Circle of Influence</strong></em><strong> vs. </strong><em><strong>Circle of Concern</strong></em> resonated with me (it originates from <a href="https://en.wikipedia.org/wiki/Stoicism" rel="" target="_blank">Stoicism</a>). I learned to focus my energy on things I can impact (like mastering a new technology or helping a teammate), rather than worrying about external factors I can’t control, e.g., other people's opinions.</p><figure><img loading="lazy" referrerpolicy="no-referrer" src="https://substackcdn.com/image/fetch/$s_!RISf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5bfcd962-d3ef-4b75-92dd-0ba82abc8e02_2659x1976.png"/><figcaption>3 Circles of Influence (inspired by Stephen Covey)</figcaption></figure><p>Another lesson was “<strong>seek first to understand, then to be understood</strong>,” which transformed how I communicate with my team (and listen actively). As Covey famously put it, <em>“Most people do not listen with the intent to understand; they listen with the intent to reply.” </em>When I read that line, I had to pause; it rang so true. </p><p>How often had I been in code reviews or planning meetings, already formulating my response instead of truly listening? Covey’s message: <strong>slow down and listen first. </strong></p><p>Now, when a teammate comes to me with an issue, <strong>I make a conscious effort to understand their perspective fully before I chime in</strong>. It’s improved trust and clarity in our conversations immensely.</p><p>Another habit that resonated with me was <strong>“Sharpen the saw,”</strong> which is Covey’s way of saying: invest in yourself continuously. It reminded me that taking time to read, learn new skills, or even recharge isn’t downtime; it’s an investment in effectiveness. I started blocking out a little time each week for self-improvement (sometimes that means reading a new book or exploring a tech I’m curious about). </p><p>It’s ironic, but by stepping away from the day-to-day hustle to <em>sharpen my saw</em>, I come back to work sharper and more productive.</p><p>After reading Covey, I began weekly <strong>planning with my long-term goals in mind</strong>, making sure that even as an engineer buried in code, I never lost sight of broader objectives. And this means creating <a href="https://www.patreon.com/techworld_with_milan/shop/how-to-set-and-achieve-any-goal-e-book-312287" rel="" target="_blank">proper long-term and short-term goals</a>.</p><figure><img loading="lazy" referrerpolicy="no-referrer" src="https://substackcdn.com/image/fetch/$s_!1MuY!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3fa1d566-aca0-48ef-adf7-5ac0c4eb7d66_1280x720.png"/><figcaption>7 Habits of Highly Effective People</figcaption></figure><p>This book’s <em>timeless principles</em> became <strong>the foundations of how I operate</strong>: it instilled discipline, proactivity, empathy, and a habit of continuous improvement that prepared me for the challenges of CTO life.</p><p>This book is like a mirror; it forces you to <strong>reflect on your habits and mindset</strong>. For me, that reflection led to tangible changes in how I lead my engineering team. </p><p>➡️ Read my book review, including Covey's book “<strong><a href="https://amzn.to/4l14RO0" rel="" target="_blank">The 8th Habit</a></strong>”:</p><p>Writing this, I noticed something: none of these books taught me a new programming language or how to use a specific framework. Yet each one was transformational in my growth as an engineer and a leader. </p><p>They expanded my perspective, from how I manage myself, to how I design software, to how I make decisions. Together, these books became a sort of <em>continuing education curriculum</em> that no formal degree or job training could offer. They reminded me that our <strong>shelf of books can double as a shelf of mentors</strong>, each one imparting lessons whenever we’re ready to absorb them.</p><p>If there’s one thing I hope you take away, it’s that investing in your growth doesn’t always mean writing more code or attending more meetings. Sometimes, it’s reading and reflecting. I often encourage younger engineers to build their own “<strong>mentor bookshelf</strong>” to find books that challenge the way they think. </p><p>The five I shared here are just a starting point, and your list might be different. What matters is staying curious and <strong>never thinking you’re done growing</strong>. </p><p>As a CTO, I’m still that engineer with a book in hand after work, excited about the next insight that will make me better. </p><p>So, what books will you reach for on your journey? </p><p>The answers might shape your career in ways you can’t imagine today. </p><p>Happy reading, and happy growing!</p><p>-- Milan</p><figure><img loading="lazy" referrerpolicy="no-referrer" src="https://substackcdn.com/image/fetch/$s_!ZGsN!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5d29e636-2e61-46d0-9e87-b540169abe0c_3063x2959.jpeg"/><figcaption>A part of my physical bookshelf (many more on Kindle)</figcaption></figure><p>➡️ Check my review of <strong>the best books</strong> I’ve read and recommend to everyone:</p><p></p><hr/><p>If you're interested in expanding your perspective on how systems work and how to make good decisions, I highly recommend the following two books:</p><p><strong>Author:</strong> <em>Donella Meadows</em> </p><p>I learned from the book that structure drives behavior, which is something most people don't see.</p><p>When sales drop, teams add features. Systems thinkers ask: what structure created this decline? Maybe the real problem is customer onboarding, not the product itself.</p><p>The book teaches you to spot feedback loops everywhere. Once you see them, you stop fixing symptoms and start fixing causes. </p><p>Learn to change the system, not the output.</p><p><strong>Authors:</strong> <em>Brian Christian and Tom Griffiths </em></p><p>This book applies computer science principles to everyday life. It shows how algorithms can optimize your personal and professional challenges, from decision-making to scheduling. </p><p>You will learn more about algorithms like optimal stopping, explore-exploit trade-offs, and caching, providing actionable advice for managing time, resources, and uncertainty.</p><p>What is the best hiring strategy? Look at the first 37% of candidates, then hire the next one who beats them all.</p><p>The "optimal stopping" rule applies everywhere: house hunting, dating, even picking a parking spot. And we already have this solved by computer science.</p><p>I learn from the book that algorithms aren't just for code, they're patterns for better everyday decisions.</p><figure><img loading="lazy" referrerpolicy="no-referrer" src="https://substackcdn.com/image/fetch/$s_!Y-zl!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1909385c-5f8a-4d3f-ae6a-c792488ee1bf_800x450.jpeg"/><figcaption>Two books that changed the way I think</figcaption></figure><p>Both books are packed with insights that resonate far beyond their topics. </p><p>I highly recommend it for engineers, managers, and anyone curious about improving their thinking.</p><p></p><hr/><p>Do you want to learn more from my journey with thousands of books? I prepared something special for you.</p><p>Here is the mini‑book, where you’ll find one‑page briefs on the 100 best books, organized by craft (architecture, leadership, productivity, thinking, wealth, and more), taken from my notes. </p><p>Each brief tags <strong>Why it mattered</strong> and <strong>How I used it</strong>, so you see the principle and the practice side‑by‑side.</p><p>I layered personal margin notes throughout, detailing what I challenged, where I failed, and the tweaks that ultimately worked.</p><p>You’ll also receive a bonus section, <strong>my four-step method for reading</strong> non-fiction so the ideas stick and compound (hint: it starts before you open the cover).</p><p>➡️ Get the book by <strong>subscribing to my newsletter</strong>:</p><p></p><hr/><ul><li><p><a href="https://www.patreon.com/techworld_with_milan/shop/ultimate-net-bundle-for-2025-1519389?utm_medium=clipboard_copy&amp;utm_source=copyLink&amp;utm_campaign=productshare_creator&amp;utm_content=join_link" rel="" target="_blank">📚 </a><strong><a href="https://www.patreon.com/techworld_with_milan/shop/ultimate-net-bundle-for-2025-1519389?utm_medium=clipboard_copy&amp;utm_source=copyLink&amp;utm_campaign=productshare_creator&amp;utm_content=join_link" rel="" target="_blank">The Ultimate .NET Bundle 2025</a></strong> 🆕. 500+ pages distilled from 30 real projects show you how to own modern C#, ASP.NET Core, patterns, and the whole .NET ecosystem. You also get 200+ interview Q&amp;As, a C# cheat sheet, and bonus guides on middleware and best practices to improve your career and land new .NET roles. <strong><a href="https://www.patreon.com/techworld_with_milan/shop/ultimate-net-bundle-for-2025-1519389?utm_medium=clipboard_copy&amp;utm_source=copyLink&amp;utm_campaign=productshare_creator&amp;utm_content=join_link" rel="" target="_blank">Join 1,000+ engineers</a></strong>. </p></li><li><p><a href="https://www.patreon.com/techworld_with_milan/shop/premium-resume-package-1721454?utm_medium=clipboard_copy&amp;utm_source=copyLink&amp;utm_campaign=productshare_creator&amp;utm_content=join_link" rel="" target="_blank">📦 </a><strong><a href="https://www.patreon.com/techworld_with_milan/shop/premium-resume-package-1721454?utm_medium=clipboard_copy&amp;utm_source=copyLink&amp;utm_campaign=productshare_creator&amp;utm_content=join_link" rel="" target="_blank">Premium Resume Package</a> 🆕</strong>. Built from over 300 interviews, this system enables you to craft a clear, job-ready resume quickly and efficiently. You get ATS-friendly templates (summary, project-based, and more), a cover letter, AI prompts, and bonus guides on writing resumes and prepping LinkedIn. <strong><a href="https://www.patreon.com/techworld_with_milan/shop/premium-resume-package-1721454?utm_medium=clipboard_copy&amp;utm_source=copyLink&amp;utm_campaign=productshare_creator&amp;utm_content=join_link" rel="" target="_blank">Join 500+ people</a></strong>. </p></li><li><p><a href="https://www.patreon.com/techworld_with_milan/shop/complete-tech-resume-reality-check-311008?utm_medium=clipboard_copy&amp;utm_source=copyLink&amp;utm_campaign=productshare_creator&amp;utm_content=join_link" rel="" target="_blank">📄 </a><strong><a href="https://www.patreon.com/techworld_with_milan/shop/complete-tech-resume-reality-check-311008?utm_medium=clipboard_copy&amp;utm_source=copyLink&amp;utm_campaign=productshare_creator&amp;utm_content=join_link" rel="" target="_blank">Resume Reality Check</a></strong>. Get a CTO-level teardown of your CV and LinkedIn profile. I flag what stands out, fix what drags, and show you how hiring managers judge you in 30 seconds. <strong><a href="https://www.patreon.com/techworld_with_milan/shop/complete-tech-resume-reality-check-311008?utm_medium=clipboard_copy&amp;utm_source=copyLink&amp;utm_campaign=productshare_creator&amp;utm_content=join_link" rel="" target="_blank">Join 100+ people</a></strong>.</p></li><li><p><a href="https://www.patreon.com/techworld_with_milan/shop/short-linkedin-content-creator-311232?utm_medium=clipboard_copy&amp;utm_source=copyLink&amp;utm_campaign=productshare_creator&amp;utm_content=join_link" rel="" target="_blank">📢 </a><strong><a href="https://www.patreon.com/techworld_with_milan/shop/short-linkedin-content-creator-311232?utm_medium=clipboard_copy&amp;utm_source=copyLink&amp;utm_campaign=productshare_creator&amp;utm_content=join_link" rel="" target="_blank">LinkedIn Content Creator Masterclass</a></strong>. I share the system that grew my tech following to over 100,000 in 6 months (now over 255,000), covering audience targeting, algorithm triggers, and a repeatable writing framework. Leave with a 90-day content plan that turns expertise into daily growth. <strong><a href="https://www.patreon.com/techworld_with_milan/shop/short-linkedin-content-creator-311232?utm_medium=clipboard_copy&amp;utm_source=copyLink&amp;utm_campaign=productshare_creator&amp;utm_content=join_link" rel="" target="_blank">Join 1,000+ creators</a></strong>.</p></li><li><p><a href="https://www.patreon.com/c/techworld_with_milan" rel="" target="_blank">✨ </a><strong><a href="https://www.patreon.com/c/techworld_with_milan" rel="" target="_blank">Join My Patreon</a></strong><a href="https://www.patreon.com/c/techworld_with_milan" rel="" target="_blank"> </a><strong><a href="https://www.patreon.com/c/techworld_with_milan" rel="" target="_blank">Community</a> and <a href="https://www.patreon.com/c/techworld_with_milan/shop" rel="" target="_blank">My Shop</a></strong>. Unlock every book, template, and future drop, plus early access, behind-the-scenes notes, and priority requests. Your support enables me to continue writing in-depth articles at no cost. <strong><a href="https://www.patreon.com/c/techworld_with_milan" rel="" target="_blank">Join 2,000+ insiders</a></strong>.</p></li><li><p><a href="https://newsletter.techworld-with-milan.com/p/coaching-services" rel="" target="_blank">🤝 </a><strong><a href="https://newsletter.techworld-with-milan.com/p/coaching-services" rel="" target="_blank">1:1 Coaching</a></strong> – Book a focused session to crush your biggest engineering or leadership roadblock. I’ll map next steps, share battle-tested playbooks, and hold you accountable. <strong><a href="https://newsletter.techworld-with-milan.com/p/coaching-services" rel="" target="_blank">Join 100+ coachees</a></strong>.</p></li></ul><p></p><hr/><p>If your company is interested in reaching an audience of founders, executives, and decision-makers, you may want to <strong><a href="https://newsletter.techworld-with-milan.com/p/sponsorship-of-tech-world-with-milan" rel="" target="_blank">consider advertising with us</a></strong>.</p><p></p><hr/><p>Share it with your friends by using the button below to get benefits (my books and resources).</p><p><a href="https://newsletter.techworld-with-milan.com/?utm_source=substack&amp;utm_medium=email&amp;utm_content=share&amp;action=share" rel="" target="_blank">Share Tech World With Milan Newsletter</a></p><p><a href="https://newsletter.techworld-with-milan.com/leaderboard" rel="" target="_blank">Track your referrals here</a>.</p><p></p><p></p>

content_text:
I love books. When I look at the 5000 books on my bookshelf, I see more than paper and ink; I see

a personal board of mentors I never had

. Over the years, as an avid reader, these books have

quietly

reshaped how I approach coding, leadership, and even thinking itself.

Early in my career, I assumed improving my skills was all about writing more code or learning the latest framework, like everyone else. But the most significant improvements in my growth came from hours spent

flipping pages in some of the best books out there

.

There’s something tactile and deliberate about reading, scribbling notes in the margins, dog-earing pages, that forces me to slow down and reflect.

In this newsletter issue, I want to share five books that influenced my journey from engineer to CTO the most. Each of

these books contributed beyond just technical knowledge

; they challenged my assumptions, introduced new mental models, and ultimately changed how I work and lead teams.

Books that improved my life (note:

note all are in the image

)

Here, I’ll explain

why each book mattered to me

and

how it changed my approach

as a technologist and leader, from timeless principles of personal effectiveness to systems architecture.

Together, these books became my off-hours crash course in becoming a better engineer, architect, and decision-maker.

So, let’s dive in.

JetBrains is excited to unveil the public preview of the ReSharper and AI Assistant extensions for VS Code.

Install them today

, try out their powerful new features, and share your feedback to help shape their evolution.

Try it out!

Sponsor this newsletter

Authors

:

David Thomas, Andrew Hunt

One book I always recommend to developers is "

The Pragmatic Programmer

" by David Thomas and Andrew Hunt. Although it is not new, it builds on a foundation that is important today and will continue to be.

When it was released in 1999, it was revolutionary for me, as this book wrote about the things that make developers professionals in a highly unregulated profession. When I read it, I understand that.

Here are a few lessons that stayed with me:

Write good enough software

. Striving for perfection can lead to never-ending projects. Instead, aim for "good enough" and iterate quickly. Deliver value early and often (all of this was before Scrum). Write code that works, prove it by writing tests, and ensure they are executed automatically.

Use Tracer bullets

. Use tracer code or prototypes to validate your understanding of a problem. This helps in clarifying requirements before fully implementing a feature. This was all before the MVP approach.

Don’t blame, fix!

Sometimes, you will find bad code that someone else wrote. Sometimes, you will find bad code that you wrote some time ago. Pragmatic programmers don't say, "

I didn't write this, so I will not fix it

," but do something about it and open a debate in a team about why.

Don’t live with broken windows

. In the book, you will learn about the "

Broken Windows Theory

," which states that if something is broken, others will break it even more in the future. As a pragmatic programmer, you should try to fix all major problems you find in the system while working on it.

Keep your knowledge up to date, always

. Continuous learning is essential. Stay updated with new tools, technologies, and best practices. Learn at least one programming language or tech stack every year. Read technical and non-technical books every once in a while (yes!). This investment in your skills pays off in your career growth.

Be DRY

. Redundant code is more complex to maintain. When you notice duplication, refactor it into a shared function or module. This will promote consistency and reduce errors.

Learn more in the book:

The Pragmatic Programmer

taught me that good engineering isn’t just about writing code that works; it’s about building code that

stays

working, adaptable, and clean for years to come.

Here is

the unified mind map

of the whole book with chapters and tips:

“The Pragmatic Programmer”, the unified mind map

ℹ️

A similar book to “The Pragmatic Programmer” is “

Code Complete

”. This book provides similar valuable tips, but is a bit longer and harder to follow.

The third book in this group is “

The Clean Coder

”, a manifesto of professionalism in software development by Robert Martin.

Author

:

Martin Kleppman

Even with two decades in software, I found

Martin Kleppmann

’s

Designing Data-Intensive Applications (DDIA)

a humbling and

mind-expanding

read. It’s often dubbed the

encyclopedia of modern data systems

, and for good reason.

Before DDIA, terms like

replication

,

partitioning

, and

CAP theorem

were things I thought I understood. Reading this book (twice) was like getting a masterclass in how data systems work.

We can say that it is one of

the most influential books in backend engineering

, and I can see why.

Here are a few things I learned in the book:

One key takeaway was learning to

recognize trade-offs

in any architectural decision. For example, the book discusses how specific databases favor consistency over availability (and vice versa), and how

there’s no free lunch

; you must choose based on your requirements. This helped me to understand that there is no single best choice we should use for everything. Instead, I started asking “

What problem are we solving, and what trade-off fits best here

?”

Focus on fundamentals

. I also appreciated how DDIA emphasized the fundamentals: storage engines, indexing, and distributed consensus, which remain unchanged even when new products come along. Concretely, after reading, I revisited our data architecture and improved how we handle data replication and consistency in our systems. For instance, we had a microservice that occasionally served stale data under high load. Applying principles from DDIA, we refined our approach to

replication lag and read-replicas

so that the service could scale without sacrificing the freshness of data.

Evaluating databases

. I also became more rigorous in assessing databases: instead of going by vendor hype, I’d map each option to the use-case (does it need transactions? horizontal scale? etc.). Then, select the proper database type and vendor.

Deep dive into distributed transactions

. Kleppmann’s detailed explanation of concepts like

distributed transactions and log-based streaming

provided insights into properly diagnosing system behavior in complex distributed systems.

More than anything, this book taught me to think in terms of

reliability, scalability, and maintainability

as first-class design goals. It shifted my mindset that system design is about making clear-headed trade-offs, not just following trends.

The result is I now approach new architecture decisions with far more confidence and clarity. DDIA truly “bridged the gap” between what I thought I knew and how things work under the hood.

It is worth mentioning that DIA isn't light reading; it's 500 pages of dense material. People who should read it are: mid-career engineers designing systems, or anyone preparing for systems design interviews.

Still, I consider it not suitable for junior developers or those looking for quick tutorials.

Here are my notes from the book:

➡️ Read my

full book review

:

Author:

John Ousterhout

As an engineer, I’ve seen how

code complexity

can become the silent killer of productivity (and projects). Prof. John Ousterhout’s

A Philosophy of Software Design

gave me a vocabulary and mindset to attack this problem at the source.

The book is relatively short but packed with wisdom on how to break down systems into simpler, more manageable modules and

design for readability and maintainability

. Ousterhout defines complexity as anything related to code that makes it harder to modify or reason about, and then systematically shows how to reduce it.

He argues that code should be written for

ease of reading

by humans, not just ease of writing for the compiler. This gave me a new lens for code reviews. I now ask,

Will someone understand this in a year?

- and if not, we need to simplify.

One of the most powerful concepts I learned was the idea of

“deep modules”

: modules with simple interfaces but rich functionality under the hood (something contrary to the Clean Code). If an interface hides a lot of complexity, the module is doing its job well.

This was eye-opening; it flipped my approach to abstraction. Instead of pushing complexity onto callers (shallow modules), I now try to have more complexity inside modules, exposing only what’s necessary. This concept of

deep vs. shallow modules

fundamentally changed how I structure libraries and services, encouraging me to

spend more effort designing clean interfaces

upfront.

The book also provides a list of

red flags

for bad design, things like “classes that are too big” or “primitive obsession,” which I began using as an early warning system in code reviews.

I remember after reading Ousterhout’s philosophy, I went back to one of our core projects and refactored a tangle of classes into a few modules with clearer APIs. The result was that onboarding new developers became easier, and the number of bugs dropped because the code’s intent was more explicit.

Another quote from the book stayed with me:

“

complexity is incremental

”

, creeps in a little at a time, through small decisions. This prompted me to be more vigilant; I now tackle potential complexity at the design stage before it becomes larger.

In short, this book taught me that

working code isn’t enough

; if the code isn’t clean and simple, it’s a liability. That perspective has made me a more pragmatic architect who values quality over quick fixes.

➡️ Read my

full book review

:

Author:

Daniel Kahneman

Daniel Kahneman

showed the world that we don’t think as rationally as we believe, and won a Nobel Prize for that (in behavioral economics).

His book

Thinking, Fast and Slow

turned decades of research into a global bestseller, making his ideas accessible for everyone, not only for scientists.

Daniel Kahneman passed away in 2024. (Image:

Kurzweil

)

This is one of the books that improved my critical thinking the most. Kahneman’s research focuses on how our minds operate in two modes:

System 1 (Fast)

. Intuitive, automatic, emotional, our default setting. We use it for quick judgments and everyday decisions.

System 2 (Slow)

. Analytical, effortful, and deliberate, the system we use consciously for critical thinking. We use it for complex problems that require focused effort.

As it turns out, we humans, even the best engineers, are often overconfident in our intuition and routinely trick ourselves without knowing it.

Reading this book was so revealing for me. I realized how many bugs, design flaws, or even management slip-ups in my career boiled down to cognitive biases and snap judgments.

Before, if I were debugging a problem or making a system design decision, I often trusted my gut feeling on the solution.

Afterward, I started forcing myself to slow down.

I’d ask:

What assumption am I making? Could I be jumping to conclusions?

For example, if an outage happened, instead of immediately blaming the newest deployment (a System 1 reaction), I now methodically gather data and consider multiple hypotheses, a more System 2 approach.

Kahneman also introduced me to ideas like

the anchoring effect

and

confirmation bias

, which made me more willing to question my initial impressions.

One powerful concept that stuck with me was

WYSIATI (What You See Is All There Is)

. It describes how quickly we jump straight to conclusions, but often miss essential facts. “

Information that is not retrieved (even unconsciously) from memory might as well not exist

,” Kahneman states. Our minds search for good stories, yet they show a stubborn resistance to the quality and quantity of objective information that informs those narratives.

This phenomenon explained many frustrations I had experienced as a leader: why people made wrong (but fast) judgments, ignored contrary evidence, or stubbornly stuck to incomplete stories.

After reading Kahneman, I became intensely aware of how biases influence technical decisions and team dynamics. Perhaps most importantly, it changed

how I interact with my team

. I listen more carefully and I’m quicker to admit when I might be wrong, echoing Kahneman’s central lesson that

we’re all fallible, and that’s okay

. In meetings, I’ll say, “

This is my intuition speaking, but let’s verify it with data

.”

This book taught me that being a better engineer or CTO isn’t just about what you know; it’s about recognizing the

blind spots in your thinking

and compensating for them. It’s a mental toolkit that helped me make better decisions.

The book is

an absolute must-read for everyone!

➡️ Learn more about

how biases and mental models shape your life

and projects:

Author

:

Stephen Covey

Covey’s book taught me that effective engineering leadership starts with

personal

leadership. It gives a framework of seven habits, from being proactive and

“beginning with the end in mind”

to seeking win-win solutions and sharpening the saw (continuous improvement).

These habits might sound like common sense to most people, but reading this book was a wake-up call for me. Covey challenged me to take responsibility for my growth and to prioritize principles over urgency.

I consider it one of those books

that changed my life and career

10x

.

The concept of the

Circle of Influence

vs.

Circle of Concern

resonated with me (it originates from

Stoicism

). I learned to focus my energy on things I can impact (like mastering a new technology or helping a teammate), rather than worrying about external factors I can’t control, e.g., other people's opinions.

3 Circles of Influence (inspired by Stephen Covey)

Another lesson was “

seek first to understand, then to be understood

,” which transformed how I communicate with my team (and listen actively). As Covey famously put it,

“Most people do not listen with the intent to understand; they listen with the intent to reply.”

When I read that line, I had to pause; it rang so true.

How often had I been in code reviews or planning meetings, already formulating my response instead of truly listening? Covey’s message:

slow down and listen first.

Now, when a teammate comes to me with an issue,

I make a conscious effort to understand their perspective fully before I chime in

. It’s improved trust and clarity in our conversations immensely.

Another habit that resonated with me was

“Sharpen the saw,”

which is Covey’s way of saying: invest in yourself continuously. It reminded me that taking time to read, learn new skills, or even recharge isn’t downtime; it’s an investment in effectiveness. I started blocking out a little time each week for self-improvement (sometimes that means reading a new book or exploring a tech I’m curious about).

It’s ironic, but by stepping away from the day-to-day hustle to

sharpen my saw

, I come back to work sharper and more productive.

After reading Covey, I began weekly

planning with my long-term goals in mind

, making sure that even as an engineer buried in code, I never lost sight of broader objectives. And this means creating

proper long-term and short-term goals

.

7 Habits of Highly Effective People

This book’s

timeless principles

became

the foundations of how I operate

: it instilled discipline, proactivity, empathy, and a habit of continuous improvement that prepared me for the challenges of CTO life.

This book is like a mirror; it forces you to

reflect on your habits and mindset

. For me, that reflection led to tangible changes in how I lead my engineering team.

➡️ Read my book review, including Covey's book “

The 8th Habit

”:

Writing this, I noticed something: none of these books taught me a new programming language or how to use a specific framework. Yet each one was transformational in my growth as an engineer and a leader.

They expanded my perspective, from how I manage myself, to how I design software, to how I make decisions. Together, these books became a sort of

continuing education curriculum

that no formal degree or job training could offer. They reminded me that our

shelf of books can double as a shelf of mentors

, each one imparting lessons whenever we’re ready to absorb them.

If there’s one thing I hope you take away, it’s that investing in your growth doesn’t always mean writing more code or attending more meetings. Sometimes, it’s reading and reflecting. I often encourage younger engineers to build their own “

mentor bookshelf

” to find books that challenge the way they think.

The five I shared here are just a starting point, and your list might be different. What matters is staying curious and

never thinking you’re done growing

.

As a CTO, I’m still that engineer with a book in hand after work, excited about the next insight that will make me better.

So, what books will you reach for on your journey?

The answers might shape your career in ways you can’t imagine today.

Happy reading, and happy growing!

-- Milan

A part of my physical bookshelf (many more on Kindle)

➡️ Check my review of

the best books

I’ve read and recommend to everyone:

If you're interested in expanding your perspective on how systems work and how to make good decisions, I highly recommend the following two books:

Author:

Donella Meadows

I learned from the book that structure drives behavior, which is something most people don't see.

When sales drop, teams add features. Systems thinkers ask: what structure created this decline? Maybe the real problem is customer onboarding, not the product itself.

The book teaches you to spot feedback loops everywhere. Once you see them, you stop fixing symptoms and start fixing causes.

Learn to change the system, not the output.

Authors:

Brian Christian and Tom Griffiths

This book applies computer science principles to everyday life. It shows how algorithms can optimize your personal and professional challenges, from decision-making to scheduling.

You will learn more about algorithms like optimal stopping, explore-exploit trade-offs, and caching, providing actionable advice for managing time, resources, and uncertainty.

What is the best hiring strategy? Look at the first 37% of candidates, then hire the next one who beats them all.

The "optimal stopping" rule applies everywhere: house hunting, dating, even picking a parking spot. And we already have this solved by computer science.

I learn from the book that algorithms aren't just for code, they're patterns for better everyday decisions.

Two books that changed the way I think

Both books are packed with insights that resonate far beyond their topics.

I highly recommend it for engineers, managers, and anyone curious about improving their thinking.

Do you want to learn more from my journey with thousands of books? I prepared something special for you.

Here is the mini‑book, where you’ll find one‑page briefs on the 100 best books, organized by craft (architecture, leadership, productivity, thinking, wealth, and more), taken from my notes.

Each brief tags

Why it mattered

and

How I used it

, so you see the principle and the practice side‑by‑side.

I layered personal margin notes throughout, detailing what I challenged, where I failed, and the tweaks that ultimately worked.

You’ll also receive a bonus section,

my four-step method for reading

non-fiction so the ideas stick and compound (hint: it starts before you open the cover).

➡️ Get the book by

subscribing to my newsletter

:

📚

The Ultimate .NET Bundle 2025

🆕. 500+ pages distilled from 30 real projects show you how to own modern C#, ASP.NET Core, patterns, and the whole .NET ecosystem. You also get 200+ interview Q&As, a C# cheat sheet, and bonus guides on middleware and best practices to improve your career and land new .NET roles.

Join 1,000+ engineers

.

📦

Premium Resume Package

🆕

. Built from over 300 interviews, this system enables you to craft a clear, job-ready resume quickly and efficiently. You get ATS-friendly templates (summary, project-based, and more), a cover letter, AI prompts, and bonus guides on writing resumes and prepping LinkedIn.

Join 500+ people

.

📄

Resume Reality Check

. Get a CTO-level teardown of your CV and LinkedIn profile. I flag what stands out, fix what drags, and show you how hiring managers judge you in 30 seconds.

Join 100+ people

.

📢

LinkedIn Content Creator Masterclass

. I share the system that grew my tech following to over 100,000 in 6 months (now over 255,000), covering audience targeting, algorithm triggers, and a repeatable writing framework. Leave with a 90-day content plan that turns expertise into daily growth.

Join 1,000+ creators

.

✨

Join My Patreon

Community

and

My Shop

. Unlock every book, template, and future drop, plus early access, behind-the-scenes notes, and priority requests. Your support enables me to continue writing in-depth articles at no cost.

Join 2,000+ insiders

.

🤝

1:1 Coaching

– Book a focused session to crush your biggest engineering or leadership roadblock. I’ll map next steps, share battle-tested playbooks, and hold you accountable.

Join 100+ coachees

.

If your company is interested in reaching an audience of founders, executives, and decision-makers, you may want to

consider advertising with us

.

Share it with your friends by using the button below to get benefits (my books and resources).

Share Tech World With Milan Newsletter

Track your referrals here

.

thumb_url:
https://substackcdn.com/image/fetch/$s_!TMUC!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F01c22b39-796c-4ba1-bc45-cf8039816a85_3008x2151.jpeg


url: https://github.com/rasbt/reasoning-from-scratch
content_text: rasbt/reasoning-from-scratch You can’t perform that action at this time.
added_at: 2025-08-31 21:56:24
thumb_url: https://repository-images.githubusercontent.com/942696497/552ca932-7f7c-4436-a111-c9ece9cff77d
title:
GitHub - rasbt/reasoning-from-scratch: Implement a reasoning LLM in PyTorch from scratch, step by step

content_html:
<p></p>





<h1>rasbt/reasoning-from-scratch</h1>














<p>
    You can’t perform that action at this time.
  </p>









<p></p>


title: Memory-R1: How Reinforcement Learning Supercharges LLM Memory Agents - MarkTechPost
added_at: 2025-09-01 03:02:52
url:
https://www.marktechpost.com/2025/08/28/memory-r1-how-reinforcement-learning-supercharges-llm-memory-agents/?amp

content_html:
<figure><a href="https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.49.48-PM.png" rel="noopener noreferrer" target="_blank"><img alt="" loading="lazy" referrerpolicy="no-referrer" src="https://www.marktechpost.com/2025/08/28/memory-r1-how-reinforcement-learning-supercharges-llm-memory-agents/?amp"/><img alt="" loading="lazy" referrerpolicy="no-referrer" sizes="(max-width: 1694px) 100vw, 1694px" src="https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.49.48-PM.png" srcset="https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.49.48-PM.png 1694w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.49.48-PM-300x190.png 300w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.49.48-PM-1024x649.png 1024w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.49.48-PM-768x487.png 768w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.49.48-PM-1536x974.png 1536w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.49.48-PM-662x420.png 662w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.49.48-PM-150x95.png 150w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.49.48-PM-696x441.png 696w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.49.48-PM-1068x677.png 1068w" title="Screenshot 2025-08-28 at 8.49.48 PM"/></a><figcaption>https://arxiv.org/abs/2508.19828</figcaption></figure>

<p>Large language models (LLMs) now stand at the center of countless AI breakthroughs—chatbots, coding assistants, question answering, creative writing, and much more. But despite their prowess, they remain <strong>stateless</strong>: each query arrives with no memory of what came before. Their fixed <strong>context windows</strong> mean they can’t accumulate persistent knowledge across long conversations or multi-session tasks, and they struggle to reason over complex histories. Recent solutions, like retrieval-augmented generation (RAG), append past information to prompts, but this often leads to noisy, unfiltered context—flooding the model with too much irrelevant detail or missing crucial facts.</p>
<p>A team of researchers from University of Munich, Technical University of Munich, University of Cambridge and University of Hong Kong introduced <strong>Memory-R1</strong>, a framework that teaches LLM agents to decide what to remember and how to use it. Its LLM agent learns to <strong>actively manage and utilize external memory</strong>—deciding what to add, update, delete, or ignore, and filtering out noise when answering questions. The breakthrough? It trains these behaviors with <strong>reinforcement learning (RL)</strong>, using only outcome-based rewards, so it needs minimal supervision and <strong>generalizes robustly across models and tasks</strong>.</p>
<h3><strong>But Why LLMs Struggle with Memory</strong>?</h3>
<p>Imagine a multi-session conversation: in the first session, a user says, “I adopted a dog named Buddy.” Later, they add, “I adopted another dog named Scout.” Should the system <strong>replace</strong> the first statement with the second, <strong>merge</strong> them, or <strong>ignore</strong> the update? Vanilla memory pipelines often fail—they might erase “Buddy” and add “Scout,” misinterpreting the new information as a contradiction rather than a consolidation. Over time, such systems lose coherence, fragmenting user knowledge rather than evolving it.</p>
<p><strong>RAG systems</strong> retrieve information but don’t filter it: irrelevant entries pollute reasoning, and the model gets distracted by noise. <strong>Humans</strong>, by contrast, retrieve widely but then <strong>selectively filter</strong> what matters. Most AI memory systems are <strong>static</strong>, relying on handcrafted heuristics for what to remember, rather than learning from feedback.</p>

<figure><img alt="" loading="lazy" referrerpolicy="no-referrer" src="https://www.marktechpost.com/2025/08/28/memory-r1-how-reinforcement-learning-supercharges-llm-memory-agents/?amp"/><img alt="" loading="lazy" referrerpolicy="no-referrer" sizes="(max-width: 1024px) 100vw, 1024px" src="https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.42-PM-1-1024x645.png" srcset="https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.42-PM-1-1024x645.png 1024w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.42-PM-1-300x189.png 300w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.42-PM-1-768x484.png 768w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.42-PM-1-1536x968.png 1536w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.42-PM-1-667x420.png 667w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.42-PM-1-150x95.png 150w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.42-PM-1-696x439.png 696w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.42-PM-1-1068x673.png 1068w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.42-PM-1.png 1622w"/><figcaption>https://arxiv.org/pdf/2508.19828</figcaption></figure>
<h3><strong>The Memory-R1 Framework</strong></h3>
<p>Memory-R1 is built around <strong>two specialized, RL-fine-tuned agents</strong>:</p>
<ul>
<li><strong>Memory Manager</strong>: Decides which memory operations (<strong>ADD</strong>, <strong>UPDATE</strong>, <strong>DELETE</strong>, <strong>NOOP</strong>) to perform after each dialogue turn, updating the external memory bank dynamically.</li>
<li><strong>Answer Agent</strong>: For each user question, retrieves up to 60 candidate memories, <strong>distills</strong> them to the most relevant subset, then reasons over this filtered context to generate an answer.</li>
</ul>
<p>Both components are <strong>trained with reinforcement learning RL</strong>—using either Proximal Policy Optimization (PPO) or Group Relative Policy Optimization (GRPO)—with <strong>only question-answer correctness as the reward signal</strong>. This means that, instead of requiring manually labeled memory operations, the agents learn by trial and error, optimizing for <strong>final task performance</strong>.</p>

<figure><img alt="" loading="lazy" referrerpolicy="no-referrer" src="https://www.marktechpost.com/2025/08/28/memory-r1-how-reinforcement-learning-supercharges-llm-memory-agents/?amp"/><img alt="" loading="lazy" referrerpolicy="no-referrer" sizes="(max-width: 1024px) 100vw, 1024px" src="https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.59-PM-1-1024x508.png" srcset="https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.59-PM-1-1024x508.png 1024w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.59-PM-1-300x149.png 300w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.59-PM-1-768x381.png 768w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.59-PM-1-1536x761.png 1536w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.59-PM-1-847x420.png 847w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.59-PM-1-150x74.png 150w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.59-PM-1-696x345.png 696w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.59-PM-1-1068x529.png 1068w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.59-PM-1-324x160.png 324w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.46.59-PM-1.png 1626w"/><figcaption>https://arxiv.org/pdf/2508.19828</figcaption></figure>
<h3><strong>Memory Manager: Learning to Edit Knowledge</strong></h3>
<p>After each dialogue turn, an LLM extracts key facts. The <strong>Memory Manager</strong> then retrieves related entries from the memory bank, and chooses an operation:</p>
<ul>
<li><strong>ADD</strong>: Insert new information not already present.</li>
<li><strong>UPDATE</strong>: Merge new details into existing memories when they elaborate or refine previous facts.</li>
<li><strong>DELETE</strong>: Remove outdated or contradictory information.</li>
<li><strong>NOOP</strong>: Leave memory unchanged if nothing relevant is added.</li>
</ul>
<p><strong>Training</strong>: The Memory Manager is updated based on the quality of answers the Answer Agent generates from the newly edited memory bank. If a memory operation enables the Answer Agent to respond accurately, the Memory Manager receives a positive reward. This <strong>outcome-driven reward</strong> eliminates the need for costly manual annotation of memory operations.</p>
<p><strong>Example</strong>: When a user first mentions adopting a dog named Buddy, then later adds that they adopted another dog named Scout, a vanilla system might delete “Buddy” and add “Scout,” treating it as a contradiction. The RL-trained Memory Manager, however, <strong>updates</strong> the memory: “Andrew adopted two dogs, Buddy and Scout,” maintaining a coherent, evolving knowledge base.</p>
<p><strong>Ablation</strong>: RL fine-tuning improves memory management significantly—PPO and GRPO both outperform in-context, heuristic-based managers. The system learns to <strong>consolidate</strong> rather than <strong>fragment</strong> knowledge.</p>
<h3><strong>Answer Agent: Selective Reasoning</strong></h3>
<p>For each question, the system <strong>retrieves up to 60 candidate memories</strong> with RAG. But instead of feeding all these to the LLM, the <strong>Answer Agent</strong> first <strong>distills</strong> the set—keeping only the most relevant entries. Only then does it generate an answer.</p>
<p><strong>Training</strong>: The Answer Agent is also trained with RL, using the <strong>exact match</strong> between its answer and the gold answer as the reward. This encourages it to focus on <strong>filtering out noise</strong> and <strong>reasoning over high-quality context</strong>.</p>
<p><strong>Example</strong>: Asked “Does John live close to a beach or the mountains?”, a vanilla LLM might output “mountains,” influenced by irrelevant memories. Memory-R1’s Answer Agent, however, <strong>surfaces only beach-related entries</strong> before answering, leading to a correct “beach” response.</p>
<p><strong>Ablation</strong>: RL fine-tuning improves answer quality over static retrieval. <strong>Memory distillation</strong> (filtering out irrelevant memories) further boosts performance. The gains are even <strong>larger with a stronger memory manager</strong>, showing compounding improvements.</p>
<h3><strong>Training Data Efficiency</strong></h3>
<p>Memory-R1 is <strong>data-efficient</strong>: it achieves strong results with only <strong>152 question-answer pairs</strong> for training. This is possible because the agent learns from <strong>outcomes</strong>, not from thousands of hand-labeled memory operations. Supervision is kept to a minimum, and the system scales to large, real-world dialogue histories.</p>
<p>The <strong>LOCOMO benchmark</strong>, used for evaluation, consists of multi-turn dialogues (about 600 turns per dialogue, 26,000 tokens on average) and associated QA pairs spanning single-hop, multi-hop, open-domain, and temporal reasoning—ideal for testing <strong>long-horizon memory management</strong>.</p>
<h3><strong>Experimental Results</strong></h3>
<p>Memory-R1 was tested on <strong>LLaMA-3.1-8B-Instruct</strong> and <strong>Qwen-2.5-7B-Instruct</strong> backbones, against competitive baselines (LOCOMO, Zep, A-Mem, LangMem, Mem0). The key metrics are:</p>
<ul>
<li><strong>F1</strong>: Measures overlap between predicted and correct answers.</li>
<li><strong>BLEU-1</strong>: Captures lexical similarity at the unigram level.</li>
<li><strong>LLM-as-a-Judge</strong>: Uses a separate LLM to evaluate factual accuracy, relevance, and completeness—a <a href="https://www.marktechpost.com/2025/08/08/what-is-a-proxy-server-a-technical-deep-dive-with-trends-and-top-proxy-servers-2025-edition/?amp" rel="noopener noreferrer" target="_blank">proxy</a> for human judgment.</li>
</ul>
<p><strong>Results</strong>: Memory-R1-GRPO achieves the <strong>best overall performance</strong>, improving over Mem0 (the previous best baseline) by 48% in F1, 69% in BLEU-1, and 37% in LLM-as-a-Judge on LLaMA-3.1-8B. Similar gains are seen on Qwen-2.5-7B. The improvements are <strong>broad-based</strong>, spanning all question types, and <strong>generalize across model architectures</strong>.</p>

<figure><img alt="" loading="lazy" referrerpolicy="no-referrer" src="https://www.marktechpost.com/2025/08/28/memory-r1-how-reinforcement-learning-supercharges-llm-memory-agents/?amp"/><img alt="" loading="lazy" referrerpolicy="no-referrer" sizes="(max-width: 1024px) 100vw, 1024px" src="https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.47.17-PM-1-1024x449.png" srcset="https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.47.17-PM-1-1024x449.png 1024w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.47.17-PM-1-300x132.png 300w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.47.17-PM-1-768x337.png 768w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.47.17-PM-1-1536x674.png 1536w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.47.17-PM-1-958x420.png 958w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.47.17-PM-1-150x66.png 150w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.47.17-PM-1-696x305.png 696w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.47.17-PM-1-1068x468.png 1068w, https://www.marktechpost.com/wp-content/uploads/2025/08/Screenshot-2025-08-28-at-8.47.17-PM-1.png 1678w"/><figcaption>https://arxiv.org/pdf/2508.19828</figcaption></figure>
<h3><strong>Why This Matters</strong></h3>
<p>Memory-R1 shows that <strong>memory management and utilization can be learned</strong>—LLM agents don’t need to rely on brittle heuristics. By grounding decisions in outcome-driven RL, the system:</p>
<ul>
<li><strong>Automatically consolidates knowledge</strong> as conversations evolve, rather than fragmenting or overwriting it.</li>
<li><strong>Filters out noise</strong> when answering, improving factual accuracy and reasoning quality.</li>
<li><strong>Learns efficiently</strong> with little supervision, and <strong>scales</strong> to real-world, long-horizon tasks.</li>
<li><strong>Generalizes across models</strong>, making it a promising foundation for the next generation of agentic, memory-aware AI systems.</li>
</ul>
<h3><strong>Conclusion</strong></h3>
<p><strong>Memory-R1</strong> unshackles LLM agents from their stateless constraints, giving them the ability to learn—through reinforcement—how to manage and use long-term memories effectively. By framing memory operations and filtering as <strong>RL problems</strong>, it achieves <strong>state-of-the-art performance</strong> with <strong>minimal supervision</strong> and <strong>strong generalization</strong>. This marks a major step toward AI systems that not only converse fluently, but remember, learn, and reason like humans—offering richer, more persistent, and more useful experiences for users everywhere.</p>
<hr/>
<h3><strong>FAQs</strong></h3>
<p><strong>FAQ 1: What makes Memory-R1 better than typical LLM memory systems?</strong></p>
<p>Memory-R1 uses reinforcement learning to actively control memory—deciding which information to add, update, delete, or keep—enabling smarter consolidation and less fragmentation than static, heuristic-based approaches.</p>
<p><strong>FAQ 2: How does Memory-R1 improve answer quality from long dialogue histories?</strong></p>
<p>The Answer Agent applies a “memory distillation” policy: it filters up to 60 retrieved memories to surface only those most relevant for each question, reducing noise and improving factual accuracy compared to simply passing all context to the model.</p>
<p><strong>FAQ 3:</strong> <strong>Is Memory-R1 data-efficient for training?</strong></p>
<p>Yes, Memory-R1 achieves state-of-the-art gains using only 152 QA training pairs, as its outcome-based RL rewards eliminate the need for costly manual annotation of each memory operation.</p>
<hr/>
<p>Check out the <strong><a href="https://arxiv.org/abs/2508.19828" rel="noreferrer noopener" target="_blank">Paper here.</a></strong> Feel free to check out our <strong><a href="https://github.com/Marktechpost/AI-Tutorial-Codes-Included" rel="noreferrer noopener" target="_blank">GitHub Page for Tutorials, Codes and Notebooks</a></strong>. Also, feel free to follow us on <strong><a href="https://x.com/intent/follow?screen_name=marktechpost" rel="noreferrer noopener" target="_blank">Twitter</a></strong> and don’t forget to join our <strong><a href="https://www.reddit.com/r/machinelearningnews/" rel="noreferrer noopener" target="_blank">100k+ ML SubReddit</a></strong> and Subscribe to <strong><a href="https://www.aidevsignals.com/" rel="noreferrer noopener" target="_blank">our Newsletter</a></strong>.</p>
<p></p><p></p><p>Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.</p><p></p>

content_text:
https://arxiv.org/abs/2508.19828

Large language models (LLMs) now stand at the center of countless AI breakthroughs—chatbots, coding assistants, question answering, creative writing, and much more. But despite their prowess, they remain

stateless

: each query arrives with no memory of what came before. Their fixed

context windows

mean they can’t accumulate persistent knowledge across long conversations or multi-session tasks, and they struggle to reason over complex histories. Recent solutions, like retrieval-augmented generation (RAG), append past information to prompts, but this often leads to noisy, unfiltered context—flooding the model with too much irrelevant detail or missing crucial facts.

A team of researchers from University of Munich, Technical University of Munich, University of Cambridge and University of Hong Kong introduced

Memory-R1

, a framework that teaches LLM agents to decide what to remember and how to use it. Its LLM agent learns to

actively manage and utilize external memory

—deciding what to add, update, delete, or ignore, and filtering out noise when answering questions. The breakthrough? It trains these behaviors with

reinforcement learning (RL)

, using only outcome-based rewards, so it needs minimal supervision and

generalizes robustly across models and tasks

.

But Why LLMs Struggle with Memory

?

Imagine a multi-session conversation: in the first session, a user says, “I adopted a dog named Buddy.” Later, they add, “I adopted another dog named Scout.” Should the system

replace

the first statement with the second,

merge

them, or

ignore

the update? Vanilla memory pipelines often fail—they might erase “Buddy” and add “Scout,” misinterpreting the new information as a contradiction rather than a consolidation. Over time, such systems lose coherence, fragmenting user knowledge rather than evolving it.

RAG systems

retrieve information but don’t filter it: irrelevant entries pollute reasoning, and the model gets distracted by noise.

Humans

, by contrast, retrieve widely but then

selectively filter

what matters. Most AI memory systems are

static

, relying on handcrafted heuristics for what to remember, rather than learning from feedback.

https://arxiv.org/pdf/2508.19828

The Memory-R1 Framework

Memory-R1 is built around

two specialized, RL-fine-tuned agents

:

Memory Manager

: Decides which memory operations (

ADD

,

UPDATE

,

DELETE

,

NOOP

) to perform after each dialogue turn, updating the external memory bank dynamically.

Answer Agent

: For each user question, retrieves up to 60 candidate memories,

distills

them to the most relevant subset, then reasons over this filtered context to generate an answer.

Both components are

trained with reinforcement learning RL

—using either Proximal Policy Optimization (PPO) or Group Relative Policy Optimization (GRPO)—with

only question-answer correctness as the reward signal

. This means that, instead of requiring manually labeled memory operations, the agents learn by trial and error, optimizing for

final task performance

.

https://arxiv.org/pdf/2508.19828

Memory Manager: Learning to Edit Knowledge

After each dialogue turn, an LLM extracts key facts. The

Memory Manager

then retrieves related entries from the memory bank, and chooses an operation:

ADD

: Insert new information not already present.

UPDATE

: Merge new details into existing memories when they elaborate or refine previous facts.

DELETE

: Remove outdated or contradictory information.

NOOP

: Leave memory unchanged if nothing relevant is added.

Training

: The Memory Manager is updated based on the quality of answers the Answer Agent generates from the newly edited memory bank. If a memory operation enables the Answer Agent to respond accurately, the Memory Manager receives a positive reward. This

outcome-driven reward

eliminates the need for costly manual annotation of memory operations.

Example

: When a user first mentions adopting a dog named Buddy, then later adds that they adopted another dog named Scout, a vanilla system might delete “Buddy” and add “Scout,” treating it as a contradiction. The RL-trained Memory Manager, however,

updates

the memory: “Andrew adopted two dogs, Buddy and Scout,” maintaining a coherent, evolving knowledge base.

Ablation

: RL fine-tuning improves memory management significantly—PPO and GRPO both outperform in-context, heuristic-based managers. The system learns to

consolidate

rather than

fragment

knowledge.

Answer Agent: Selective Reasoning

For each question, the system

retrieves up to 60 candidate memories

with RAG. But instead of feeding all these to the LLM, the

Answer Agent

first

distills

the set—keeping only the most relevant entries. Only then does it generate an answer.

Training

: The Answer Agent is also trained with RL, using the

exact match

between its answer and the gold answer as the reward. This encourages it to focus on

filtering out noise

and

reasoning over high-quality context

.

Example

: Asked “Does John live close to a beach or the mountains?”, a vanilla LLM might output “mountains,” influenced by irrelevant memories. Memory-R1’s Answer Agent, however,

surfaces only beach-related entries

before answering, leading to a correct “beach” response.

Ablation

: RL fine-tuning improves answer quality over static retrieval.

Memory distillation

(filtering out irrelevant memories) further boosts performance. The gains are even

larger with a stronger memory manager

, showing compounding improvements.

Training Data Efficiency

Memory-R1 is

data-efficient

: it achieves strong results with only

152 question-answer pairs

for training. This is possible because the agent learns from

outcomes

, not from thousands of hand-labeled memory operations. Supervision is kept to a minimum, and the system scales to large, real-world dialogue histories.

The

LOCOMO benchmark

, used for evaluation, consists of multi-turn dialogues (about 600 turns per dialogue, 26,000 tokens on average) and associated QA pairs spanning single-hop, multi-hop, open-domain, and temporal reasoning—ideal for testing

long-horizon memory management

.

Experimental Results

Memory-R1 was tested on

LLaMA-3.1-8B-Instruct

and

Qwen-2.5-7B-Instruct

backbones, against competitive baselines (LOCOMO, Zep, A-Mem, LangMem, Mem0). The key metrics are:

F1

: Measures overlap between predicted and correct answers.

BLEU-1

: Captures lexical similarity at the unigram level.

LLM-as-a-Judge

: Uses a separate LLM to evaluate factual accuracy, relevance, and completeness—a

proxy

for human judgment.

Results

: Memory-R1-GRPO achieves the

best overall performance

, improving over Mem0 (the previous best baseline) by 48% in F1, 69% in BLEU-1, and 37% in LLM-as-a-Judge on LLaMA-3.1-8B. Similar gains are seen on Qwen-2.5-7B. The improvements are

broad-based

, spanning all question types, and

generalize across model architectures

.

https://arxiv.org/pdf/2508.19828

Why This Matters

Memory-R1 shows that

memory management and utilization can be learned

—LLM agents don’t need to rely on brittle heuristics. By grounding decisions in outcome-driven RL, the system:

Automatically consolidates knowledge

as conversations evolve, rather than fragmenting or overwriting it.

Filters out noise

when answering, improving factual accuracy and reasoning quality.

Learns efficiently

with little supervision, and

scales

to real-world, long-horizon tasks.

Generalizes across models

, making it a promising foundation for the next generation of agentic, memory-aware AI systems.

Conclusion

Memory-R1

unshackles LLM agents from their stateless constraints, giving them the ability to learn—through reinforcement—how to manage and use long-term memories effectively. By framing memory operations and filtering as

RL problems

, it achieves

state-of-the-art performance

with

minimal supervision

and

strong generalization

. This marks a major step toward AI systems that not only converse fluently, but remember, learn, and reason like humans—offering richer, more persistent, and more useful experiences for users everywhere.

FAQs

FAQ 1: What makes Memory-R1 better than typical LLM memory systems?

Memory-R1 uses reinforcement learning to actively control memory—deciding which information to add, update, delete, or keep—enabling smarter consolidation and less fragmentation than static, heuristic-based approaches.

FAQ 2: How does Memory-R1 improve answer quality from long dialogue histories?

The Answer Agent applies a “memory distillation” policy: it filters up to 60 retrieved memories to surface only those most relevant for each question, reducing noise and improving factual accuracy compared to simply passing all context to the model.

FAQ 3:

Is Memory-R1 data-efficient for training?

Yes, Memory-R1 achieves state-of-the-art gains using only 152 QA training pairs, as its outcome-based RL rewards eliminate the need for costly manual annotation of each memory operation.

Check out the

Paper here.

Feel free to check out our

GitHub Page for Tutorials, Codes and Notebooks

. Also, feel free to follow us on

Twitter

and don’t forget to join our

100k+ ML SubReddit

and Subscribe to

our Newsletter

.

Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.

thumb_url:
https://www.marktechpost.com/2025/08/28/memory-r1-how-reinforcement-learning-supercharges-llm-memory-agents/?amp


title: The end of dashboards? GenAI and agentic workflows transform business intelligence | CIO
added_at: 2025-09-01 03:05:38
url:
https://www.cio.com/article/4046967/the-end-of-dashboards-genai-and-agentic-workflows-transform-business-intelligence.html

content_html:
<p>I recently attended a series of marketing-centric webinars hosted by industry-leading enterprise data cloud vendors, all proclaiming that the business intelligence (BI) dashboard is unofficially dead and that Generative AI-based cloud data platform interfaces would provide a renaissance, if not a clear path to redemption for the future of business intelligence. Further, some recent architectural efforts I’ve been associated with also suggested that many large enterprises in multiple verticals struggle with data operations and the need for dynamic, outcome-driven interaction with data beyond investments in data lake houses, medallion or lambda architectures, and semantic models. All of these are foundational to making BI useful.</p>
<p>After some reflection, I was forced to consider the very real possibility that the BI landscape had not really diversified significantly in at least a decade. To be fair, industry leaders like Microsoft and Salesforce (Power BI and Tableau respectively) according to Gartner, have a significant <a href="https://www.gartner.com/en/documents/5519595" rel="nofollow" target="_blank">installed base</a> in the BI client arena and provide tooling that is sophisticated and evolved enough to engage business decision makers with compelling data and data visualizations yet, as I noted in <a href="https://www.cio.com/article/3957217/data-trust-and-the-evolution-of-enterprise-analytics-in-the-age-of-ai.html" rel="noopener noreferrer" target="_blank">Data trust and the evolution of enterprise analytics in the age of AI</a>, 58% of business decisionmakers rely on gut feel or experience rather than data and information.</p>
<p>While much of the issue is data trust, a larger portion is also based on the need for democratized inquiry, interaction, discovery and most of all, time-to-execution. Creating a semantic layer that is chained to static dashboards doesn’t really provide a significant advantage for anyone who needs to operate at the speed of business. The new basis of competition and market differentiator is not just time-to-insight, it is also time-to-execution. According to <a href="https://www.bvp.com/atlas/roadmap-ai-systems-of-action" rel="nofollow" target="_blank">Besemer Venture Partners</a>, unlocking data with a path to execution as opposed to aggregating and storing data means moving from “systems of record to systems of action.” So, what is the path forward to enabling both insight and action?</p>

content_text:
I recently attended a series of marketing-centric webinars hosted by industry-leading enterprise data cloud vendors, all proclaiming that the business intelligence (BI) dashboard is unofficially dead and that Generative AI-based cloud data platform interfaces would provide a renaissance, if not a clear path to redemption for the future of business intelligence. Further, some recent architectural efforts I’ve been associated with also suggested that many large enterprises in multiple verticals struggle with data operations and the need for dynamic, outcome-driven interaction with data beyond investments in data lake houses, medallion or lambda architectures, and semantic models. All of these are foundational to making BI useful.

After some reflection, I was forced to consider the very real possibility that the BI landscape had not really diversified significantly in at least a decade. To be fair, industry leaders like Microsoft and Salesforce (Power BI and Tableau respectively) according to Gartner, have a significant

installed base

in the BI client arena and provide tooling that is sophisticated and evolved enough to engage business decision makers with compelling data and data visualizations yet, as I noted in

Data trust and the evolution of enterprise analytics in the age of AI

, 58% of business decisionmakers rely on gut feel or experience rather than data and information.

While much of the issue is data trust, a larger portion is also based on the need for democratized inquiry, interaction, discovery and most of all, time-to-execution. Creating a semantic layer that is chained to static dashboards doesn’t really provide a significant advantage for anyone who needs to operate at the speed of business. The new basis of competition and market differentiator is not just time-to-insight, it is also time-to-execution. According to

Besemer Venture Partners

, unlocking data with a path to execution as opposed to aggregating and storing data means moving from “systems of record to systems of action.” So, what is the path forward to enabling both insight and action?

thumb_url:
https://www.cio.com/wp-content/uploads/2025/08/4046967-0-02094400-1756381523-Overlay-dashboard-or-trader-with-stress-headache-or-burnout-from-trading-data-bad-investment-or-stocks.-Anxiety-debt-or-frustrated-man-with-digital-ui-or-ux-on-financial-loss-on-screen-at-night.jpg?quality=50&strip=all&w=1024


added_at: 2025-09-02 11:27:05
thumb_url: https://img.youtube.com/vi/WkolFcU8_gE/maxresdefault.jpg
url:
https://www.tomshardware.com/tech-industry/artificial-intelligence/ai-converts-ascii-game-to-real-time-ai-rendered-graphics-thunder-lizard-ascii-visuals-transformed-but-latency-and-consistency-need-improvement

title:
AI converts ASCII game to real-time AI-rendered graphics – Thunder Lizard ASCII visuals transformed, but latency and consistency need improvement | Tom's Hardware

content_html:
<p>Software engineer Jeff Schomay has been experimenting with techniques to transform game visuals using artificial intelligence-powered rendering. In a recent <a href="https://blog.jeffschomay.com/rendering-a-game-in-real-time-with-ai" referrerpolicy="no-referrer-when-downgrade" rel="noopener noreferrer" target="_blank">blog post,</a> he shared screens from his Thunder Lizard “old-school ASCII RPG style game” and you can see various AI models reconstructing the title’s basic character-based visuals into far plusher full-motion graphics. However, there are quite a few compromises needed to achieve a ‘playable’ 10fps and 1ms latency AI visual enhanced game.</p><p></p><a href="https://www.tomshardware.com/tech-industry/artificial-intelligence/ai-converts-ascii-game-to-real-time-ai-rendered-graphics-thunder-lizard-ascii-visuals-transformed-but-latency-and-consistency-need-improvement" rel="noopener noreferrer" target="_blank"></a><h2>The current game</h2><p>You can <a href="https://enegames.itch.io/thunder-lizard-eat-or-be-eaten" referrerpolicy="no-referrer-when-downgrade" rel="noopener noreferrer" target="_blank">play Thunder Lizard</a>, using the current <a href="https://www.tomshardware.com/tech-industry/artificial-intelligence/researchers-jailbreak-ai-chatbots-with-ascii-art-artprompt-bypasses-safety-measures-to-unlock-malicious-queries" rel="noopener noreferrer" target="_blank">ASCII</a> rogue-like engine visuals, right now, via that link. In brief, you use the cursor keys to navigate around the prehistoric landscape while attempting to catch and east smaller dinosaurs, while avoiding larger ones. You grow as you eat smaller prey, but you have to flourish enough to become the most powerful dinosaur before the volcano destroys the entire landscape.</p><a href="https://www.tomshardware.com/tech-industry/artificial-intelligence/ai-converts-ascii-game-to-real-time-ai-rendered-graphics-thunder-lizard-ascii-visuals-transformed-but-latency-and-consistency-need-improvement" rel="noopener noreferrer" target="_blank"></a><p>Wouldn’t this game be more fun with snazzy pixel art visuals? Schomay must have thought so, too. And you can see various samples of AI-rendered alternatives to the ASCII throughout his blog post.</p><p></p><a href="https://www.tomshardware.com/tech-industry/artificial-intelligence/ai-converts-ascii-game-to-real-time-ai-rendered-graphics-thunder-lizard-ascii-visuals-transformed-but-latency-and-consistency-need-improvement" rel="noopener noreferrer" target="_blank"></a><h2>Cake and eat it – not possible</h2><p>As Schomay put it in his blog post, there is a “need for speed” in gaming, that isn’t yet met by many AI renderers. At least concerning what is available to indie devs.</p><p></p><p>We get to see some fantastic looking prehistoric landscape renderers, which would really add something to Thunder Lizard. However, the software engineer couldn’t coax better than “a four-second latency, it was unplayable” from the most handsome AI game graphics renderer.</p><a href="https://www.tomshardware.com/tech-industry/artificial-intelligence/ai-converts-ascii-game-to-real-time-ai-rendered-graphics-thunder-lizard-ascii-visuals-transformed-but-latency-and-consistency-need-improvement" rel="noopener noreferrer" target="_blank"></a><h2>The compromise</h2><p>You can read much more about Schomay’s efforts with various AI models in the linked blog, but for actual playability the choice was ultimately <a href="http://fal.ai/" referrerpolicy="no-referrer-when-downgrade" rel="noopener noreferrer" target="_blank">Fal.ai</a>. This model beat the other contenders for its “very fast generation times, source image adherence, and decent look and feel,” according to the software engineer.</p>


AI rendered ASCII game in real time - YouTube
<img alt="AI rendered ASCII game in real time - YouTube" loading="lazy" referrerpolicy="no-referrer" src="https://img.youtube.com/vi/WkolFcU8_gE/maxresdefault.jpg"/>



<a href="https://youtu.be/WkolFcU8_gE" referrerpolicy="no-referrer-when-downgrade" rel="noopener noreferrer" target="_blank">Watch On </a>
<p>Though what you see is an (AI) engine capable of just 10fps and 1ms latency, it’s the best option, according to Schomay’s efforts, in this case.</p><section><p>Get Tom's Hardware's best news and in-depth reviews, straight to your inbox.</p></section><p>Pixel peepers might feel a little let down by the chosen AI visuals, which eventually won out. However, Schomay ends his blog on an upbeat note, stating that “it is amazing to see AI image generation happen in real-time.” He is also excited about the future, but hopes some strides can be made in frame to frame consistency.</p><p>We've previously reported on the use of AI in developing and enhancing games and graphics. Nvidia shared a lot of RTX technology reliant <a href="https://www.tomshardware.com/pc-components/gpus/nvidia-rtx-ai-pcs-and-generative-ai-for-games-how-the-blackwell-gpus-and-rtx-50-series-aim-to-change-the-way-we-work-and-play" rel="noopener noreferrer" target="_blank">AI-enhanced development tool</a> information earlier this year.</p><p><em>Follow </em><a href="https://news.google.com/publications/CAAqLAgKIiZDQklTRmdnTWFoSUtFSFJ2YlhOb1lYSmtkMkZ5WlM1amIyMG9BQVAB" referrerpolicy="no-referrer-when-downgrade" rel="noopener noreferrer" target="_blank"><em>Tom's Hardware on Google News</em></a><em>, or </em><a href="https://google.com/preferences/source?q=" referrerpolicy="no-referrer-when-downgrade" rel="noopener noreferrer" target="_blank"><em>add us as a preferred source</em></a><em>, to get our up-to-date news, analysis, and reviews in your feeds. Make sure to click the Follow button!</em></p>

content_text:
Software engineer Jeff Schomay has been experimenting with techniques to transform game visuals using artificial intelligence-powered rendering. In a recent

blog post,

he shared screens from his Thunder Lizard “old-school ASCII RPG style game” and you can see various AI models reconstructing the title’s basic character-based visuals into far plusher full-motion graphics. However, there are quite a few compromises needed to achieve a ‘playable’ 10fps and 1ms latency AI visual enhanced game.

The current game

You can

play Thunder Lizard

, using the current

ASCII

rogue-like engine visuals, right now, via that link. In brief, you use the cursor keys to navigate around the prehistoric landscape while attempting to catch and east smaller dinosaurs, while avoiding larger ones. You grow as you eat smaller prey, but you have to flourish enough to become the most powerful dinosaur before the volcano destroys the entire landscape.

Wouldn’t this game be more fun with snazzy pixel art visuals? Schomay must have thought so, too. And you can see various samples of AI-rendered alternatives to the ASCII throughout his blog post.

Cake and eat it – not possible

As Schomay put it in his blog post, there is a “need for speed” in gaming, that isn’t yet met by many AI renderers. At least concerning what is available to indie devs.

We get to see some fantastic looking prehistoric landscape renderers, which would really add something to Thunder Lizard. However, the software engineer couldn’t coax better than “a four-second latency, it was unplayable” from the most handsome AI game graphics renderer.

The compromise

You can read much more about Schomay’s efforts with various AI models in the linked blog, but for actual playability the choice was ultimately

Fal.ai

. This model beat the other contenders for its “very fast generation times, source image adherence, and decent look and feel,” according to the software engineer.

AI rendered ASCII game in real time - YouTube

Watch On

Though what you see is an (AI) engine capable of just 10fps and 1ms latency, it’s the best option, according to Schomay’s efforts, in this case.

Get Tom's Hardware's best news and in-depth reviews, straight to your inbox.

Pixel peepers might feel a little let down by the chosen AI visuals, which eventually won out. However, Schomay ends his blog on an upbeat note, stating that “it is amazing to see AI image generation happen in real-time.” He is also excited about the future, but hopes some strides can be made in frame to frame consistency.

We've previously reported on the use of AI in developing and enhancing games and graphics. Nvidia shared a lot of RTX technology reliant

AI-enhanced development tool

information earlier this year.

Follow

Tom's Hardware on Google News

, or

add us as a preferred source

, to get our up-to-date news, analysis, and reviews in your feeds. Make sure to click the Follow button!


url: https://www.scworld.com/news/state-sponsored-attacks-now-make-up-53-of-vulnerability-exploits
title: State-sponsored attacks now make up 53% of vulnerability exploits | SC Media
added_at: 2025-09-02 17:10:08
thumb_url: https://files.cyberriskalliance.com/wp-content/uploads/2025/08/AdobeStock_212155328.jpg
content_html:
<p>More than half of attacks exploiting vulnerabilities were conducted by state-backed threat groups in the first half of 2025, <a href="https://www.recordedfuture.com/research/h1-2025-malware-and-vulnerability-trends" rel="noreferrer noopener" target="_blank">according to a report</a> by Recorded Future’s Insikt Group published Thursday.</p><p>The H1 2025 Malware and Vulnerability Trends report outlined evolving exploitation patterns to help organizations better understand their attack surface and prioritize defenses.</p><p>The paper also revealed that edge devices and <a href="https://www.scworld.com/brief/more-than-100-microsoft-vulnerabilities-fixed" rel="noreferrer noopener" target="_blank">Microsoft products</a> were the most common targets for vulnerability exploitation, each making up 17% of attacks observed by Insikt Group.</p><p>Overall, 23,667 CVEs were disclosed in H1 2025, a 16% increase from H1 2024. A total of 161 CVEs were identified by Recorded Future as being actively exploited.</p><p>Most of the exploited vulnerabilities required <a href="https://www.scworld.com/news/microsoft-365-copilot-zero-click-vulnerability-enabled-data-exfiltration" rel="noreferrer noopener" target="_blank">no authentication </a>to exploit (69%) and 30% enabled RCE. Additionally, 42% had <a href="https://www.scworld.com/news/jira-tickets-become-attack-vectors-in-poc-living-off-ai-attack" rel="noreferrer noopener" target="_blank">a public proof-of-concept (PoC)</a> exploit, with exploitation activity typically picking up after the release of a PoC exploit, according to Insikt Group.</p><p>State-sponsored threat groups were behind 53% of attributed vulnerability exploit attacks, followed by financially motivated non-ransomware actors (27%) and ransomware and extortion groups (20%). Most state-linked attacks originated from China, with the group <a href="https://www.scworld.com/brief/ivanti-epmm-flaws-leveraged-in-global-chinese-cyberespionage-attacks" rel="noreferrer noopener" target="_blank">UNC5221</a> notably targeting <a href="https://www.scworld.com/brief/attacks-leveraging-ivanti-epmm-flaws-in-clouds-underway" rel="noreferrer noopener" target="_blank">Ivanti flaws</a> such as <a href="https://www.scworld.com/brief/attacks-leveraging-ivanti-epmm-flaws-in-clouds-underway" rel="noreferrer noopener" target="_blank">CVE-2025-4428</a>, <a href="https://www.scworld.com/news/mandiant-warns-of-attacks-on-newly-disclosed-ivanti-remote-takeover-threat" rel="noreferrer noopener" target="_blank">CVE-2025-22457</a> and <a href="https://www.scworld.com/brief/active-exploitation-of-ivanti-connect-secure-zero-day-ongoing" rel="noreferrer noopener" target="_blank">CVE-2025-0282</a>.</p><h2>Use of backdoors, RATs increased in H1 2025 among other malware</h2><p>Backdoors were the most common malware type deployed in attacks at 22.9%, and remote access trojans (RATs) such as AsyncRAT, XWorm and Remcos overtook infostealers in popularity in H1 2025. <a href="https://www.scworld.com/brief/crossc2-enables-multi-platform-cobalt-strike-attacks-on-linux-macos" rel="noreferrer noopener" target="_blank">Cobalt Strike</a>, a legitimate but often misused offensive security tool (OST), was the most common post-exploitation tool observed, with OST use appearing in 15.6% of attacks.</p><p>Amidst the <a href="https://www.scworld.com/brief/global-crackdown-dismantles-lumma-infostealer" rel="noreferrer noopener" target="_blank">disruption of LummaC2</a> by law enforcement, <a href="https://www.scworld.com/analysis/industrial-control-system-password-cracker-may-be-bad-actually" rel="noreferrer noopener" target="_blank">the Sality malware family</a> overtook LummaC2 as the most popular malware observed in command-and-control (C2) detection references. The report noted the resurgence of older malware families like Sality and Tofsee in 2025.</p><p>The most common MITRE ATT&amp;CK-defined tactic observed was command and control, with more than 194,000 observations, followed by data encrypted for impact at nearly 1,100 observations and data from local system at 263, emphasizing the impact of ransomware in the attack landscape.</p><p>Ransomware threat actors were noted to increasingly use <a href="https://www.scworld.com/news/clickfix-attacks-surge-as-exploits-see-drop-in-popularity" rel="noreferrer noopener" target="_blank">the ClickFix social-engineering technique</a> for initial access in H1 2025, along with bring-your-own-installer (BYOI)-based endpoint detection and response (EDR) evasion methods and just-in-time (JIT) hooking and memory injection to avoid detection, according to the report.</p><p>Cl0p claimed the most ransomware victims (374) in H1 2025, largely through its <a href="https://www.scworld.com/news/cleo-vulnerability-attacks-claimed-by-clop-ransomware-gang" rel="noreferrer noopener" target="_blank">exploitation of the Cleo managed file transfer system</a>, while LockBit was still the most common ransomware strain observed at 2,023 attacks, followed by Chaos (1288) and Akira (944).</p><p>The mobile malware landscape saw the addition of 11 new strains, with abuse of accessibility services and NFC signal relay attacks being common among mobile attackers. The report noted the use of a new virtualization-based overlay technique by the Android trojan GodFather in June 2024, as well as the emergence of a malware-as-a-service (MaaS) platform called SuperCard X that facilitates NFC relay for fraudulent contactless transactions.</p><p>Insikt Group also highlighted the evolution of the popular Magecart e-skimmer, which has remained popular after a record-high attack volume in 2024, and has now extended from targeting Adobe Commerce’s Magento platform to also targeting WooCommerce plugins for WordPress.</p><p>Ultimately, the report stresses the importance of prioritizing patching and mitigation of heavily targeted vulnerabilities such as those in Microsoft products and devices at the network edge, as well as those with public exploits available. Organizations should keep tabs on real-time threat intelligence to identify and block emerging threats, as well as expand detection capabilities to include behavioral monitoring and C2 traffic analysis for greater coverage.</p>

content_text:
More than half of attacks exploiting vulnerabilities were conducted by state-backed threat groups in the first half of 2025,

according to a report

by Recorded Future’s Insikt Group published Thursday.

The H1 2025 Malware and Vulnerability Trends report outlined evolving exploitation patterns to help organizations better understand their attack surface and prioritize defenses.

The paper also revealed that edge devices and

Microsoft products

were the most common targets for vulnerability exploitation, each making up 17% of attacks observed by Insikt Group.

Overall, 23,667 CVEs were disclosed in H1 2025, a 16% increase from H1 2024. A total of 161 CVEs were identified by Recorded Future as being actively exploited.

Most of the exploited vulnerabilities required

no authentication

to exploit (69%) and 30% enabled RCE. Additionally, 42% had

a public proof-of-concept (PoC)

exploit, with exploitation activity typically picking up after the release of a PoC exploit, according to Insikt Group.

State-sponsored threat groups were behind 53% of attributed vulnerability exploit attacks, followed by financially motivated non-ransomware actors (27%) and ransomware and extortion groups (20%). Most state-linked attacks originated from China, with the group

UNC5221

notably targeting

Ivanti flaws

such as

CVE-2025-4428

,

CVE-2025-22457

and

CVE-2025-0282

.

Use of backdoors, RATs increased in H1 2025 among other malware

Backdoors were the most common malware type deployed in attacks at 22.9%, and remote access trojans (RATs) such as AsyncRAT, XWorm and Remcos overtook infostealers in popularity in H1 2025.

Cobalt Strike

, a legitimate but often misused offensive security tool (OST), was the most common post-exploitation tool observed, with OST use appearing in 15.6% of attacks.

Amidst the

disruption of LummaC2

by law enforcement,

the Sality malware family

overtook LummaC2 as the most popular malware observed in command-and-control (C2) detection references. The report noted the resurgence of older malware families like Sality and Tofsee in 2025.

The most common MITRE ATT&CK-defined tactic observed was command and control, with more than 194,000 observations, followed by data encrypted for impact at nearly 1,100 observations and data from local system at 263, emphasizing the impact of ransomware in the attack landscape.

Ransomware threat actors were noted to increasingly use

the ClickFix social-engineering technique

for initial access in H1 2025, along with bring-your-own-installer (BYOI)-based endpoint detection and response (EDR) evasion methods and just-in-time (JIT) hooking and memory injection to avoid detection, according to the report.

Cl0p claimed the most ransomware victims (374) in H1 2025, largely through its

exploitation of the Cleo managed file transfer system

, while LockBit was still the most common ransomware strain observed at 2,023 attacks, followed by Chaos (1288) and Akira (944).

The mobile malware landscape saw the addition of 11 new strains, with abuse of accessibility services and NFC signal relay attacks being common among mobile attackers. The report noted the use of a new virtualization-based overlay technique by the Android trojan GodFather in June 2024, as well as the emergence of a malware-as-a-service (MaaS) platform called SuperCard X that facilitates NFC relay for fraudulent contactless transactions.

Insikt Group also highlighted the evolution of the popular Magecart e-skimmer, which has remained popular after a record-high attack volume in 2024, and has now extended from targeting Adobe Commerce’s Magento platform to also targeting WooCommerce plugins for WordPress.

Ultimately, the report stresses the importance of prioritizing patching and mitigation of heavily targeted vulnerabilities such as those in Microsoft products and devices at the network edge, as well as those with public exploits available. Organizations should keep tabs on real-time threat intelligence to identify and block emerging threats, as well as expand detection capabilities to include behavioral monitoring and C2 traffic analysis for greater coverage.


added_at: 2025-09-06 03:12:18
url:
https://developers.googleblog.com/en/introducing-langextract-a-gemini-powered-information-extraction-library/

title:
Introducing LangExtract: A Gemini powered information extraction library
            
            
            - Google Developers Blog

content_html:
<p>In today's data-rich world, valuable insights are often locked away in unstructured text, such as detailed clinical notes, lengthy legal documents, customer feedback threads and evolving news reports. Manually sifting through this information or building bespoke code to process the data is time-consuming and error-prone, and using modern large language models (LLMs) naively may <a href="https://cloud.google.com/discover/what-are-ai-hallucinations?hl=en" rel="noopener noreferrer" target="_blank">introduce errors</a>. <b>What if you could programmatically extract the exact information you need, while ensuring the outputs are structured and reliably tied back to its source?</b></p><p>Today, we're excited to introduce <b>LangExtract</b>, a new <a href="https://github.com/google/langextract" rel="noopener noreferrer" target="_blank">open-source Python library</a> designed to empower developers to do just that. LangExtract provides a lightweight interface to various LLMs such as our Gemini models for processing large volumes of unstructured text into structured information based on your custom instructions, ensuring both flexibility and traceability.</p><p>Whether you're working with medical reports, financial summaries, or any other text-heavy domain, LangExtract offers a flexible and powerful way to unlock the data within.</p><p>LangExtract offers a unique combination of capabilities that make it useful for information extraction:</p><p></p><ul><li><b>Precise source grounding:</b> Every extracted entity is mapped back to its exact character offsets in the source text. As demonstrated in the animations below, this feature provides traceability by visually highlighting each extraction in the original text, making it much easier to evaluate and verify the extracted information.</li></ul><p></p><p></p><ul><li><b>Optimized long-context information extraction:</b> Information retrieval from large documents can be complex. For instance, while LLMs show strong performance on many benchmarks, <a href="https://cloud.google.com/blog/products/ai-machine-learning/the-needle-in-the-haystack-test-and-how-gemini-pro-solves-it" rel="noopener noreferrer" target="_blank">needle-in-a-haystack tests</a> across million-token contexts show that recall can decrease in multi-fact retrieval scenarios. LangExtract is built to handle this using a chunking strategy, parallel processing and multiple extraction passes over smaller, focused contexts.</li></ul><p></p><ul><li><b>Interactive visualization:</b> Go from raw text to an interactive, self-contained HTML visualization in minutes. LangExtract makes it easy to review extracted entities in context, with support for exploring thousands of annotations.</li></ul><p></p><ul><li><b>Flexible support for LLM backends:</b> Work with your preferred models, whether they are cloud-based LLMs (like Google's Gemini family) or open-source on-device models.</li></ul><p></p><ul><li><b>Flexible across domains:</b> Define information extraction tasks for any domain with just a few well-chosen examples, without the need to fine-tune an LLM. LangExtract “learns” your desired output and can apply it to large, new text inputs. See how it works with this <a href="https://github.com/google/langextract/blob/main/docs/examples/medication_examples.md" rel="noopener noreferrer" target="_blank">medication extraction example</a>.</li></ul><p></p><ul><li><b>Utilizing LLM world knowledge:</b> In addition to extracting grounded entities, LangExtract can leverage a model's world knowledge to supplement extracted information. This information can be explicit (i.e., derived from the source text) or inferred (i.e., derived from the model's inherent world knowledge). The accuracy and relevance of such supplementary knowledge, particularly when inferred, are heavily influenced by the chosen LLM's capabilities and the precision of the prompt examples guiding the extraction.</li></ul><h2><br/>Quick start: From Shakespeare to structured objects</h2><p>Here's how to extract character details from a line of Shakespeare.</p><p>First, install the library:</p><p>For more detailed setup instructions, including virtual environments and API key configuration, please see the <a href="https://github.com/google/langextract#installation" rel="noopener noreferrer" target="_blank">project README</a>.</p>
 
<p>Next, define your extraction task. Provide a clear prompt and a high-quality "few-shot" example to guide the model.</p>
 
<pre><code>import textwrap
import langextract as lx

# 1. Define a concise prompt
prompt = textwrap.dedent("""\
Extract characters, emotions, and relationships in order of appearance.
Use exact text for extractions. Do not paraphrase or overlap entities.
Provide meaningful attributes for each entity to add context.""")

# 2. Provide a high-quality example to guide the model
examples = [
    lx.data.ExampleData(
        text=(
            "ROMEO. But soft! What light through yonder window breaks? It is"
            " the east, and Juliet is the sun."
        ),
        extractions=[
            lx.data.Extraction(
                extraction_class="character",
                extraction_text="ROMEO",
                attributes={"emotional_state": "wonder"},
            ),
            lx.data.Extraction(
                extraction_class="emotion",
                extraction_text="But soft!",
                attributes={"feeling": "gentle awe"},
            ),
            lx.data.Extraction(
                extraction_class="relationship",
                extraction_text="Juliet is the sun",
                attributes={"type": "metaphor"},
            ),
        ],
    )
]

# 3. Run the extraction on your input text
input_text = (
    "Lady Juliet gazed longingly at the stars, her heart aching for Romeo"
)
result = lx.extract(
    text_or_documents=input_text,
    prompt_description=prompt,
    examples=examples,
    model_id="gemini-2.5-pro",
)</code></pre>
<p>
        Python
    </p>
 
<p>The result object contains the extracted entities, which can be saved to a JSONL file. From there, you can generate an interactive HTML file to view the annotations. This visualization is great for demos or evaluating the extraction quality, saving valuable time. It works seamlessly in environments like <a href="https://colab.research.google.com/" rel="noopener noreferrer" target="_blank">Google Colab</a> or can be saved as a standalone HTML file, viewable from your browser.</p>
 
<pre><code># Save the results to a JSONL file
lx.io.save_annotated_documents([result], output_name="extraction_results.jsonl")

# Generate the interactive visualization from the file
html_content = lx.visualize("extraction_results.jsonl")
with open("visualization.html", "w") as f:
    f.write(html_content)</code></pre>
<p>
        Python
    </p>


<h2>Flexibility for specialized domains</h2><p>The same principles above apply to specialized domains like medicine, finance, engineering or law. The ideas behind LangExtract were first applied to medical information extraction and can be effective at processing clinical text. For example, it can identify medications, dosages, and other medication attributes, and then map the relationships between them. This capability was a core part of the research that led to this library, which you can read about in our <a href="https://arxiv.org/abs/2312.02296" rel="noopener noreferrer" target="_blank">paper on accelerating medical information extraction</a>.</p><p>The animation below shows LangExtract processing clinical text to extract medication-related entities and groups them to the source medication.</p>


<h2>Demo on structured radiology reporting</h2><p>To showcase LangExtract's power in a specialized field, we developed an interactive demonstration for structured radiology reporting called <a href="https://google-radextract.hf.space/" rel="noopener noreferrer" target="_blank">RadExtract</a> on Hugging Face. This demo shows how LangExtract can process a free-text radiology report and automatically convert its key findings into a structured format, also highlighting important findings. This approach is <a href="https://link.springer.com/article/10.1007/s13244-017-0588-8" rel="noopener noreferrer" target="_blank">important in radiology</a>, where structuring reports enhances clarity, ensures completeness, and improves data interoperability for research and clinical care.</p>


<p></p><hr/><p><b>Disclaimer:</b> The medication extraction example and structured reporting demo above are for illustrative purposes of LangExtract's baseline capability only. It does not represent a finished or approved product, is not intended to diagnose or suggest treatment of any disease or condition, and should not be used for medical advice.</p><h2><br/>Get started with LangExtract: Resources and next steps</h2><p>We're excited to see the innovative ways developers will use LangExtract to unlock insights from text. Dive into the documentation, explore the examples on our <a href="https://github.com/google/langextract" rel="noopener noreferrer" target="_blank">GitHub repository</a>, and start transforming your unstructured data today.</p>

content_text:
In today's data-rich world, valuable insights are often locked away in unstructured text, such as detailed clinical notes, lengthy legal documents, customer feedback threads and evolving news reports. Manually sifting through this information or building bespoke code to process the data is time-consuming and error-prone, and using modern large language models (LLMs) naively may

introduce errors

.

What if you could programmatically extract the exact information you need, while ensuring the outputs are structured and reliably tied back to its source?

Today, we're excited to introduce

LangExtract

, a new

open-source Python library

designed to empower developers to do just that. LangExtract provides a lightweight interface to various LLMs such as our Gemini models for processing large volumes of unstructured text into structured information based on your custom instructions, ensuring both flexibility and traceability.

Whether you're working with medical reports, financial summaries, or any other text-heavy domain, LangExtract offers a flexible and powerful way to unlock the data within.

LangExtract offers a unique combination of capabilities that make it useful for information extraction:

Precise source grounding:

Every extracted entity is mapped back to its exact character offsets in the source text. As demonstrated in the animations below, this feature provides traceability by visually highlighting each extraction in the original text, making it much easier to evaluate and verify the extracted information.

Optimized long-context information extraction:

Information retrieval from large documents can be complex. For instance, while LLMs show strong performance on many benchmarks,

needle-in-a-haystack tests

across million-token contexts show that recall can decrease in multi-fact retrieval scenarios. LangExtract is built to handle this using a chunking strategy, parallel processing and multiple extraction passes over smaller, focused contexts.

Interactive visualization:

Go from raw text to an interactive, self-contained HTML visualization in minutes. LangExtract makes it easy to review extracted entities in context, with support for exploring thousands of annotations.

Flexible support for LLM backends:

Work with your preferred models, whether they are cloud-based LLMs (like Google's Gemini family) or open-source on-device models.

Flexible across domains:

Define information extraction tasks for any domain with just a few well-chosen examples, without the need to fine-tune an LLM. LangExtract “learns” your desired output and can apply it to large, new text inputs. See how it works with this

medication extraction example

.

Utilizing LLM world knowledge:

In addition to extracting grounded entities, LangExtract can leverage a model's world knowledge to supplement extracted information. This information can be explicit (i.e., derived from the source text) or inferred (i.e., derived from the model's inherent world knowledge). The accuracy and relevance of such supplementary knowledge, particularly when inferred, are heavily influenced by the chosen LLM's capabilities and the precision of the prompt examples guiding the extraction.

Quick start: From Shakespeare to structured objects

Here's how to extract character details from a line of Shakespeare.

First, install the library:

For more detailed setup instructions, including virtual environments and API key configuration, please see the

project README

.

Next, define your extraction task. Provide a clear prompt and a high-quality "few-shot" example to guide the model.

import textwrap

import langextract as lx

# 1. Define a concise prompt

prompt = textwrap.dedent("""\

Extract characters, emotions, and relationships in order of appearance.

Use exact text for extractions. Do not paraphrase or overlap entities.

Provide meaningful attributes for each entity to add context.""")

# 2. Provide a high-quality example to guide the model

examples = [

lx.data.ExampleData(

text=(

"ROMEO. But soft! What light through yonder window breaks? It is"

" the east, and Juliet is the sun."

),

extractions=[

lx.data.Extraction(

extraction_class="character",

extraction_text="ROMEO",

attributes={"emotional_state": "wonder"},

),

lx.data.Extraction(

extraction_class="emotion",

extraction_text="But soft!",

attributes={"feeling": "gentle awe"},

),

lx.data.Extraction(

extraction_class="relationship",

extraction_text="Juliet is the sun",

attributes={"type": "metaphor"},

),

],

)

]

# 3. Run the extraction on your input text

input_text = (

"Lady Juliet gazed longingly at the stars, her heart aching for Romeo"

)

result = lx.extract(

text_or_documents=input_text,

prompt_description=prompt,

examples=examples,

model_id="gemini-2.5-pro",

)

Python

The result object contains the extracted entities, which can be saved to a JSONL file. From there, you can generate an interactive HTML file to view the annotations. This visualization is great for demos or evaluating the extraction quality, saving valuable time. It works seamlessly in environments like

Google Colab

or can be saved as a standalone HTML file, viewable from your browser.

# Save the results to a JSONL file

lx.io.save_annotated_documents([result], output_name="extraction_results.jsonl")

# Generate the interactive visualization from the file

html_content = lx.visualize("extraction_results.jsonl")

with open("visualization.html", "w") as f:

f.write(html_content)

Python

Flexibility for specialized domains

The same principles above apply to specialized domains like medicine, finance, engineering or law. The ideas behind LangExtract were first applied to medical information extraction and can be effective at processing clinical text. For example, it can identify medications, dosages, and other medication attributes, and then map the relationships between them. This capability was a core part of the research that led to this library, which you can read about in our

paper on accelerating medical information extraction

.

The animation below shows LangExtract processing clinical text to extract medication-related entities and groups them to the source medication.

Demo on structured radiology reporting

To showcase LangExtract's power in a specialized field, we developed an interactive demonstration for structured radiology reporting called

RadExtract

on Hugging Face. This demo shows how LangExtract can process a free-text radiology report and automatically convert its key findings into a structured format, also highlighting important findings. This approach is

important in radiology

, where structuring reports enhances clarity, ensures completeness, and improves data interoperability for research and clinical care.

Disclaimer:

The medication extraction example and structured reporting demo above are for illustrative purposes of LangExtract's baseline capability only. It does not represent a finished or approved product, is not intended to diagnose or suggest treatment of any disease or condition, and should not be used for medical advice.

Get started with LangExtract: Resources and next steps

We're excited to see the innovative ways developers will use LangExtract to unlock insights from text. Dive into the documentation, explore the examples on our

GitHub repository

, and start transforming your unstructured data today.

thumb_url:
https://storage.googleapis.com/gweb-developer-goog-blog-assets/images/Data-1-meta.2e16d0ba.fill-1200x600.png


url: https://www.infoq.com/articles/architecting-rag-pipeline/
title: Effective Practices for Architecting a RAG Pipeline - InfoQ
added_at: 2025-09-08 10:27:56
content_html:
<h3>Key Takeaways</h3>
<ul>
<li>A hybrid of vector and term-based search is the most effective strategy for RAG pipelines that answer user questions about documentation. Both vector databases and Lucene-based search engines support this, but tuning the underlying algorithm is critical for optimal results.</li>
<li>When the domain is complex enough, and the questions are sufficiently sophisticated and nuanced, similarity (which is what you get out of a document search) is not the same thing as relevance (which is what the LLM needs to answer the question).</li>
<li>Chunking refers to the process of breaking down content into smaller units when indexing documents for a database. The database search could miss the similarity if the chunks are too large or too small. The basis for chunking should differ depending on the knowledge domain and the type of content and media used to deliver it.</li>
<li>Not all types of content should be indexed the same way. Strategies vary for indexing diagrams, graphs, sample code, tabular data, and various kinds of prose.</li>
<li>Despite getting bigger with each new LLM release, the context window remains a crucial consideration. Including only the most relevant search results in a RAG prompt ensures the highest quality response.</li>
</ul>
<p></p>

<p>Having just completed a development effort to build a RAG pipeline for software architects, I decided to collect some notes from my experience and share them here. I hope that you can benefit from what I have learned.</p>
<h2>Project Solution Architecture</h2>
<p>Let’s briefly describe the project from which most of these lessons were learned. This is for a cloud based B2B solution that provides software architecture support services to technology companies that maintain their own software. You give it access to your source code and documentation.</p>
<p>After that, you can ask about the current architecture or how to enhance it. You can ask for just a high-level pitch or a detailed plan. You can also ask about reducing accidental complexity in general. Figure 1 below is a component diagram for the entire RAG pipeline for this project but this article will focus on the lessons learned from the documentation chunking, indexing, and searching parts.</p>
<p><img alt="" loading="lazy" referrerpolicy="no-referrer" rel="share" src="https://imgopt.infoq.com/fit-in/3000x4000/filters:quality(85)/filters:no_upscale()/articles/architecting-rag-pipeline/en/resources/140figure-1-1755593891004.jpg"/></p>
<p><strong>Figure 1. component diagram for the software architecture support project where these lessons were learned</strong></p>
<p>[Click here to <img alt="expand image above to full-size" loading="lazy" referrerpolicy="no-referrer" src="https://imgopt.infoq.com/fit-in/3000x4000/filters:quality(85)/filters:no_upscale()/articles/architecting-rag-pipeline/en/resources/140figure-1-1755593891004.jpg"/>]</p>
<h2>RAG revisited</h2>
<p>To properly set the context for the rest of the article, let's begin with the basics of what RAG is. I will be brief here and invite you to read this May 2025 InfoQ article <a href="https://www.infoq.com/articles/practical-design-patterns-modern-ai-systems/" rel="noopener noreferrer" target="_blank">Beyond the Gang of Four: Practical Design Patterns for Modern AI Systems</a> for more details on the subject.</p>
<p>Figure 2 illustrates the basic flow of data for a document focused RAG pipeline. Before the user can ask any questions, documents are chunked into fragments then saved to a database. The user's question is collected and undergoes some preprocessing before being combined with system instructions and database search results in a template to construct the final prompt. The RAG pipeline then submits the prompt to the LLM. The response is then collected and undergoes some post-processing before it is surfaced to the end user as an answer to their question.</p>
<p><img alt="" loading="lazy" referrerpolicy="no-referrer" rel="share" src="https://imgopt.infoq.com/fit-in/3000x4000/filters:quality(85)/filters:no_upscale()/articles/architecting-rag-pipeline/en/resources/108figure-2-1755594535405.jpg"/></p>
<p><strong>Figure 2. typical data flow in a document focused RAG pipeline</strong></p>
<p>Let's review the context window, which is the maximum amount of text in tokens that the LLM can consider or remember in any single session. If your prompt exceeds the context window, the LLM won't remember the system instructions or the question, which is typically located at the beginning of the prompt. Successive versions of LLMs have ever-increasing context windows, but this remains a consideration to be aware of.</p>
<p>Why bother with a RAG pipeline at all? Why not have the user ask their question directly to the LLM without any additional prompt engineering or supporting database search? If we consider a use case from the above mentioned project which is a user asking how to enhance an existing software system, several limitations of LLMs become obvious. Perhaps the vendor trained the LLM prior to the release of the documentation, or the documentation was not part of the training data. Even if the documentation was used to train the LLM, there may be conflicting information that requires additional context. Internal details about proprietary software are almost never released for public consumption. When the LLM knows either too little or too much, the likelihood of AI hallucination increases, and we must take steps to mitigate it through a curated process.</p>
<p>LLMs and RAG are currently a very hot topic, and there is a lot of activity in the tooling landscape that supports this. Here is the high-level anatomy of this type of tooling. It is usually an open-source framework that supports a plugin architecture. There is typically an associated hub with shareable assets, which could include models, training data, and prompts. Not only does the tool vendor contribute to this hub, but also members of the community. There is also an enterprise upsell that you may or may not feel compelled to purchase.</p>
<h2>Lessons Learned</h2>
<p>Enough with the basics. Let’s get on with what we ran into and what we did about it.</p>
<p>There are many vendor tools in this space, too numerous to evaluate much less cover here. The oldest and most well-known tool is <a href="https://github.com/langchain-ai" rel="noopener noreferrer" target="_blank">langchain</a> which is a general purpose framework that focuses on standardized component interfaces, orchestration, and observability. I decided to pass on langchain for the following reasons. I have been burned by their backwards breaking changes in a previous project. Every LLM provides their own API and they are all very similar so it is easy to swap one out for another. Connecting the output from one step to the input for another via a template is pretty easy to do with standard Python.</p>
<p>Aside from the LLM itself, there are two LLM focused tools that we did end up using. Remember question pre-processing and answer post-processing in the RAG pipeline description above? That processing is mainly in the form of safety vetting, and the <a href="https://hub.guardrailsai.com/" rel="noopener noreferrer" target="_blank">guardrails</a> product is an excellent tool for that. There are lots of checkers to choose from on their hub. Some require an API key but many do not. In the rest of this article, you will see references to sentence transformers, POS (Parts of Speech) tagging, summarization, and sentiment analysis. <a href="https://huggingface.co/" rel="noopener noreferrer" target="_blank">Huggingface</a> is an excellent resource for that functionality.</p>
<h2>Document Ingestion</h2>
<p>Before you can search a database for document excerpts to consider in answering the question, you must first save those excerpts into the database. Breaking up the overall documentation into individual document fragments and storing them in the database as separate rows is known as chunking.</p>
<p>Chunking granularity indicates the size of each separately saved document fragment. A chunk could range from a single sentence to multiple pages. If the chunks are too large, then the similarity of a small portion of a chunk with the question may get washed out of the results. Here is another issue with large chunks. What eventually gets fed into the LLM will contain a significant amount of irrelevant content. We will see later how that compromises answer quality. If we make the chunks too small, we may overlook relevant data in the search and, therefore, not include that data in the prompt.</p>
<p>I wish that there was a one-size-fits-all granularity for chunking, but that is not the case. While the character length of each document fragment should fall within a predetermined range, that should not be the guiding criteria. Chunking granularity depends on several factors, including knowledge domain, content type, and media type. For example, descriptive prose on a detailed domain that serves experts might require a larger chunking granularity than persuasive prose on a simplified domain intended for a more general audience. In many types of prose, each paragraph wants to capture a separate idea. That might be a reasonable basis for chunking. What about screenplays? Each paragraph is just the lines for the next character that speaks. If you chunk at that level, then you might miss relevant results for questions about the interaction between two characters. Other types of content where per-paragraph chunking may not serve as well include sample code, poetry, and tabular data.</p>
<p>The choice of delivery medium for the original documentation can also have a profound effect on the chunking strategy. Here is an example - files written in the Portable Document Format (PDF) group content by page. What if a paragraph gets split across two pages? It may not look any different than two pages where a paragraph ends at the bottom of the first page and a new paragraph begins at the top of the second page. How do you programmatically determine when two separate blocks of prose that span pages belong to the same paragraph?</p>
<p>For our project, we decided to ignore this problem and just live with the consequences. Some PDFs are internally organized such that each page is just an image of text, graphs, tables, you name it. There are strategies for extracting the content but it isn’t easy. Examples include asking the LLM and using OCR. Fortunately for us, those types of PDFs are more likely intended for public facing marketing collateral than internal facing software architecture. For our project, we coded up a custom PDF importer using <a href="https://github.com/py-pdf/pypdf" rel="noopener noreferrer" target="_blank">pypdf</a> because we needed more control over the chunking. If you want to go with something more off-the-shelf, then I hear that <a href="https://github.com/pymupdf/RAG" rel="noopener noreferrer" target="_blank">pymupdf4llm</a> + <a href="https://github.com/tesseract-ocr/tesseract" rel="noopener noreferrer" target="_blank">tesseract</a> is a good choice for that.</p>
<p>What about long paragraphs? According to <a href="https://web.archive.org/web/20250510224728/https://www.grammarly.com/blog/sentences/how-long-is-a-paragraph/" rel="noopener noreferrer" target="_blank">Grammarly</a>, the average paragraph size is somewhere between 100 and 200 words. That is a reasonable chunking size, but <a href="https://irisreading.com/exceptionally-long-sentences-literature-why-they-work/" rel="noopener noreferrer" target="_blank">not all writers follow that rule</a>. For our project, we decided to ignore this problem since this issue does not occur much in the world of software architecture documentation.</p>
<p>Indexing web pages are another example of a media-specific chunking strategy. You need to filter out all the navigation and boilerplate content from each page. Examples include headers, footers, navigation menus, and breadcrumbs. You will likely want to follow links that take you to another area of the documentation to be indexed, but not outside of that area. Examples include other domains, about us, site map, and shopping cart. For our project, we coded up a custom web scraper because we needed more control over the chunking. This custom web scraper uses the <a href="https://www.crummy.com/software/BeautifulSoup/" rel="noopener noreferrer" target="_blank">Beautiful Soup</a> library.</p>
<p>Different knowledge domains have varying reliance on graphical content. The graphics are superfluous in some types of content. In other types of content, a narrative description of what is relevant in each image may already be there, although not in all types. How can you collect that for a document search? You can embed images into databases for vector search (more on this later), but what gets embedded may not be relevant to the types of questions asked. A cartoon of a boy and his dog will most likely be embedded accurately. A graph of political election results could be represented as red and blue circles of varying sizes, which may not be particularly useful. You may need to generate and save custom, domain-specific text summaries for specific content types, including diagrams, charts, sample code, and tabular data.</p>
<p>For the project that I was working on, we would periodically scrape the web sites containing the documentation. This was mostly architectural documentation but also user and administrator guides were scraped because they can include useful information and are usually in the same web site. We also imported PDF files containing educational collateral. Both HTML and PDF assets were usually hosted on an internal CMS. We would use heuristics to break up each page into blocks of prose, sample code, diagrams, and configuration. We would chunk at the paragraph level for the prose and ask the LLM to generate a summary for the other types of content.</p>
<h2>Document Search</h2>
<p>Now that you have indexed all of the document fragments into your database, what is the best way to search for them in the context of the original question? For the project that I was working on, I found that the hybrid search approach was the most effective. Hybrid search casts a wider net so you are more likely to collect enough relevant results. Hybrid search involves multiple searches, typically comprising one or two vector searches and a term search. The results are merged and reranked, usually with the Reciprocal Rank Fusion (RRF) algorithm. The query for the vector search is either the question itself or a normalized version of it. We haven’t done this yet but you should consider correcting for grammatical errors and summarizing if the question is too long or meanders.</p>
<p>The query for the term search could be the question, or you could extract the most relevant terms from the question. For many domains, that would be the proper nouns, and you can use POS tagging to identify those. That was our approach and we used the huggingface plugin for <a href="https://github.com/flairNLP/flair" rel="noopener noreferrer" target="_blank">flair</a> to accomplish POS tagging. This approach does not work for "What does it do?" style questions that lack any proper nouns. We ended up treating that type of question as invalid and would direct the user to a page on how to write good questions.</p>
<p>We have already covered the process of breaking down your documentation into text fragments and then saving each fragment to the database. The process of converting that text into a searchable vector is known as embedding, and there are different types. Use dense vectors for what is typically known as vector search. Use a sentence transformer to embed text into a dense vector.</p>
<p>The most popular transformers are derived from the Bidirectional Encoder Representations from Transformers (BERT) algorithm. Two popular sentence transformers to consider are <a href="https://huggingface.co/sentence-transformers/all-distilroberta-v1" rel="noopener noreferrer" target="_blank">all-distilroberta-v1</a> and <a href="https://huggingface.co/sentence-transformers/nli-mpnet-base-v2" rel="noopener noreferrer" target="_blank">nli-mpnet-base-v2</a>. Check out <a href="https://huggingface.co/spaces/mteb/leaderboard" rel="noopener noreferrer" target="_blank">this leaderboard</a> if you're more interested in the latest developments in embedding models. Embedding images require a vision transformer such as <a href="https://huggingface.co/docs/transformers/en/model_doc/clip" rel="noopener noreferrer" target="_blank">CLIP</a> or <a href="https://huggingface.co/docs/transformers/v4.28.1/model_doc/swin" rel="noopener noreferrer" target="_blank">Swin</a>. Vector databases use sparse vectors to accommodate term-based search. There is a BM25-based embedder for this, but I recommend a SParse Lexical AnD Expansion Model (SPLADE) <a href="https://huggingface.co/naver/splade-v3" rel="noopener noreferrer" target="_blank">embedder</a>, which is also based on BERT. You should use the same embedding algorithm for indexing document fragments as you use for embedding the question at time of search. Different record fields in the database can be populated with different embedding functions but always use the same embedding function for the same field in all records of the database.</p>
<p>I am writing this in 2025, so the most prominent databases to consider using are the vector-only databases, such as <a href="https://github.com/qdrant/qdrant" rel="noopener noreferrer" target="_blank">Qdrant</a>, <a href="https://www.pinecone.io/" rel="noopener noreferrer" target="_blank">Pinecone</a>, and <a href="https://github.com/milvus-io/milvus" rel="noopener noreferrer" target="_blank">Milvus</a>, as well as the Lucene-based databases, including <a href="https://github.com/elastic/elasticsearch" rel="noopener noreferrer" target="_blank">Elasticsearch</a>, <a href="https://solr.apache.org/" rel="noopener noreferrer" target="_blank">Solr</a>, and <a href="https://opensearch.org/" rel="noopener noreferrer" target="_blank">OpenSearch</a>. The Lucene-based databases provide term-based search using an inverted index data structure. The algorithm is BM25, which ranks results by a normalized term frequency (TF) and inverse document frequency (IDF). Lucene also supports vector search algorithms. You can use the community edition of Elasticsearch to do this, but you have to use the sentence transformer in your code.</p>
<p>The more premium licenses of Elasticsearch give you the ability to configure an ML pipeline that runs within the Elasticsearch process space and does the embedding for you automatically. All of these databases support hybrid search but we ended up querying the database separately and implementing the merge and reranking of the results in our code. We felt like we needed more control over that because it is so critical to the quality of the responses.</p>
<p>The actual algorithm for vector search is also known as kNN or k nearest neighbors. There is a <a href="https://www.elastic.co/blog/ann-vs-knn" rel="noopener noreferrer" target="_blank">faster version of kNN known as aNN</a> or approximate k nearest neighbors. The aNN approach is usually implemented with the Hierarchical Navigable Small Worlds (<a href="https://arxiv.org/abs/1603.09320" rel="noopener noreferrer" target="_blank">HNSW</a>) algorithm. Think of HNSW as hierarchical proximity graphs over probabilistic skip lists. An alternative algorithm for aNN is <a href="https://faiss.ai/" rel="noopener noreferrer" target="_blank">FAISS</a>, but most databases don't support it yet.</p>
<p>What determines just how near the neighbors are is what is known as the distance function. For dense vectors, the most popular algorithm for that is called cosine similarity. There are alternatives, such as Euclidean distance and Manhattan distance. Use a weighted dot product for sparse vectors. The choice of distance function rarely changes but is loosely based on choice of embedding function. Cosine similarity does not take the amplitude of the two vectors into account whereas Euclidean distance does. Manhattan distance is more about the number of discrete changes required to transform one vector into the other. We used cosine similarity.</p>
<h2>Document Retrieval and Reranking</h2>
<p>Given the nature of the project where the questions are not simple and the answers require deeper reasoning, we quickly learned something that may not be obvious to RAG pipeline newcomers. The results from a database search provides document fragments that are <em>similar</em> to the question. What the LLM needs to answer the question are <em>relevant</em> document fragments. <em>Relevance</em> is the likelihood that the document fragment will help answer the question. For most non-trivial use cases, <em>similarity</em> is not equivalent to <em>relevance</em>. Why not include all the search results in the prompt and let the LLM determine what is relevant, silently ignoring the rest? Even when a prompt fits within the context window, responses tend to degrade as token count approaches the limit. Those answers will sound more buzzwordy and less articulate, providing less value to your end user. This is known as <a href="https://research.trychroma.com/context-rot" rel="noopener noreferrer" target="_blank">context rot</a> and measurable degradation can occur even after the input token size exceeds 1% of the context window.</p>
<p>You must figure out a way to rerank the search results based on relevance and include only the most relevant results in the prompt. Perhaps you can find some domain-specific heuristics for estimating relevance. For our purposes, results that included enough proper nouns from the question (see POS tagging above) were considered to be relevant. If you are in a contentious and polarizing domain, you should also include sentiment analysis with proper nouns, although this was not necessary for our project. If we didn’t get enough relevant results this way, we would try the worst-case solution which is to loop through all the search results and then prompt the LLM to calculate the relevance of each result. The system instructions could appear as follows:</p>
<blockquote>
<p>Study the following question and data, then return the relevance or likelihood that the supplied data could be used to answer the question. This relevance would be expressed as a floating-point number between 0.0 and 1.0, where 0.0 indicates that the data isn't useful at all, 1.0 means that the data is useful with complete certainty, and 0.5 suggests that the data is just as likely to be useful as not - output as JSON in the specified schema. The response MUST be a valid JSON object and NOTHING else.</p>
</blockquote>
<p>That approach can be expensive and time-consuming, making it less suitable for your purposes if scalability and costs are considerations.</p>
<p><img alt="" loading="lazy" referrerpolicy="no-referrer" rel="share" src="https://imgopt.infoq.com/fit-in/3000x4000/filters:quality(85)/filters:no_upscale()/articles/architecting-rag-pipeline/en/resources/89figure-3-1755594535405.jpg"/></p>
<p><strong>Figure 3. Flowchart for combining POS tagging, LLM relevance, and summarization.</strong></p>
<p>[Click here to <img alt="expand image above to full-size" loading="lazy" referrerpolicy="no-referrer" src="https://imgopt.infoq.com/fit-in/3000x4000/filters:quality(85)/filters:no_upscale()/articles/architecting-rag-pipeline/en/resources/89figure-3-1755594535405.jpg"/>]</p>
<p>If limiting by relevance is not possible, consider using the summaries of the search results in the prompt instead. Summaries might sufficiently shorten the length of the prompt, but they do take some time to generate and could also result in lowering the relevance of the data, which could decrease the value of the answer. Depending on the size of the results, we ended up doing a mixture of both relevance filtering and result summarization (see Figure 3 above). We used the <a href="https://huggingface.co/facebook/bart-large-cnn" rel="noopener noreferrer" target="_blank">BART (large-sized model), fine-tuned on CNN Daily Mail</a> model released by Meta on Huggingface to generate this kind of summarization.</p>
<h2>Prompting with the Results</h2>
<p>You may have to adjust the system instructions based on the search results. Suppose you have a healthy number of relevant search results. In that case, your system instructions can say something like, "You are an expert tasked to answer the following question based exclusively on the provided data". but if you get none or very few relevant results, then maybe the system instructions are more like, "You are an expert tasked to answer the following question based on your already existing knowledge of the subject".</p>
<p>Prompting with no results doesn't sound very RAG-like, but I can assure you from firsthand experience that the LLM can get quite snarky when you instruct it to use the data and then fail to provide any data to use.</p>
<h2>Conclusions</h2>
<p>From document ingestion to searching, retrieval, reranking, and folding into a template-based prompt for the LLM, I hope you find these notes on RAG pipelines useful in your learning journey with RAG pipeline construction. After reading this article, you should realize that there are plenty of knobs to turn and techniques to try if you are finding the LLM’s answers to be lackluster in your own pipeline.</p>

content_text:
Key Takeaways

A hybrid of vector and term-based search is the most effective strategy for RAG pipelines that answer user questions about documentation. Both vector databases and Lucene-based search engines support this, but tuning the underlying algorithm is critical for optimal results.

When the domain is complex enough, and the questions are sufficiently sophisticated and nuanced, similarity (which is what you get out of a document search) is not the same thing as relevance (which is what the LLM needs to answer the question).

Chunking refers to the process of breaking down content into smaller units when indexing documents for a database. The database search could miss the similarity if the chunks are too large or too small. The basis for chunking should differ depending on the knowledge domain and the type of content and media used to deliver it.

Not all types of content should be indexed the same way. Strategies vary for indexing diagrams, graphs, sample code, tabular data, and various kinds of prose.

Despite getting bigger with each new LLM release, the context window remains a crucial consideration. Including only the most relevant search results in a RAG prompt ensures the highest quality response.

Having just completed a development effort to build a RAG pipeline for software architects, I decided to collect some notes from my experience and share them here. I hope that you can benefit from what I have learned.

Project Solution Architecture

Let’s briefly describe the project from which most of these lessons were learned. This is for a cloud based B2B solution that provides software architecture support services to technology companies that maintain their own software. You give it access to your source code and documentation.

After that, you can ask about the current architecture or how to enhance it. You can ask for just a high-level pitch or a detailed plan. You can also ask about reducing accidental complexity in general. Figure 1 below is a component diagram for the entire RAG pipeline for this project but this article will focus on the lessons learned from the documentation chunking, indexing, and searching parts.

Figure 1. component diagram for the software architecture support project where these lessons were learned

[Click here to

]

RAG revisited

To properly set the context for the rest of the article, let's begin with the basics of what RAG is. I will be brief here and invite you to read this May 2025 InfoQ article

Beyond the Gang of Four: Practical Design Patterns for Modern AI Systems

for more details on the subject.

Figure 2 illustrates the basic flow of data for a document focused RAG pipeline. Before the user can ask any questions, documents are chunked into fragments then saved to a database. The user's question is collected and undergoes some preprocessing before being combined with system instructions and database search results in a template to construct the final prompt. The RAG pipeline then submits the prompt to the LLM. The response is then collected and undergoes some post-processing before it is surfaced to the end user as an answer to their question.

Figure 2. typical data flow in a document focused RAG pipeline

Let's review the context window, which is the maximum amount of text in tokens that the LLM can consider or remember in any single session. If your prompt exceeds the context window, the LLM won't remember the system instructions or the question, which is typically located at the beginning of the prompt. Successive versions of LLMs have ever-increasing context windows, but this remains a consideration to be aware of.

Why bother with a RAG pipeline at all? Why not have the user ask their question directly to the LLM without any additional prompt engineering or supporting database search? If we consider a use case from the above mentioned project which is a user asking how to enhance an existing software system, several limitations of LLMs become obvious. Perhaps the vendor trained the LLM prior to the release of the documentation, or the documentation was not part of the training data. Even if the documentation was used to train the LLM, there may be conflicting information that requires additional context. Internal details about proprietary software are almost never released for public consumption. When the LLM knows either too little or too much, the likelihood of AI hallucination increases, and we must take steps to mitigate it through a curated process.

LLMs and RAG are currently a very hot topic, and there is a lot of activity in the tooling landscape that supports this. Here is the high-level anatomy of this type of tooling. It is usually an open-source framework that supports a plugin architecture. There is typically an associated hub with shareable assets, which could include models, training data, and prompts. Not only does the tool vendor contribute to this hub, but also members of the community. There is also an enterprise upsell that you may or may not feel compelled to purchase.

Lessons Learned

Enough with the basics. Let’s get on with what we ran into and what we did about it.

There are many vendor tools in this space, too numerous to evaluate much less cover here. The oldest and most well-known tool is

langchain

which is a general purpose framework that focuses on standardized component interfaces, orchestration, and observability. I decided to pass on langchain for the following reasons. I have been burned by their backwards breaking changes in a previous project. Every LLM provides their own API and they are all very similar so it is easy to swap one out for another. Connecting the output from one step to the input for another via a template is pretty easy to do with standard Python.

Aside from the LLM itself, there are two LLM focused tools that we did end up using. Remember question pre-processing and answer post-processing in the RAG pipeline description above? That processing is mainly in the form of safety vetting, and the

guardrails

product is an excellent tool for that. There are lots of checkers to choose from on their hub. Some require an API key but many do not. In the rest of this article, you will see references to sentence transformers, POS (Parts of Speech) tagging, summarization, and sentiment analysis.

Huggingface

is an excellent resource for that functionality.

Document Ingestion

Before you can search a database for document excerpts to consider in answering the question, you must first save those excerpts into the database. Breaking up the overall documentation into individual document fragments and storing them in the database as separate rows is known as chunking.

Chunking granularity indicates the size of each separately saved document fragment. A chunk could range from a single sentence to multiple pages. If the chunks are too large, then the similarity of a small portion of a chunk with the question may get washed out of the results. Here is another issue with large chunks. What eventually gets fed into the LLM will contain a significant amount of irrelevant content. We will see later how that compromises answer quality. If we make the chunks too small, we may overlook relevant data in the search and, therefore, not include that data in the prompt.

I wish that there was a one-size-fits-all granularity for chunking, but that is not the case. While the character length of each document fragment should fall within a predetermined range, that should not be the guiding criteria. Chunking granularity depends on several factors, including knowledge domain, content type, and media type. For example, descriptive prose on a detailed domain that serves experts might require a larger chunking granularity than persuasive prose on a simplified domain intended for a more general audience. In many types of prose, each paragraph wants to capture a separate idea. That might be a reasonable basis for chunking. What about screenplays? Each paragraph is just the lines for the next character that speaks. If you chunk at that level, then you might miss relevant results for questions about the interaction between two characters. Other types of content where per-paragraph chunking may not serve as well include sample code, poetry, and tabular data.

The choice of delivery medium for the original documentation can also have a profound effect on the chunking strategy. Here is an example - files written in the Portable Document Format (PDF) group content by page. What if a paragraph gets split across two pages? It may not look any different than two pages where a paragraph ends at the bottom of the first page and a new paragraph begins at the top of the second page. How do you programmatically determine when two separate blocks of prose that span pages belong to the same paragraph?

For our project, we decided to ignore this problem and just live with the consequences. Some PDFs are internally organized such that each page is just an image of text, graphs, tables, you name it. There are strategies for extracting the content but it isn’t easy. Examples include asking the LLM and using OCR. Fortunately for us, those types of PDFs are more likely intended for public facing marketing collateral than internal facing software architecture. For our project, we coded up a custom PDF importer using

pypdf

because we needed more control over the chunking. If you want to go with something more off-the-shelf, then I hear that

pymupdf4llm

+

tesseract

is a good choice for that.

What about long paragraphs? According to

Grammarly

, the average paragraph size is somewhere between 100 and 200 words. That is a reasonable chunking size, but

not all writers follow that rule

. For our project, we decided to ignore this problem since this issue does not occur much in the world of software architecture documentation.

Indexing web pages are another example of a media-specific chunking strategy. You need to filter out all the navigation and boilerplate content from each page. Examples include headers, footers, navigation menus, and breadcrumbs. You will likely want to follow links that take you to another area of the documentation to be indexed, but not outside of that area. Examples include other domains, about us, site map, and shopping cart. For our project, we coded up a custom web scraper because we needed more control over the chunking. This custom web scraper uses the

Beautiful Soup

library.

Different knowledge domains have varying reliance on graphical content. The graphics are superfluous in some types of content. In other types of content, a narrative description of what is relevant in each image may already be there, although not in all types. How can you collect that for a document search? You can embed images into databases for vector search (more on this later), but what gets embedded may not be relevant to the types of questions asked. A cartoon of a boy and his dog will most likely be embedded accurately. A graph of political election results could be represented as red and blue circles of varying sizes, which may not be particularly useful. You may need to generate and save custom, domain-specific text summaries for specific content types, including diagrams, charts, sample code, and tabular data.

For the project that I was working on, we would periodically scrape the web sites containing the documentation. This was mostly architectural documentation but also user and administrator guides were scraped because they can include useful information and are usually in the same web site. We also imported PDF files containing educational collateral. Both HTML and PDF assets were usually hosted on an internal CMS. We would use heuristics to break up each page into blocks of prose, sample code, diagrams, and configuration. We would chunk at the paragraph level for the prose and ask the LLM to generate a summary for the other types of content.

Document Search

Now that you have indexed all of the document fragments into your database, what is the best way to search for them in the context of the original question? For the project that I was working on, I found that the hybrid search approach was the most effective. Hybrid search casts a wider net so you are more likely to collect enough relevant results. Hybrid search involves multiple searches, typically comprising one or two vector searches and a term search. The results are merged and reranked, usually with the Reciprocal Rank Fusion (RRF) algorithm. The query for the vector search is either the question itself or a normalized version of it. We haven’t done this yet but you should consider correcting for grammatical errors and summarizing if the question is too long or meanders.

The query for the term search could be the question, or you could extract the most relevant terms from the question. For many domains, that would be the proper nouns, and you can use POS tagging to identify those. That was our approach and we used the huggingface plugin for

flair

to accomplish POS tagging. This approach does not work for "What does it do?" style questions that lack any proper nouns. We ended up treating that type of question as invalid and would direct the user to a page on how to write good questions.

We have already covered the process of breaking down your documentation into text fragments and then saving each fragment to the database. The process of converting that text into a searchable vector is known as embedding, and there are different types. Use dense vectors for what is typically known as vector search. Use a sentence transformer to embed text into a dense vector.

The most popular transformers are derived from the Bidirectional Encoder Representations from Transformers (BERT) algorithm. Two popular sentence transformers to consider are

all-distilroberta-v1

and

nli-mpnet-base-v2

. Check out

this leaderboard

if you're more interested in the latest developments in embedding models. Embedding images require a vision transformer such as

CLIP

or

Swin

. Vector databases use sparse vectors to accommodate term-based search. There is a BM25-based embedder for this, but I recommend a SParse Lexical AnD Expansion Model (SPLADE)

embedder

, which is also based on BERT. You should use the same embedding algorithm for indexing document fragments as you use for embedding the question at time of search. Different record fields in the database can be populated with different embedding functions but always use the same embedding function for the same field in all records of the database.

I am writing this in 2025, so the most prominent databases to consider using are the vector-only databases, such as

Qdrant

,

Pinecone

, and

Milvus

, as well as the Lucene-based databases, including

Elasticsearch

,

Solr

, and

OpenSearch

. The Lucene-based databases provide term-based search using an inverted index data structure. The algorithm is BM25, which ranks results by a normalized term frequency (TF) and inverse document frequency (IDF). Lucene also supports vector search algorithms. You can use the community edition of Elasticsearch to do this, but you have to use the sentence transformer in your code.

The more premium licenses of Elasticsearch give you the ability to configure an ML pipeline that runs within the Elasticsearch process space and does the embedding for you automatically. All of these databases support hybrid search but we ended up querying the database separately and implementing the merge and reranking of the results in our code. We felt like we needed more control over that because it is so critical to the quality of the responses.

The actual algorithm for vector search is also known as kNN or k nearest neighbors. There is a

faster version of kNN known as aNN

or approximate k nearest neighbors. The aNN approach is usually implemented with the Hierarchical Navigable Small Worlds (

HNSW

) algorithm. Think of HNSW as hierarchical proximity graphs over probabilistic skip lists. An alternative algorithm for aNN is

FAISS

, but most databases don't support it yet.

What determines just how near the neighbors are is what is known as the distance function. For dense vectors, the most popular algorithm for that is called cosine similarity. There are alternatives, such as Euclidean distance and Manhattan distance. Use a weighted dot product for sparse vectors. The choice of distance function rarely changes but is loosely based on choice of embedding function. Cosine similarity does not take the amplitude of the two vectors into account whereas Euclidean distance does. Manhattan distance is more about the number of discrete changes required to transform one vector into the other. We used cosine similarity.

Document Retrieval and Reranking

Given the nature of the project where the questions are not simple and the answers require deeper reasoning, we quickly learned something that may not be obvious to RAG pipeline newcomers. The results from a database search provides document fragments that are

similar

to the question. What the LLM needs to answer the question are

relevant

document fragments.

Relevance

is the likelihood that the document fragment will help answer the question. For most non-trivial use cases,

similarity

is not equivalent to

relevance

. Why not include all the search results in the prompt and let the LLM determine what is relevant, silently ignoring the rest? Even when a prompt fits within the context window, responses tend to degrade as token count approaches the limit. Those answers will sound more buzzwordy and less articulate, providing less value to your end user. This is known as

context rot

and measurable degradation can occur even after the input token size exceeds 1% of the context window.

You must figure out a way to rerank the search results based on relevance and include only the most relevant results in the prompt. Perhaps you can find some domain-specific heuristics for estimating relevance. For our purposes, results that included enough proper nouns from the question (see POS tagging above) were considered to be relevant. If you are in a contentious and polarizing domain, you should also include sentiment analysis with proper nouns, although this was not necessary for our project. If we didn’t get enough relevant results this way, we would try the worst-case solution which is to loop through all the search results and then prompt the LLM to calculate the relevance of each result. The system instructions could appear as follows:

Study the following question and data, then return the relevance or likelihood that the supplied data could be used to answer the question. This relevance would be expressed as a floating-point number between 0.0 and 1.0, where 0.0 indicates that the data isn't useful at all, 1.0 means that the data is useful with complete certainty, and 0.5 suggests that the data is just as likely to be useful as not - output as JSON in the specified schema. The response MUST be a valid JSON object and NOTHING else.

That approach can be expensive and time-consuming, making it less suitable for your purposes if scalability and costs are considerations.

Figure 3. Flowchart for combining POS tagging, LLM relevance, and summarization.

[Click here to

]

If limiting by relevance is not possible, consider using the summaries of the search results in the prompt instead. Summaries might sufficiently shorten the length of the prompt, but they do take some time to generate and could also result in lowering the relevance of the data, which could decrease the value of the answer. Depending on the size of the results, we ended up doing a mixture of both relevance filtering and result summarization (see Figure 3 above). We used the

BART (large-sized model), fine-tuned on CNN Daily Mail

model released by Meta on Huggingface to generate this kind of summarization.

Prompting with the Results

You may have to adjust the system instructions based on the search results. Suppose you have a healthy number of relevant search results. In that case, your system instructions can say something like, "You are an expert tasked to answer the following question based exclusively on the provided data". but if you get none or very few relevant results, then maybe the system instructions are more like, "You are an expert tasked to answer the following question based on your already existing knowledge of the subject".

Prompting with no results doesn't sound very RAG-like, but I can assure you from firsthand experience that the LLM can get quite snarky when you instruct it to use the data and then fail to provide any data to use.

Conclusions

From document ingestion to searching, retrieval, reranking, and folding into a template-based prompt for the LLM, I hope you find these notes on RAG pipelines useful in your learning journey with RAG pipeline construction. After reading this article, you should realize that there are plenty of knobs to turn and techniques to try if you are finding the LLM’s answers to be lackluster in your own pipeline.

thumb_url:
https://imgopt.infoq.com/fit-in/3000x4000/filters:quality(85)/filters:no_upscale()/articles/architecting-rag-pipeline/en/resources/140figure-1-1755593891004.jpg


title: Resilient Cyber w/ Rob T. Lee - Navigating AI's Impact on Cyber & the Workforce
added_at: 2025-09-09 09:39:04
url:
https://www.resilientcyber.io/p/resilient-cyber-w-rob-t-lee-navigating?r=3tujkj&utm_medium=ios&utm_campaign=audio-player

content_html:
<p>In this episode of <strong><a href="https://www.linkedin.com/company/resilient-cyber/" rel="" target="_blank">Resilient Cyber</a></strong>, I sat down with the <strong><a href="https://www.linkedin.com/company/sans-institute/" rel="" target="_blank">SANS Institute</a></strong>'s Chief of Research (COR) &amp; Chief AI Officer (CAIO), <strong><a href="https://www.linkedin.com/in/leerob/" rel="" target="_blank">Rob T. Lee</a></strong> to discuss AI's impact on cybersecurity and the workforce. <br/><br/>We discussed SANS Critical AI Security Guidelines, the opportunities and obstacles AI presents for cybersecurity, and how practitioners should navigate AI's impact on the workforce.</p><p></p><hr/><p></p><hr/><p><em><strong>Interested in sponsoring an issue of Resilient Cyber?</strong></em></p><p><em><strong>This includes reaching over 45,000 subscribers, ranging from Developers, Engineers, Architects, CISO’s/Security Leaders and Business Executives</strong></em></p><p><em><strong>Reach out below!</strong></em></p><p><a href="mailto:sponsorships@resilientcyber.io" rel="" target="_blank">-&gt; Contact Us! &lt;-</a></p><p></p><hr/><p>Listen on <strong><a href="https://open.spotify.com/episode/1sYxHTMwPYpQhZBByDyjTm?si=E_WZPAMGQfeNHA2MMEggKA" rel="" target="_blank">Spotify</a></strong>, <strong><a href="https://youtu.be/miAUdGD19n0?si=mQLXWEL6YcTTvqko" rel="" target="_blank">YouTube</a>,</strong> or <strong><a href="https://podcasts.apple.com/us/podcast/resilient-cyber-w-rob-t-lee-navigating-ais-impact-on/id1555928024?i=1000725313040" rel="" target="_blank">Apple Podcasts</a></strong>. </p><p></p><hr/><p>Rob and I discussed:</p><ul><li><p>SANS Critical AI Security Guidelines, including what categories of activities and security they cover for securing AI and how the resource can be used by practitioners</p></li><li><p>The six categories of controls in the guideline: Access Control, Data Protection, Deployment Strategies, Inference Security, Monitoring and GRC</p></li><li><p>Why not using AI is actually the biggest risk of AI, and how security can transition from being a laggard to an early adopter and innovator when it comes to AI</p></li><li><p>The double edged sword of AI, offering opportunities to innovate in problematic areas of cybersecurity, but also needing to be secured itself</p></li><li><p>AI’s current and potential impact on the cybersecurity workforce and how practitioners should be preparing, including what resources and training SANS offers when it comes to AI security</p></li></ul><p></p>

content_text:
In this episode of

Resilient Cyber

, I sat down with the

SANS Institute

's Chief of Research (COR) & Chief AI Officer (CAIO),

Rob T. Lee

to discuss AI's impact on cybersecurity and the workforce.

We discussed SANS Critical AI Security Guidelines, the opportunities and obstacles AI presents for cybersecurity, and how practitioners should navigate AI's impact on the workforce.

Interested in sponsoring an issue of Resilient Cyber?

This includes reaching over 45,000 subscribers, ranging from Developers, Engineers, Architects, CISO’s/Security Leaders and Business Executives

Reach out below!

-> Contact Us! <-

Listen on

Spotify

,

YouTube

,

or

Apple Podcasts

.

Rob and I discussed:

SANS Critical AI Security Guidelines, including what categories of activities and security they cover for securing AI and how the resource can be used by practitioners

The six categories of controls in the guideline: Access Control, Data Protection, Deployment Strategies, Inference Security, Monitoring and GRC

Why not using AI is actually the biggest risk of AI, and how security can transition from being a laggard to an early adopter and innovator when it comes to AI

The double edged sword of AI, offering opportunities to innovate in problematic areas of cybersecurity, but also needing to be secured itself

AI’s current and potential impact on the cybersecurity workforce and how practitioners should be preparing, including what resources and training SANS offers when it comes to AI security

thumb_url:
https://substackcdn.com/image/fetch/$s_!JsmW!,w_1200,h_600,c_fill,f_jpg,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-video.s3.amazonaws.com%2Fvideo_upload%2Fpost%2F172976076%2F3f23a6d3-9368-48c2-a9af-c4c45e2c9255%2Ftranscoded-1757191778.png


url: https://www.xda-developers.com/port-forwarding-alternative-tailscale/
title: Stop port forwarding your self-hosted services: Do this instead
added_at: 2025-09-09 11:10:56
content_html:
<p>Running a Plex server? Or, for that matter, any server, maybe a dashboard you need external access for, or even a personal website — port<a href="https://www.xda-developers.com/what-is-port-forwarding-why/" rel="noopener noreferrer" target="_blank"> forwarding</a> is one of the most common issues to come across. It sounds simple, and on paper it is.</p> <p>All you've got to do is get your router to open a port, point it at the right device or service, and let the internet pass through. But it rarely works as easily as that. Fact of the matter is, even when using high-end equipment, port forwarding can be a never-ending hassle. But there's a better way to handle this. Just install <a href="https://tailscale.com/" rel="noopener noreferrer nofollow" target="_blank">Tailscale</a> instead.</p> <h2>
                        Why Tailscale is better than port-forwarding
               </h2><h3>
            A secure mesh that just works
    </h3>
<p> Unlike port forwarding that pokes holes in your home network, <a href="https://www.xda-developers.com/tailscale-funnel-access-self-hosted-services/" rel="noopener noreferrer" target="_blank">Tailscale</a> creates a secure mesh internet connection between your devices. Effectively, this allows all your devices to behave as if they were on the same local network. Moreover, Tailscale works across different internet connections and without a complicated router setup. While port forwarding is a credible solution, the fact of the matter is that Tailscale is the better option for most users.</p> <p>Now, if you're wondering why you need this when <a href="https://www.xda-developers.com/cgnat-port-forwarding-workarounds/" rel="noopener noreferrer" target="_blank">port forwarding</a> works just the same, I've got your back. As I mentioned earlier, port forwarding basically opens specific doors or windows into your home network. These doorways allow internet traffic between, say, your <a href="https://www.xda-developers.com/free-and-open-source-media-servers-that-make-plex-pass-look-overpriced/" rel="noopener noreferrer" target="_blank">Plex server</a> and a remote client, but they can also be scanned, and ports left open inadvertently can be exploited by others wanting to use your network.</p> <p>It's a security risk waiting to happen unless you are diligent about setting up <a href="https://www.xda-developers.com/regret-not-deploying-custom-home-firewall-sooner/" rel="noopener noreferrer" target="_blank">firewalls</a> and traffic monitoring. Brute force attempts and bots scanning for open ports are far more common than most people realize, and anything that you can do to reduce your attack surface is a positive.</p>
<p>Port forwarding can also be problematic if your ISP assigns you a <a href="https://www.xda-developers.com/dynamic-ip-address-better-than-static/" rel="noopener noreferrer" target="_blank">dynamic IP </a>address. You'll want to set up a dynamic DNS like No-IP to handle configuration changes, or you could find yourself locked out of your own network while trying to access it remotely. Moreover, it's a hassle to set up on a per-device and software basis. Plex might require specific ports to be opened, while a game server might require a different set of ports. You catch my drift. It's just too easy to drop your guard.</p> <p>Tailscale makes it much simpler. For one, the software uses the fully encrypted and secure <a href="https://www.xda-developers.com/benefits-using-wireguard-home-lab-openvpn/" rel="noopener noreferrer" target="_blank">Wireguard</a> protocol to create its internet tunnels. Moreover, the service doesn't need to expose any ports to the internet. Instead, it creates an encrypted peer-to-peer mesh network between all your devices.</p> <p>Instead, every computer or smartphone that you install Tailscale on becomes part of a private network between all these devices and any other person that you might invite to your network. This allows all services to behave as if they are on the same local network, even though you might be halfway across the world. In fact, depending on your home network configuration, you might even be able to install it on a router to give the same local access to every device on your network.</p> <p></p><h2>
                        Simple setup with no router configuration needed
               </h2>
<p>If that's not enough, Tailscale offers additional benefits as well. Not only does it ensure that your ports stay closed and secure, but it is also significantly easier to install. Almost as easy as installing an app from your computer or smartphone's app store. You can even install it directly on your NAS. There's no digging into router configurations.</p> <p>All you've got to do is head on over to the Tailscale website, download the relevant app for your operating system, sign in, and you're good to go. Once the client is installed and linked to your account, that's it. The device becomes a part of your local network and gets its own IP address. No worrying about dynamic IP or DNS.</p>
<p>Of course, that's just touching the surface. Tailscale offers a range of additional functionality, including advanced solutions such as access control lists and <a href="https://www.xda-developers.com/network-features-i-regret-not-using-sooner/" rel="noopener noreferrer" target="_blank">network segmentation</a>. If your family members are part of your Tailscale tunnel, you can choose to restrict their access to just a media server <a href="https://www.xda-developers.com/best-and-worst-nas-apps/" rel="noopener noreferrer" target="_blank">NAS</a> but not to other aspects of your home network. It's very flexible. Moreover, built on the WireGuard protocol, it's incredibly efficient at keeping latency levels low.</p> <p>Another feature that makes WireGuard an overall winner has nothing to do with port-forwarding, but everything to do with convenience. You see, WireGuard effectively operates like a VPN tunnel between your devices and home network. Using WireGuard's Funnel feature, you can use split tunneling to control which aspects of your device act like it's on a local network, and which part accesses the open web.</p> <h3>
            Tailscale makes remote access simple and secure
    </h3>
<p>Look, port forwarding is a credible solution in many situations. But if your goal is to reach specific devices and <a href="https://www.xda-developers.com/self-hosting-apps-wish-knew-sooner/" rel="noopener noreferrer" target="_blank">self-hosted</a> services over the internet, there are better ways to achieve it. Tailscale helps you avoid unnecessary risks, IP changes, and more. Creating a private, encrypted network between your devices makes remote access safer, simpler, and far more reliable. P</p> <p>Additionally, features like its easy scalability through apps, advanced network management features, and split-tunneling make it a no-brainer if it fits your needs.</p> <p></p>

content_text:
Running a Plex server? Or, for that matter, any server, maybe a dashboard you need external access for, or even a personal website — port

forwarding

is one of the most common issues to come across. It sounds simple, and on paper it is.

All you've got to do is get your router to open a port, point it at the right device or service, and let the internet pass through. But it rarely works as easily as that. Fact of the matter is, even when using high-end equipment, port forwarding can be a never-ending hassle. But there's a better way to handle this. Just install

Tailscale

instead.

Why Tailscale is better than port-forwarding

A secure mesh that just works

Unlike port forwarding that pokes holes in your home network,

Tailscale

creates a secure mesh internet connection between your devices. Effectively, this allows all your devices to behave as if they were on the same local network. Moreover, Tailscale works across different internet connections and without a complicated router setup. While port forwarding is a credible solution, the fact of the matter is that Tailscale is the better option for most users.

Now, if you're wondering why you need this when

port forwarding

works just the same, I've got your back. As I mentioned earlier, port forwarding basically opens specific doors or windows into your home network. These doorways allow internet traffic between, say, your

Plex server

and a remote client, but they can also be scanned, and ports left open inadvertently can be exploited by others wanting to use your network.

It's a security risk waiting to happen unless you are diligent about setting up

firewalls

and traffic monitoring. Brute force attempts and bots scanning for open ports are far more common than most people realize, and anything that you can do to reduce your attack surface is a positive.

Port forwarding can also be problematic if your ISP assigns you a

dynamic IP

address. You'll want to set up a dynamic DNS like No-IP to handle configuration changes, or you could find yourself locked out of your own network while trying to access it remotely. Moreover, it's a hassle to set up on a per-device and software basis. Plex might require specific ports to be opened, while a game server might require a different set of ports. You catch my drift. It's just too easy to drop your guard.

Tailscale makes it much simpler. For one, the software uses the fully encrypted and secure

Wireguard

protocol to create its internet tunnels. Moreover, the service doesn't need to expose any ports to the internet. Instead, it creates an encrypted peer-to-peer mesh network between all your devices.

Instead, every computer or smartphone that you install Tailscale on becomes part of a private network between all these devices and any other person that you might invite to your network. This allows all services to behave as if they are on the same local network, even though you might be halfway across the world. In fact, depending on your home network configuration, you might even be able to install it on a router to give the same local access to every device on your network.

Simple setup with no router configuration needed

If that's not enough, Tailscale offers additional benefits as well. Not only does it ensure that your ports stay closed and secure, but it is also significantly easier to install. Almost as easy as installing an app from your computer or smartphone's app store. You can even install it directly on your NAS. There's no digging into router configurations.

All you've got to do is head on over to the Tailscale website, download the relevant app for your operating system, sign in, and you're good to go. Once the client is installed and linked to your account, that's it. The device becomes a part of your local network and gets its own IP address. No worrying about dynamic IP or DNS.

Of course, that's just touching the surface. Tailscale offers a range of additional functionality, including advanced solutions such as access control lists and

network segmentation

. If your family members are part of your Tailscale tunnel, you can choose to restrict their access to just a media server

NAS

but not to other aspects of your home network. It's very flexible. Moreover, built on the WireGuard protocol, it's incredibly efficient at keeping latency levels low.

Another feature that makes WireGuard an overall winner has nothing to do with port-forwarding, but everything to do with convenience. You see, WireGuard effectively operates like a VPN tunnel between your devices and home network. Using WireGuard's Funnel feature, you can use split tunneling to control which aspects of your device act like it's on a local network, and which part accesses the open web.

Tailscale makes remote access simple and secure

Look, port forwarding is a credible solution in many situations. But if your goal is to reach specific devices and

self-hosted

services over the internet, there are better ways to achieve it. Tailscale helps you avoid unnecessary risks, IP changes, and more. Creating a private, encrypted network between your devices makes remote access safer, simpler, and far more reliable. P

Additionally, features like its easy scalability through apps, advanced network management features, and split-tunneling make it a no-brainer if it fits your needs.

thumb_url:
https://static0.xdaimages.com/wordpress/wp-content/uploads/wm/2025/09/tailscale-website-on-laptop.jpeg?w=1600&h=900&fit=crop


url: https://silvercut.com/products/keychain
title: Silvercut® Life-Like Dog Keychain With Custom Engraved Photo
added_at: 2025-09-14 13:34:17
content_html:
<p>If your product arrives damaged, incorrect, or not as expected, we will make it right. This may include a free remake, partial refund, or another solution depending on the issue.</p><p>Our support team reviews each case individually and typically responds within one business day. We do not offer returns for custom products, but we stand by the quality of our work.</p><p>You can read verified customer reviews where we are rated 4.6 / 5 stars.</p>

content_text:
If your product arrives damaged, incorrect, or not as expected, we will make it right. This may include a free remake, partial refund, or another solution depending on the issue.

Our support team reviews each case individually and typically responds within one business day. We do not offer returns for custom products, but we stand by the quality of our work.

You can read verified customer reviews where we are rated 4.6 / 5 stars.

thumb_url:
http://silvercut.com/cdn/shop/products/Life_like_2_-_White_-_Silver_-_Front_b7d3165c-62c8-447f-aa04-0c5836010319.jpg?v=1756974411


url: https://ppc.land/google-engineer-releases-free-400-page-guide-to-agentic-ai-systems/
title: Google engineer releases free 400-page guide to agentic AI systems
added_at: 2025-09-15 15:19:08
thumb_url: https://ppc.land/content/images/size/w1200/2025/09/Agentic-Design-Patterns.webp
content_html:
<section>
<p>A Google distinguished engineer has published a comprehensive 400-page technical guide to building autonomous AI systems, offering detailed blueprints for creating sophisticated artificial intelligence agents. <a href="https://www.linkedin.com/in/searchguy/?ref=ppc.land" rel="noopener noreferrer" target="_blank">Antonio Gulli</a>, Senior Director and Distinguished Engineer in Google's CTO Office, announced <a href="https://docs.google.com/document/d/1rsaK53T3Lg5KoGwvf8ukOUvbELRtH-V0LnOIFDxBryE/preview?tab=t.0&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Agentic Design Patterns: A Hands-On Guide to Building Intelligent Systems</a> with a scheduled release date of December 3, 2025.</p><p>The publication addresses a critical gap in AI development methodology. According to Gulli, building effective agentic systems requires more than just a powerful language model—it demands structured architectural blueprints. "It's about moving from raw capability to robust, real-world applications," Gulli stated in the book's introduction.</p>




<p><b><strong>Subscribe PPC Land newsletter</strong></b> ✉️ for similar stories like this one. Receive the news every day in your inbox. Free of ads. 10 USD per year.</p>

<a href="https://ppc.land/google-engineer-releases-free-400-page-guide-to-agentic-ai-systems/#/portal/signup" rel="noopener noreferrer" target="_blank">
                            Subscribe
                        </a>



<p>The guide presents 21 distinct agentic patterns that serve as fundamental building blocks for autonomous AI systems. These patterns range from foundational concepts such as Prompt Chaining and Tool Use to advanced implementations including Multi-Agent Collaboration and Self-Correction frameworks. Each pattern represents a reusable solution to common challenges encountered when building intelligent, goal-oriented systems.</p><p>Technical specifications detailed in the book cover multiple implementation frameworks. The guide utilizes three prominent development platforms: LangChain and its extension LangGraph for building complex operational sequences, CrewAI for orchestrating multiple agents, and the Google Agent Developer Kit for evaluation and deployment processes. This multi-framework approach ensures broad applicability across different technical environments.</p><p>The publication structure follows a practical methodology. Each chapter focuses on a single agentic pattern, providing pattern overviews, use cases, hands-on code examples, and key takeaways. According to the table of contents, Part One covers 103 pages of core execution patterns including Prompt Chaining, Routing, Parallelization, Reflection, Tool Use, Planning, and Multi-Agent systems.</p><p>Part Two addresses 61 pages of memory management and learning capabilities. This section explores Memory Management, Learning and Adaptation, Model Context Protocol (MCP), and Goal Setting frameworks. The technical depth continues through Parts Three and Four, covering 114 pages of advanced topics including Exception Handling, Human-in-the-Loop patterns, Knowledge Retrieval, and Safety implementations.</p>




<p>Buy ads on PPC Land. PPC Land has standard and native ad formats via major DSPs and ad platforms like Google Ads. Via an auction CPM, you can reach industry professionals.</p>

<a href="https://ppc.land/advertise/" rel="noopener noreferrer" target="_blank">
                            Learn more
                        </a>



<p>The book's technical approach emphasizes practical implementation over theoretical discussion. According to the publication details, the guide includes executable code examples, architectural diagrams, and step-by-step implementation instructions. This hands-on methodology addresses the growing demand for actionable AI development resources in enterprise environments.</p><p>Industry validation for the guide emerged through social media discussions among AI practitioners. Multiple technology leaders shared positive assessments of the publication's practical value. The book received recognition as a "#1 New Release in Probability &amp; Statistics" on <a href="https://www.amazon.com/Agentic-Design-Patterns-Hands-Intelligent/dp/3032014018/?ref=ppc.land" rel="noopener noreferrer" target="_blank">Amazon</a> with a December 3, 2025 release date.</p><p>Gulli brings extensive technical credentials to the publication. His background includes over 30 years of relevant experience in AI, Search, and Cloud technologies. He holds a Ph.D. in Computer Science from the University of Pisa and has previously authored technical publications including "Deep Learning for Keras" across multiple editions and languages.</p><p>The economic context for agentic AI development shows significant market potential. <a href="https://ppc.land/google-cloud-projects-1-trillion-agentic-ai-market-by-2040/" rel="noopener noreferrer" target="_blank">Recent research published on PPC Land indicates Google Cloud projects the agentic AI market could reach $1 trillion by 2040</a>, with 90% enterprise adoption expected. This projection reflects growing demand for autonomous AI systems capable of executing complex workflows with minimal human intervention.</p><p>The timing of Gulli's publication coincides with increased industry focus on AI agent development. <a href="https://ppc.land/major-tech-companies-unveil-ai-agent-guides/" rel="noopener noreferrer" target="_blank">Major technology companies have recently released comprehensive AI agent guides</a>, marking a shift toward more autonomous systems. Companies including Anthropic, OpenAI, and McKinsey have published complementary resources, though Gulli's guide stands out for its comprehensive technical depth and practical implementation focus.</p><p>The book addresses critical challenges in AI agent reliability and safety. Traditional single-prompt interactions often prove insufficient for complex, multi-step tasks. Agentic patterns provide structured approaches to decomposing complex objectives into manageable components while maintaining coherence across extended workflows.</p><p>Pattern composition represents a key advancement outlined in the guide. The publication demonstrates how individual patterns combine to create sophisticated systems. For example, an autonomous research assistant might integrate Planning patterns for task decomposition, Tool Use for information gathering, Multi-Agent Collaboration for specialized analysis, and Reflection for quality assurance.</p><p>Memory Management patterns detailed in the book enable agents to maintain context across interactions while learning from experience. These capabilities distinguish true agentic systems from simple reactive models. The technical specifications include both short-term conversational context and long-term knowledge retention mechanisms.</p><p>Safety and alignment considerations receive dedicated coverage through specialized "Guardrails/Safety Patterns." These frameworks address challenges of autonomous operation while maintaining alignment with intended objectives. The patterns include input validation, output filtering, human oversight integration, and graceful degradation capabilities.</p><p>The publication includes extensive technical documentation spanning 424 total pages. Appendices provide advanced prompting techniques, framework overviews, and implementation guidelines. A comprehensive glossary defines technical terms and concepts used throughout the guide.</p><p>Distribution of the guide follows open-access principles. Google has made the technical documentation publicly available through standard channels, enabling widespread practitioner access. This approach supports broader adoption of structured AI agent development methodologies across the industry.</p><h2>Agentic Design Patterns (the book in Google Docs - not in pdf)</h2><h3>Part One</h3><ol><li><a href="https://docs.google.com/document/d/1flxKGrbnF2g8yh3F-oVD5Xx7ZumId56HbFpIiPdkqLI/edit?usp=sharing&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 1: Prompt Chaining</a> (<a href="https://colab.research.google.com/drive/15XCzDOvBhIQaZ__xkvruf5sP9OznAbK9?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li><li><a href="https://docs.google.com/document/d/1ux_n8n3T4bYndOjs1DKW5ccpC802KISdy2IWnlvYbas/edit?tab=t.0&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 2: Routing</a> (<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li><li><a href="https://docs.google.com/document/d/1XVMp4RcRkoUJTVbrP2foWZX703CUJpWkrhyFU2cfUOA/edit?tab=t.0&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 3: Parallelization</a> (<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li><li><a href="https://docs.google.com/document/d/1HXXJOQIMWowtLw4WMiSR360caDAlZPtl5dPPgvq9IT4/edit?tab=t.0&amp;ref=ppc.land#heading=h.a7nkedxjnyap" rel="noopener noreferrer" target="_blank">Chapter 4: Reflection </a>(<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li><li><a href="https://docs.google.com/document/d/1bE4iMljhppqGY1p48gQWtZvk6MfRuJRCiba1yRykGNE/edit?usp=sharing&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 5: Tool Use</a> <a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">(code</a>)</li><li><a href="https://docs.google.com/document/d/18vvNESEwHnVUREzIipuaDNCnNAREGqEfy9MQYC9wb4o/edit?usp=sharing&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 6: Planning</a> (<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li><li><a href="https://docs.google.com/document/d/1RZ5-2fykDQKOBx01pwfKkDe0GCs5ydca7xW9Q4wqS_M/edit?tab=t.0&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 7: Multi-Agent</a> (<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li></ol><h3>Part Two</h3><ol><li><a href="https://docs.google.com/document/d/1asVTObtzIye0I9ypAztaeeI_sr_Hx2TORE02uUuqH_c/edit?tab=t.0&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 8: Memory Management</a> (<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li><li><a href="https://docs.google.com/document/d/1UHTEDCmSM1nwB-iyMoHuYzVcu_B_4KkJ2ITGGUKqo8s/edit?tab=t.0&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 9: Learning and Adaptation</a> (<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li><li><a href="https://docs.google.com/document/d/1e6XimYczKmhX9zpqEyxLFWPQgGuG0brp7Hic2sFl_qw/edit?usp=sharing&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 10: Model Context Protocol (MCP)</a> (<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li><li><a href="https://docs.google.com/document/d/10ndlCB39BWjyFRWKpcoKib4vuPD1ojD-x0-ynMaf5uw/edit?tab=t.0&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 11: Goal Setting and Monitoring</a> (<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li></ol><h3>Part Three</h3><ol><li><a href="https://docs.google.com/document/d/1C07AuMur6-infwE0viCp4QtAy_wWI-uceFm6MaYHQGk/edit?tab=t.0&amp;ref=ppc.land#heading=h.m2kk4kdjt6ir" rel="noopener noreferrer" target="_blank">Chapter 12: Exception Handling and Recovery</a> (<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li><li><a href="https://docs.google.com/document/d/1ImOZcw6yeb7a-uRBMNP1VdovYfyip4IdsAcLu9yue-0/edit?usp=sharing&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 13: Human-in-the-Loop</a> (<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li><li><a href="https://docs.google.com/document/d/1v96Oobio6xDOqbK8ejsXjmOc4Dp2uoLMo5_gfJgi-NE/edit?usp=sharing&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 14: Knowledge Retrieval (RAG)</a> (<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li></ol><h3>Part Four</h3><ol><li><a href="https://docs.google.com/document/d/1H6HmUYcy5kugt5gt7Kh2Zzb8C62d5pu36RsgMNDCX24/edit?usp=sharing&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 15: Inter-Agent Communication (A2A</a>) (<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li><li><a href="https://docs.google.com/document/d/1nAN58l6JjqEJHk43126uh7xgdEblCpcbsNUHXgtBmJQ/edit?usp=sharing&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 16: Resource-Aware Optimization</a> (<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li><li><a href="https://docs.google.com/document/d/1Yt1W_hLaC6ZNgJXfT4W6NrCL4TzNVdKOX50kgpHiIq4/edit?usp=sharing&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 17: Reasoning Techniques</a> (<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li><li><a href="https://docs.google.com/document/d/1Gpc5af_okze1kprRLohP6-81e1KwL6HggjeLvxQyIuk/edit?usp=sharing&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 18: Guardrails/Safety Patterns</a> (<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li><li><a href="https://docs.google.com/document/d/1G3zOZM2ZOd0gUp5dy66FUjKMOcALh9l-JpvPxgGMm8w/edit?usp=sharing&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 19: Evaluation and Monitoring</a> (<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li><li><a href="https://docs.google.com/document/d/1qyXxGM2hNqW_qjXuBFxrEUeoYVO79BoW1ogKu1bfdCY/edit?usp=sharing&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 20: Prioritization</a> (<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li><li><a href="https://docs.google.com/document/d/1zeeMVTqjqRIli6G9MMWThhoQhvKqLOjJF2EHHUXLhdk/edit?usp=sharing&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Chapter 21: Exploration and Discovery</a> (<a href="https://drive.google.com/drive/u/0/folders/1Y3U3IrYCiJ3E45Z8okR5eCg7OPnWQtPV?ref=ppc.land" rel="noopener noreferrer" target="_blank">code</a>)</li></ol><h3>Appendix</h3><ol><li><a href="https://docs.google.com/document/d/1V7EKEWibOH6IhHD_PtbFZiml492-2191jDQCcTkhtTI/edit?usp=sharing&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Appendix A: Advanced Prompting Techniques</a></li><li><a href="https://docs.google.com/document/d/11pma_tCoC7uZ2SFKjcR5KyIq0_ooMGSoadI6f9mxG2I/edit?tab=t.0&amp;ref=ppc.land#heading=h.4bsgiycebw26" rel="noopener noreferrer" target="_blank">Appendix B - AI Agentic ….: From GUI to Real world environment</a></li><li><a href="https://docs.google.com/document/d/151rGsiEYOkXUcNDRus_N8TxxuvjoyTDViBhzt9z0Mfw/edit?tab=t.0&amp;ref=ppc.land#heading=h.8y2llpsk10ds" rel="noopener noreferrer" target="_blank">Appendix C - Quick overview of Agentic Frameworks</a></li><li><a href="https://docs.google.com/document/d/1bDRJ8mKtLTeWNC-cGD0Cr8pEJQgJHNcjqz5ekloAjaE/edit?tab=t.0&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Appendix D - Building an Agent with AgentSpace (on-line only)</a></li><li><a href="http://docs.google.com/document/d/1W4znto0a8Ikajw5a4tEyRAaB2nJPJw_iFc4w4qNnjho/edit?tab=t.0&amp;ref=ppc.land#heading=h.6zaq0kvy131m" rel="noopener noreferrer" target="_blank">Appendix E - AI Agents on the CLI (online)</a></li><li><a href="https://docs.google.com/document/d/14q3fQ-FZmDgiughno_WLSILMWkURvUgR7mlGiFtvwd4/edit?tab=t.0&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Appendix F - Under the Hood: An Inside Look at the Agents’ Reasoning Engines</a></li><li><a href="https://docs.google.com/document/d/1tVyhgwrD4fu_D_pHUrwhNxoguRG3tLc1KObXFxrxE_s/edit?tab=t.0&amp;ref=ppc.land" rel="noopener noreferrer" target="_blank">Appendix G -  Coding agents</a></li></ol><p><strong>The </strong><a href="https://docs.google.com/document/d/1rsaK53T3Lg5KoGwvf8ukOUvbELRtH-V0LnOIFDxBryE/preview?ref=ppc.land&amp;tab=t.0#heading=h.pxcur8v2qagu" rel="noopener noreferrer" target="_blank"><strong>table of contents</strong></a></p><h2>Why this matters for marketing</h2><p>The release of this comprehensive guide signals the maturation of agentic AI from experimental technology to practical implementation framework. For marketing professionals, these developments indicate significant opportunities for campaign automation and optimization capabilities that extend far beyond current programmatic advertising approaches.</p><p><a href="https://ppc.land/ahrefs-study-finds-ai-search-visitors-convert-23x-higher-than-organic-traffic/" rel="noopener noreferrer" target="_blank">The emergence of agentic AI capabilities in marketing contexts has already shown measurable impact</a>, with AI search traffic converting at rates 23 times higher than traditional organic search visitors despite representing minimal traffic volume. This pattern suggests that AI-powered systems are fundamentally changing how users discover and interact with content.</p><p><a href="https://ppc.land/google-search-calls-small-businesses-for-customers-automatically/" rel="noopener noreferrer" target="_blank">Google's recent introduction of automated calling features</a> demonstrates practical agentic implementations in customer service contexts. The system autonomously contacts businesses to gather pricing and availability information on behalf of users, representing the type of goal-oriented behavior that Gulli's patterns enable at scale.</p><p>The technical frameworks outlined in the guide provide marketing teams with structured approaches to building custom AI agents for campaign management, content optimization, and customer interaction automation. Rather than relying on black-box solutions, these patterns enable transparent, controllable implementations that align with specific business objectives.</p>




<p><b><strong>Subscribe PPC Land newsletter</strong></b> ✉️ for similar stories like this one. Receive the news every day in your inbox. Free of ads. 10 USD per year.</p>

<a href="https://ppc.land/google-engineer-releases-free-400-page-guide-to-agentic-ai-systems/#/portal/signup" rel="noopener noreferrer" target="_blank">
                            Subscribe
                        </a>



<h2>Timeline</h2>




<p><b><strong>Subscribe PPC Land newsletter</strong></b> ✉️ for similar stories like this one. Receive the news every day in your inbox. Free of ads. 10 USD per year.</p>

<a href="https://ppc.land/google-engineer-releases-free-400-page-guide-to-agentic-ai-systems/#/portal/signup" rel="noopener noreferrer" target="_blank">
                            Subscribe
                        </a>



<h2>Summary</h2><p><strong>Who</strong>: Antonio Gulli, Senior Director and Distinguished Engineer in Google's CTO Office, with over 30 years of experience in AI, Search, and Cloud technologies and a Ph.D. in Computer Science from the University of Pisa.</p><p><strong>What</strong>: A comprehensive 400-page technical guide titled "Agentic Design Patterns: A Hands-On Guide to Building Intelligent Systems" that presents 21 distinct patterns for building autonomous AI agents, covering everything from basic prompt chaining to advanced multi-agent collaboration frameworks.</p><p><strong>When</strong>: Announced with a scheduled release date of December 3, 2025, with the book being listed as a "#1 New Release in Probability &amp; Statistics" on Amazon.</p><p><strong>Where</strong>: Announced through multiple channels including social media and Amazon pre-orders, with Google making the technical documentation publicly available through standard distribution mechanisms.</p><p><strong>Why</strong>: The guide addresses the critical gap between powerful language models and practical autonomous systems, providing structured architectural blueprints necessary for building reliable, goal-oriented AI agents that can operate with minimal human intervention in real-world applications.</p>
</section>

content_text:
A Google distinguished engineer has published a comprehensive 400-page technical guide to building autonomous AI systems, offering detailed blueprints for creating sophisticated artificial intelligence agents.

Antonio Gulli

, Senior Director and Distinguished Engineer in Google's CTO Office, announced

Agentic Design Patterns: A Hands-On Guide to Building Intelligent Systems

with a scheduled release date of December 3, 2025.

The publication addresses a critical gap in AI development methodology. According to Gulli, building effective agentic systems requires more than just a powerful language model—it demands structured architectural blueprints. "It's about moving from raw capability to robust, real-world applications," Gulli stated in the book's introduction.

Subscribe PPC Land newsletter

✉️ for similar stories like this one. Receive the news every day in your inbox. Free of ads. 10 USD per year.

Subscribe

The guide presents 21 distinct agentic patterns that serve as fundamental building blocks for autonomous AI systems. These patterns range from foundational concepts such as Prompt Chaining and Tool Use to advanced implementations including Multi-Agent Collaboration and Self-Correction frameworks. Each pattern represents a reusable solution to common challenges encountered when building intelligent, goal-oriented systems.

Technical specifications detailed in the book cover multiple implementation frameworks. The guide utilizes three prominent development platforms: LangChain and its extension LangGraph for building complex operational sequences, CrewAI for orchestrating multiple agents, and the Google Agent Developer Kit for evaluation and deployment processes. This multi-framework approach ensures broad applicability across different technical environments.

The publication structure follows a practical methodology. Each chapter focuses on a single agentic pattern, providing pattern overviews, use cases, hands-on code examples, and key takeaways. According to the table of contents, Part One covers 103 pages of core execution patterns including Prompt Chaining, Routing, Parallelization, Reflection, Tool Use, Planning, and Multi-Agent systems.

Part Two addresses 61 pages of memory management and learning capabilities. This section explores Memory Management, Learning and Adaptation, Model Context Protocol (MCP), and Goal Setting frameworks. The technical depth continues through Parts Three and Four, covering 114 pages of advanced topics including Exception Handling, Human-in-the-Loop patterns, Knowledge Retrieval, and Safety implementations.

Buy ads on PPC Land. PPC Land has standard and native ad formats via major DSPs and ad platforms like Google Ads. Via an auction CPM, you can reach industry professionals.

Learn more

The book's technical approach emphasizes practical implementation over theoretical discussion. According to the publication details, the guide includes executable code examples, architectural diagrams, and step-by-step implementation instructions. This hands-on methodology addresses the growing demand for actionable AI development resources in enterprise environments.

Industry validation for the guide emerged through social media discussions among AI practitioners. Multiple technology leaders shared positive assessments of the publication's practical value. The book received recognition as a "#1 New Release in Probability & Statistics" on

Amazon

with a December 3, 2025 release date.

Gulli brings extensive technical credentials to the publication. His background includes over 30 years of relevant experience in AI, Search, and Cloud technologies. He holds a Ph.D. in Computer Science from the University of Pisa and has previously authored technical publications including "Deep Learning for Keras" across multiple editions and languages.

The economic context for agentic AI development shows significant market potential.

Recent research published on PPC Land indicates Google Cloud projects the agentic AI market could reach $1 trillion by 2040

, with 90% enterprise adoption expected. This projection reflects growing demand for autonomous AI systems capable of executing complex workflows with minimal human intervention.

The timing of Gulli's publication coincides with increased industry focus on AI agent development.

Major technology companies have recently released comprehensive AI agent guides

, marking a shift toward more autonomous systems. Companies including Anthropic, OpenAI, and McKinsey have published complementary resources, though Gulli's guide stands out for its comprehensive technical depth and practical implementation focus.

The book addresses critical challenges in AI agent reliability and safety. Traditional single-prompt interactions often prove insufficient for complex, multi-step tasks. Agentic patterns provide structured approaches to decomposing complex objectives into manageable components while maintaining coherence across extended workflows.

Pattern composition represents a key advancement outlined in the guide. The publication demonstrates how individual patterns combine to create sophisticated systems. For example, an autonomous research assistant might integrate Planning patterns for task decomposition, Tool Use for information gathering, Multi-Agent Collaboration for specialized analysis, and Reflection for quality assurance.

Memory Management patterns detailed in the book enable agents to maintain context across interactions while learning from experience. These capabilities distinguish true agentic systems from simple reactive models. The technical specifications include both short-term conversational context and long-term knowledge retention mechanisms.

Safety and alignment considerations receive dedicated coverage through specialized "Guardrails/Safety Patterns." These frameworks address challenges of autonomous operation while maintaining alignment with intended objectives. The patterns include input validation, output filtering, human oversight integration, and graceful degradation capabilities.

The publication includes extensive technical documentation spanning 424 total pages. Appendices provide advanced prompting techniques, framework overviews, and implementation guidelines. A comprehensive glossary defines technical terms and concepts used throughout the guide.

Distribution of the guide follows open-access principles. Google has made the technical documentation publicly available through standard channels, enabling widespread practitioner access. This approach supports broader adoption of structured AI agent development methodologies across the industry.

Agentic Design Patterns (the book in Google Docs - not in pdf)

Part One

Chapter 1: Prompt Chaining

(

code

)

Chapter 2: Routing

(

code

)

Chapter 3: Parallelization

(

code

)

Chapter 4: Reflection

(

code

)

Chapter 5: Tool Use

(code

)

Chapter 6: Planning

(

code

)

Chapter 7: Multi-Agent

(

code

)

Part Two

Chapter 8: Memory Management

(

code

)

Chapter 9: Learning and Adaptation

(

code

)

Chapter 10: Model Context Protocol (MCP)

(

code

)

Chapter 11: Goal Setting and Monitoring

(

code

)

Part Three

Chapter 12: Exception Handling and Recovery

(

code

)

Chapter 13: Human-in-the-Loop

(

code

)

Chapter 14: Knowledge Retrieval (RAG)

(

code

)

Part Four

Chapter 15: Inter-Agent Communication (A2A

) (

code

)

Chapter 16: Resource-Aware Optimization

(

code

)

Chapter 17: Reasoning Techniques

(

code

)

Chapter 18: Guardrails/Safety Patterns

(

code

)

Chapter 19: Evaluation and Monitoring

(

code

)

Chapter 20: Prioritization

(

code

)

Chapter 21: Exploration and Discovery

(

code

)

Appendix

Appendix A: Advanced Prompting Techniques

Appendix B - AI Agentic ….: From GUI to Real world environment

Appendix C - Quick overview of Agentic Frameworks

Appendix D - Building an Agent with AgentSpace (on-line only)

Appendix E - AI Agents on the CLI (online)

Appendix F - Under the Hood: An Inside Look at the Agents’ Reasoning Engines

Appendix G -  Coding agents

The

table of contents

Why this matters for marketing

The release of this comprehensive guide signals the maturation of agentic AI from experimental technology to practical implementation framework. For marketing professionals, these developments indicate significant opportunities for campaign automation and optimization capabilities that extend far beyond current programmatic advertising approaches.

The emergence of agentic AI capabilities in marketing contexts has already shown measurable impact

, with AI search traffic converting at rates 23 times higher than traditional organic search visitors despite representing minimal traffic volume. This pattern suggests that AI-powered systems are fundamentally changing how users discover and interact with content.

Google's recent introduction of automated calling features

demonstrates practical agentic implementations in customer service contexts. The system autonomously contacts businesses to gather pricing and availability information on behalf of users, representing the type of goal-oriented behavior that Gulli's patterns enable at scale.

The technical frameworks outlined in the guide provide marketing teams with structured approaches to building custom AI agents for campaign management, content optimization, and customer interaction automation. Rather than relying on black-box solutions, these patterns enable transparent, controllable implementations that align with specific business objectives.

Subscribe PPC Land newsletter

✉️ for similar stories like this one. Receive the news every day in your inbox. Free of ads. 10 USD per year.

Subscribe

Timeline

Subscribe PPC Land newsletter

✉️ for similar stories like this one. Receive the news every day in your inbox. Free of ads. 10 USD per year.

Subscribe

Summary

Who

: Antonio Gulli, Senior Director and Distinguished Engineer in Google's CTO Office, with over 30 years of experience in AI, Search, and Cloud technologies and a Ph.D. in Computer Science from the University of Pisa.

What

: A comprehensive 400-page technical guide titled "Agentic Design Patterns: A Hands-On Guide to Building Intelligent Systems" that presents 21 distinct patterns for building autonomous AI agents, covering everything from basic prompt chaining to advanced multi-agent collaboration frameworks.

When

: Announced with a scheduled release date of December 3, 2025, with the book being listed as a "#1 New Release in Probability & Statistics" on Amazon.

Where

: Announced through multiple channels including social media and Amazon pre-orders, with Google making the technical documentation publicly available through standard distribution mechanisms.

Why

: The guide addresses the critical gap between powerful language models and practical autonomous systems, providing structured architectural blueprints necessary for building reliable, goal-oriented AI agents that can operate with minimal human intervention in real-world applications.


url: https://scriptingosx.com/2019/06/moving-to-zsh/
title: Moving to zsh – Scripting OS X
added_at: 2025-09-29 01:25:31
thumb_url: https://i0.wp.com/scriptingosx.com/wp-content/uploads/2019/06/typewriter.jpg?fit=800%2C326&ssl=1
content_html:
<p>Apple has announced that in <a href="https://www.apple.com/newsroom/2019/06/apple-previews-macos-catalina/" rel="noopener noreferrer" target="_blank">macOS 10.15 Catalina</a> the <a href="https://support.apple.com/en-us/HT208050" rel="noopener noreferrer" target="_blank">default shell will be <code>zsh</code></a>.</p>
<p>In this series, I will document my experiences moving <code>bash</code> settings, configurations, and scripts over to <code>zsh</code>.</p>
<p><code>zsh</code> (I believe it is pronounced <em>zee-shell</em>, though <em>zish</em> is fun to say) will succeed <code>bash</code> as the default shell. <code>bash</code> has been the default shell since Mac OS X 10.3 Panther.</p>
<blockquote><p>This series <a href="https://scriptingosx.com/2019/11/new-book-release-day-moving-to-zsh/" rel="noopener noreferrer" target="_blank">has grown into a book</a>: reworked and expanded with more detail and topics. <a href="https://scriptingosx.com/2019/10/packaging-book-update-v1-10/" rel="noopener noreferrer" target="_blank">Like my other books,</a> I plan to update and add to it after release as well, keeping it relevant and useful. You <a href="https://books.apple.com/us/book/moving-to-zsh/id1483591353?mt=11&amp;app=itunes&amp;at=1001lpyX" rel="noopener noreferrer" target="_blank">can order it on the Apple Books Store now</a>.</p></blockquote>
<h2>Why?</h2>
<p>The <code>bash</code> binary bundled with macOS has been stuck on version 3.2 for a long time now. <code>bash</code> v4 was released in 2009 and <code>bash</code> v5 in January 2019. The reason Apple has not switched to these newer versions is that they are licensed with GPL v3. <code>bash</code> v3 is still GPL v2.</p>
<p><code>zsh</code>, on the other hand, has an ‘MIT-like’ license, which makes it much more palatable for Apple to include in the system by default. <code>zsh</code> has been available as on macOS for a long time. The <code>zsh</code> version on macOS 10.14 Mojave is fairly new (5.3). macOS 10.15 Catalina has the current <code>zsh</code> 5.7.1.</p>
<h2>Is bash gone!?</h2>
<p>No.</p>
<p>macOS Catalina still has the same <code>/bin/bash</code> (version 3.2.57) as Mojave and earlier macOS versions. This change is only for new accounts created on macOS Catalina. When you upgrade to Catalina, a user’s default shell will remain what it was before.</p>
<p>Many scripts in macOS, management systems, and Apple and third party installers rely on <code>/bin/bash</code>. If Apple just yanked this binary in macOS 10.15 Catalina or even 10.16. Many installers and other solutions would break and simply cease to function.</p>
<p>Users that have <code>/bin/bash</code> as their default shell on Catalina will see a prompt at the start of each Terminal session stating that <code>zsh</code> is now the recommended default shell. If you want to continue using <code>/bin/bash</code>, you can supress this message by <a href="https://scriptingosx.com/2017/04/about-bash_profile-and-bashrc-on-macos/" rel="noopener noreferrer" target="_blank">setting an environment variable in your <code>.bash_profile</code> or <code>.bashrc</code></a>.</p>
<pre><code>export BASH_SILENCE_DEPRECATION_WARNING=1
</code></pre>
<p>You can also <a href="https://scriptingosx.com/2019/02/install-bash-5-on-macos/" rel="noopener noreferrer" target="_blank">download and install a newer version of bash</a> yourself. Keep in mind that custom bash installations reside in a different directory, usually <code>/usr/local/bin/bash</code>.</p>
<h2>Will bash remain indefinitely?</h2>
<p>Apple is strongly messaging that you should switch shells. This is different from the last switch in Mac OS X 10.3 Panther, when Apple switched the default to <code>bash</code>, but didn’t really care if you remained on <code>tcsh</code>. In fact, <code>tcsh</code> is still present on macOS.</p>
<p>Apple’s messaging should tell us, that the days of <code>/bin/bash</code> <em>are</em> numbered. Probably not <em>very</em> soon, but eventually keeping a more than ten year old version of <code>bash</code> on the system will turn into a liability. The built-in bash had to be patched in 2014 to mitigate the ‘Shellshock’ vulnerability. At some point Apple will consider the cost of continued maintenance too high.</p>
<p>Another clue is that a new shell appeared on macOS Catalina (and is mentioned in the support article). The ‘Debian Almquist Shell’ <code>dash</code> has been added to the lineup of shells. <code>dash</code> is designed to be a minimal implementation of the Posix standard shell <code>sh</code>. So far, in macOS (including Catalina),<code>sh</code> invokes <code>bash</code> in <code>sh</code>-compatibility mode.</p>
<p>As <a href="https://support.apple.com/en-us/HT208050" rel="noopener noreferrer" target="_blank">Apple’s support article mentions</a>, Catalina also adds a new mechanism for users and admins to change which shell handles <code>sh</code> invocations. MacAdmins or users can change the symbolic link stored in <code>/var/select/sh</code> to point to a shell other than <code>/bin/bash</code>. This changes which shell interprets scripts the <code>#!/bin/sh</code> shebang or scripts invoked with <code>sh -c</code>. Changing the interpreter for <code>sh</code> should not, but may change the behavior of several crucial scripts in the system, management tools, and in installers, but may be very useful for testing purposes.</p>
<p>All of these changes are indicators that Apple is preparing to remove <code>/bin/bash</code> at some, yet indeterminate, time in the future.</p>
<h2>Do I need to wait for Catalina to switch to zsh?</h2>
<p>No, <code>zsh</code> is available Mojave and on older macOS versions. You can start testing <code>zsh</code> or even switch your default shell already.</p>
<p>If you want to just see how <code>zsh</code> works, you can just open Terminal and type <code>zsh</code>:</p>
<pre><code>$ zsh
MacBook%
</code></pre>
<p>The main change you will see is that the prompt looks different. <code>zsh</code> uses the <code>%</code> character as the default prompt. (You can change that, of course.) Most navigation keystrokes and other behaviors will remain the same as in <code>bash</code>.</p>
<p>If you want to already switch your default shell to <code>zsh</code> you can use the <code>chsh</code> command:</p>
<pre><code>$ chsh -s /bin/zsh
</code></pre>
<p>This will prompt for your password. This command will not change the current shell, but all new ones, so close the current Terminal windows and tabs and open a new one.</p>
<h2>How is zsh different?</h2>
<p>Like <code>bash</code> (‘Bourne again shell’ ), <code>zsh</code>derives from the ‘Bourne’ family of shells. Because of this common ancestry, it behaves very similar in day-to-day use. The most obvious change will be the different prompt.</p>
<p>The main difference between <code>bash</code> and <code>zsh</code> is configuration. Since <code>zsh</code> ignores the <code>bash</code> configuration files (<code>.bash_profile</code> or <code>.bashrc</code>) you cannot simply copy customized bash settings over to <code>zsh</code>. <code>zsh</code> has much more options and points to change <code>zsh</code> configuration and behavior. There is an entire eco-system of configuration tools and themes called <a href="https://ohmyz.sh" rel="noopener noreferrer" target="_blank"><code>oh-my-zsh</code></a> which is very popular.</p>
<p><code>zsh</code> also offers better configuration for auto-completion which is far easier than in <code>bash</code>.</p>
<p>I am planning a separate post, describing how to transfer (and translate) your configurations from <code>bash</code> to <code>zsh</code>.</p>
<h2>What about scripting?</h2>
<p>Since <code>zsh</code> has been present on macOS for a long time, you could start moving your scripts from <code>bash</code> to <code>zsh</code> right away and not lose backwards compatibility. Just remember to set the shebang in your scripts to <code>#!/bin/zsh</code>.</p>
<p>You will gain some features where <code>zsh</code> is superior to <code>bash</code> v3, such as arrays and associative arrays (dictionaries).</p>
<p>There is one exception where I would now recommend to use <code>/bin/sh</code> for your scripts: the Recovery system does <em>not</em> contain the <code>/bin/zsh</code> shell, even on the Catalina beta. This could still change during the beta phase, or even later, but then you still have to consider <em>older</em> macOS installations where <code>zsh</code> is definitely not present in Recovery.</p>
<p>When you plan to use your scripts or pkgs with installation scripts in a Recovery (or NetInstall, or bootable USB drive) context, such as <a href="https://twocanoes.com/products/mac/mac-deploy-stick/" rel="noopener noreferrer" target="_blank">Twocanoes MDS</a>, <a href="https://github.com/munki/installr" rel="noopener noreferrer" target="_blank">installr</a> or <a href="https://github.com/munki/bootstrappr" rel="noopener noreferrer" target="_blank">bootstrappr</a>, then you <em>cannot</em> rely on <code>/bin/zsh</code>.</p>
<p>Since we now know that <code>bash</code> is eventually going away, the only common choice left is <code>/bin/sh</code>.</p>
<p>When you build an installer package, it can be difficult to anticipate all the contexts in which it might be deployed. So, for installation pre- and postinstall scripts, I would recommend using <code>/bin/sh</code> as the shebang from now on.</p>
<p>I used to recommend using <code>/bin/bash</code> for everything MacAdmin related. <code>/bin/sh</code> is definitely a step down in functionality, but it seems like the safest choice for continued support.</p>
<h2>Summary</h2>
<p>Overall, while the messaging from Apple is very interesting, the change itself is less dramatic than the headlines. Apple is not ‘replacing’ <code>bash</code> with <code>zsh</code>, at least not yet. Overall, we will have to re-think and re-learn a few things, but there is also much to be gained by finally switching from a ten-year-old shell to a new modern one!</p>
<h2>Next</h2>
<p>In the next part we will look at <a href="https://scriptingosx.com/2019/06/moving-to-zsh-part-2-configuration-files/" rel="noopener noreferrer" target="_blank">the configuration files for <code>zsh</code></a>.</p>
<p>
</p><h3><em>Related</em></h3>

content_text:
Apple has announced that in

macOS 10.15 Catalina

the

default shell will be

zsh

.

In this series, I will document my experiences moving

bash

settings, configurations, and scripts over to

zsh

.

zsh

(I believe it is pronounced

zee-shell

, though

zish

is fun to say) will succeed

bash

as the default shell.

bash

has been the default shell since Mac OS X 10.3 Panther.

This series

has grown into a book

: reworked and expanded with more detail and topics.

Like my other books,

I plan to update and add to it after release as well, keeping it relevant and useful. You

can order it on the Apple Books Store now

.

Why?

The

bash

binary bundled with macOS has been stuck on version 3.2 for a long time now.

bash

v4 was released in 2009 and

bash

v5 in January 2019. The reason Apple has not switched to these newer versions is that they are licensed with GPL v3.

bash

v3 is still GPL v2.

zsh

, on the other hand, has an ‘MIT-like’ license, which makes it much more palatable for Apple to include in the system by default.

zsh

has been available as on macOS for a long time. The

zsh

version on macOS 10.14 Mojave is fairly new (5.3). macOS 10.15 Catalina has the current

zsh

5.7.1.

Is bash gone!?

No.

macOS Catalina still has the same

/bin/bash

(version 3.2.57) as Mojave and earlier macOS versions. This change is only for new accounts created on macOS Catalina. When you upgrade to Catalina, a user’s default shell will remain what it was before.

Many scripts in macOS, management systems, and Apple and third party installers rely on

/bin/bash

. If Apple just yanked this binary in macOS 10.15 Catalina or even 10.16. Many installers and other solutions would break and simply cease to function.

Users that have

/bin/bash

as their default shell on Catalina will see a prompt at the start of each Terminal session stating that

zsh

is now the recommended default shell. If you want to continue using

/bin/bash

, you can supress this message by

setting an environment variable in your

.bash_profile

or

.bashrc

.

export BASH_SILENCE_DEPRECATION_WARNING=1

You can also

download and install a newer version of bash

yourself. Keep in mind that custom bash installations reside in a different directory, usually

/usr/local/bin/bash

.

Will bash remain indefinitely?

Apple is strongly messaging that you should switch shells. This is different from the last switch in Mac OS X 10.3 Panther, when Apple switched the default to

bash

, but didn’t really care if you remained on

tcsh

. In fact,

tcsh

is still present on macOS.

Apple’s messaging should tell us, that the days of

/bin/bash

are

numbered. Probably not

very

soon, but eventually keeping a more than ten year old version of

bash

on the system will turn into a liability. The built-in bash had to be patched in 2014 to mitigate the ‘Shellshock’ vulnerability. At some point Apple will consider the cost of continued maintenance too high.

Another clue is that a new shell appeared on macOS Catalina (and is mentioned in the support article). The ‘Debian Almquist Shell’

dash

has been added to the lineup of shells.

dash

is designed to be a minimal implementation of the Posix standard shell

sh

. So far, in macOS (including Catalina),

sh

invokes

bash

in

sh

-compatibility mode.

As

Apple’s support article mentions

, Catalina also adds a new mechanism for users and admins to change which shell handles

sh

invocations. MacAdmins or users can change the symbolic link stored in

/var/select/sh

to point to a shell other than

/bin/bash

. This changes which shell interprets scripts the

#!/bin/sh

shebang or scripts invoked with

sh -c

. Changing the interpreter for

sh

should not, but may change the behavior of several crucial scripts in the system, management tools, and in installers, but may be very useful for testing purposes.

All of these changes are indicators that Apple is preparing to remove

/bin/bash

at some, yet indeterminate, time in the future.

Do I need to wait for Catalina to switch to zsh?

No,

zsh

is available Mojave and on older macOS versions. You can start testing

zsh

or even switch your default shell already.

If you want to just see how

zsh

works, you can just open Terminal and type

zsh

:

$ zsh

MacBook%

The main change you will see is that the prompt looks different.

zsh

uses the

%

character as the default prompt. (You can change that, of course.) Most navigation keystrokes and other behaviors will remain the same as in

bash

.

If you want to already switch your default shell to

zsh

you can use the

chsh

command:

$ chsh -s /bin/zsh

This will prompt for your password. This command will not change the current shell, but all new ones, so close the current Terminal windows and tabs and open a new one.

How is zsh different?

Like

bash

(‘Bourne again shell’ ),

zsh

derives from the ‘Bourne’ family of shells. Because of this common ancestry, it behaves very similar in day-to-day use. The most obvious change will be the different prompt.

The main difference between

bash

and

zsh

is configuration. Since

zsh

ignores the

bash

configuration files (

.bash_profile

or

.bashrc

) you cannot simply copy customized bash settings over to

zsh

.

zsh

has much more options and points to change

zsh

configuration and behavior. There is an entire eco-system of configuration tools and themes called

oh-my-zsh

which is very popular.

zsh

also offers better configuration for auto-completion which is far easier than in

bash

.

I am planning a separate post, describing how to transfer (and translate) your configurations from

bash

to

zsh

.

What about scripting?

Since

zsh

has been present on macOS for a long time, you could start moving your scripts from

bash

to

zsh

right away and not lose backwards compatibility. Just remember to set the shebang in your scripts to

#!/bin/zsh

.

You will gain some features where

zsh

is superior to

bash

v3, such as arrays and associative arrays (dictionaries).

There is one exception where I would now recommend to use

/bin/sh

for your scripts: the Recovery system does

not

contain the

/bin/zsh

shell, even on the Catalina beta. This could still change during the beta phase, or even later, but then you still have to consider

older

macOS installations where

zsh

is definitely not present in Recovery.

When you plan to use your scripts or pkgs with installation scripts in a Recovery (or NetInstall, or bootable USB drive) context, such as

Twocanoes MDS

,

installr

or

bootstrappr

, then you

cannot

rely on

/bin/zsh

.

Since we now know that

bash

is eventually going away, the only common choice left is

/bin/sh

.

When you build an installer package, it can be difficult to anticipate all the contexts in which it might be deployed. So, for installation pre- and postinstall scripts, I would recommend using

/bin/sh

as the shebang from now on.

I used to recommend using

/bin/bash

for everything MacAdmin related.

/bin/sh

is definitely a step down in functionality, but it seems like the safest choice for continued support.

Summary

Overall, while the messaging from Apple is very interesting, the change itself is less dramatic than the headlines. Apple is not ‘replacing’

bash

with

zsh

, at least not yet. Overall, we will have to re-think and re-learn a few things, but there is also much to be gained by finally switching from a ten-year-old shell to a new modern one!

Next

In the next part we will look at

the configuration files for

zsh

.

Related


url: https://www.xda-developers.com/favorite-mcp-server-use-local-llm/
title: This is my favorite MCP server to use with my local LLM
added_at: 2025-10-19 04:42:00
content_html:
<p><a href="https://www.xda-developers.com/mcp-accidentally-became-best-common-language-services-talk/" rel="noopener noreferrer" target="_blank">Model Context Protocol</a> has been an incrasingly more important piece of technology in the large language model (LLM) space, and with good reason. MCP essentially acts as a translator between a language model and the digital world, abstracting data from a particular service and presenting it in a way that a language model can access as a "tool" or platform. What this means is that, instead of manually pasting snippets from your browser, music player, or notes app, a model can choose to collect that information if required. There's one MCP server that I've been using the most, though, and that's <a href="https://www.xda-developers.com/how-hosted-own-search-engine-nas/" rel="noopener noreferrer" target="_blank">SearXNG</a>.</p> <p>SearXNG is a self-hosted metasearch search engine, meaning that it combines the results of multiple search engines in order to surface the results to you in a particular order. You control what search engines that it uses, what data it looks for, and you can even install and set up plugins that will filter or <a href="https://github.com/joestump/searxng-privacy-plugins" rel="noopener noreferrer nofollow" target="_blank">modify URLs that were collected</a> to go through a privacy-preserving proxy. However, there's also a fantastic MCP server that can present itself as a search tool to your local LLM, giving it one of the most powerful tools normally only available to cloud-based alternatives: search.</p> <h2>
                        Why is search so important for a local LLM?
               </h2><h3>
            It bypasses a few problems
    </h3>


<figure><picture><source srcset="https://static0.xdaimages.com/wordpress/wp-content/uploads/wm/2025/10/searchtool-finding-punk-rock-album-october-2025.png?q=49&amp;fit=crop&amp;w=500&amp;dpr=2"/>
<source srcset="https://static0.xdaimages.com/wordpress/wp-content/uploads/wm/2025/10/searchtool-finding-punk-rock-album-october-2025.png?q=49&amp;fit=crop&amp;w=800&amp;dpr=2"/>
<source srcset="https://static0.xdaimages.com/wordpress/wp-content/uploads/wm/2025/10/searchtool-finding-punk-rock-album-october-2025.png?q=49&amp;fit=crop&amp;w=825&amp;dpr=2"/>
<img alt="Using a SearXNG MCP server to find a future album release with a local LLM" loading="lazy" referrerpolicy="no-referrer" src="https://static0.xdaimages.com/wordpress/wp-content/uploads/wm/2025/10/searchtool-finding-punk-rock-album-october-2025.png?q=49&amp;fit=crop&amp;w=825&amp;dpr=2"/>
</picture><figcaption> 
<p></p>
</figcaption>Credit:  
<p></p>
</figure>


<p>Search is one of the most important tools of <em>any</em> language model these days, but particularly so in the case of a local language model. When a model is trained, it uses a "knowledge cut-off," meaning that the corpus of data that it used to build its knowledge only goes to a certain point. If an album from your favorite artist just came out, and you want to ask about it, a model without search capabilities will not even know that it exists and will either give a bogus answer, or say that it has no knowledge about it. Even the biggest and most powerful language models suffer from these cut-off dates, with GPT-5's cut-off being in October 2024 and Gemini 2.5 Pro's cut-off being in January 2025. That's why both of these models can search the web to fill those knowledge gaps, even if it requires you to tell them to search the web sometimes.</p> <p>A local language model, by default, doesn't have that capability. Not only that, given their small size comparatively, they often lack a lot of information that might normally be in the training data of significantly larger models, so even asking about an album from your favorite artist (that may have been released a few years ago) could still result in completely incorrect information. This problem becomes even more apparent when using a language model in a rapidly changing or growing space, such as development. Best practices in the language of your choosing could have evolved significantly since the training data of your chosen model was compiled, and it could provide incorrect or out of date guidance as a result.</p> <p>As a result, search is one of the most powerful tools you can give your local language model. Not only does it allow you to find information that's <em>current</em>, but it can also fill in the gaps of what the language model does and <em>doesn't</em> know. The only thing you need to be aware of is context length: essentially, the search tool will supply the chosen page as text for the LLM to parse and pull the information out of. This uses up a lot of tokens, and a search request can burn a couple of thousand tokens in order to collect all of the information required to answer your query. The two queries above used up 6,825 tokens, which is above the 4,096 default token length that LM Studio chooses. You can increase the context window yourself, but it's something that you'll need to keep in mind.</p> <p>It's important to stress that search capabilities are not a foolproof way to prevent hallucinations or incorrect information being provided. You should still always sanity check the responses that you get, but it's a quick and easy way to massively improve the quality of your responses.</p> <h2>
                        Setting up SearXNG MCP
               </h2><h3>
            It's really quick and easy
    </h3>
<p>First and foremost, you'll need to either find a SearXNG server that will allow you to use it as an MCP server, or you can host your own. It's not an intensive application at all, so you could host it using Docker on the same PC that you're running your local LLM on. I'm running it in a Proxmox LXC, but you'll need to make sure that SearXNG's JSON output format is enabled. This is in SearXNG's settings.yml file, and all you need to do is add the "json" line and restart the container. You can subvert all of that by just using a public instance of SearXNG instead, though this mitigates some of the privacy benefits of self-hosting and also requires finding an instance that has JSON enabled.</p> <p>Once you have it set up and ready to go, you'll need to set up the tool in LM Studio, Ollama, or whatever provider you use to run your language models. In LM Studio, you can add the following to mcp.json, while ensuring that you use proper indentation:</p> <p></p><pre>{<br/> "mcpServers": {<br/> "searxng": {<br/> "command": "npx",<br/> "args": [<br/> "-y",<br/> "mcp-searxng"<br/> ],<br/> "env": {<br/> "SEARXNG_URL": "http://192.168.2.109:8888"<br/> }<br/> }<br/> }<br/>}<br/></pre> <p>That's literally all you need! Once you've got it added, you can enable the SearXNG MCP server in the sidebar of LM Studio, or enable it on a per-conversation basis. From there, language models that support tool calling can invoke it and use it to find up-to-date information pulled from the web, bypassing the knowledge cut-off and ensuring that the information you get is valid and up to date. If you suspect that something went wrong, you can also click the tool call in the conversation in order to see where exactly the information was pulled from and <em>what</em> it pulled.</p> <p>SearXNG is incredibly lightweight, to the point that it's honestly a waste <em>not</em> to pair it with your local LLM. It unlocks a lot more possibilities, and paired with even more tools, the privacy benefits alone <a href="https://www.xda-developers.com/mcp-servers-changed-local-llm-better-than-cloud/" rel="noopener noreferrer" target="_blank">make it better than any cloud-based provider for me</a>.</p> <p></p>

content_text:
Model Context Protocol

has been an incrasingly more important piece of technology in the large language model (LLM) space, and with good reason. MCP essentially acts as a translator between a language model and the digital world, abstracting data from a particular service and presenting it in a way that a language model can access as a "tool" or platform. What this means is that, instead of manually pasting snippets from your browser, music player, or notes app, a model can choose to collect that information if required. There's one MCP server that I've been using the most, though, and that's

SearXNG

.

SearXNG is a self-hosted metasearch search engine, meaning that it combines the results of multiple search engines in order to surface the results to you in a particular order. You control what search engines that it uses, what data it looks for, and you can even install and set up plugins that will filter or

modify URLs that were collected

to go through a privacy-preserving proxy. However, there's also a fantastic MCP server that can present itself as a search tool to your local LLM, giving it one of the most powerful tools normally only available to cloud-based alternatives: search.

Why is search so important for a local LLM?

It bypasses a few problems

Credit:

Search is one of the most important tools of

any

language model these days, but particularly so in the case of a local language model. When a model is trained, it uses a "knowledge cut-off," meaning that the corpus of data that it used to build its knowledge only goes to a certain point. If an album from your favorite artist just came out, and you want to ask about it, a model without search capabilities will not even know that it exists and will either give a bogus answer, or say that it has no knowledge about it. Even the biggest and most powerful language models suffer from these cut-off dates, with GPT-5's cut-off being in October 2024 and Gemini 2.5 Pro's cut-off being in January 2025. That's why both of these models can search the web to fill those knowledge gaps, even if it requires you to tell them to search the web sometimes.

A local language model, by default, doesn't have that capability. Not only that, given their small size comparatively, they often lack a lot of information that might normally be in the training data of significantly larger models, so even asking about an album from your favorite artist (that may have been released a few years ago) could still result in completely incorrect information. This problem becomes even more apparent when using a language model in a rapidly changing or growing space, such as development. Best practices in the language of your choosing could have evolved significantly since the training data of your chosen model was compiled, and it could provide incorrect or out of date guidance as a result.

As a result, search is one of the most powerful tools you can give your local language model. Not only does it allow you to find information that's

current

, but it can also fill in the gaps of what the language model does and

doesn't

know. The only thing you need to be aware of is context length: essentially, the search tool will supply the chosen page as text for the LLM to parse and pull the information out of. This uses up a lot of tokens, and a search request can burn a couple of thousand tokens in order to collect all of the information required to answer your query. The two queries above used up 6,825 tokens, which is above the 4,096 default token length that LM Studio chooses. You can increase the context window yourself, but it's something that you'll need to keep in mind.

It's important to stress that search capabilities are not a foolproof way to prevent hallucinations or incorrect information being provided. You should still always sanity check the responses that you get, but it's a quick and easy way to massively improve the quality of your responses.

Setting up SearXNG MCP

It's really quick and easy

First and foremost, you'll need to either find a SearXNG server that will allow you to use it as an MCP server, or you can host your own. It's not an intensive application at all, so you could host it using Docker on the same PC that you're running your local LLM on. I'm running it in a Proxmox LXC, but you'll need to make sure that SearXNG's JSON output format is enabled. This is in SearXNG's settings.yml file, and all you need to do is add the "json" line and restart the container. You can subvert all of that by just using a public instance of SearXNG instead, though this mitigates some of the privacy benefits of self-hosting and also requires finding an instance that has JSON enabled.

Once you have it set up and ready to go, you'll need to set up the tool in LM Studio, Ollama, or whatever provider you use to run your language models. In LM Studio, you can add the following to mcp.json, while ensuring that you use proper indentation:

{

"mcpServers": {

"searxng": {

"command": "npx",

"args": [

"-y",

"mcp-searxng"

],

"env": {

"SEARXNG_URL": "http://192.168.2.109:8888"

}

}

}

}

That's literally all you need! Once you've got it added, you can enable the SearXNG MCP server in the sidebar of LM Studio, or enable it on a per-conversation basis. From there, language models that support tool calling can invoke it and use it to find up-to-date information pulled from the web, bypassing the knowledge cut-off and ensuring that the information you get is valid and up to date. If you suspect that something went wrong, you can also click the tool call in the conversation in order to see where exactly the information was pulled from and

what

it pulled.

SearXNG is incredibly lightweight, to the point that it's honestly a waste

not

to pair it with your local LLM. It unlocks a lot more possibilities, and paired with even more tools, the privacy benefits alone

make it better than any cloud-based provider for me

.

thumb_url:
https://static0.xdaimages.com/wordpress/wp-content/uploads/wm/2025/10/searchtool-finding-punk-rock-album-october-2025.png?q=49&fit=crop&w=825&dpr=2


url: https://www.xda-developers.com/my-entire-smart-home-runs-from-mini-pc-with-home-assistant/
title: I run my entire smart home from a single mini PC with Home Assistant
added_at: 2025-10-21 08:19:34
content_html:
<p>I've been dreaming of a <a href="https://www.xda-developers.com/built-my-own-home-automation-network-without-using-smart-home-hub/" rel="noopener noreferrer" target="_blank">unified smart home</a> since I first got a Philips Hue starter kit years ago. Then I set up the kit, and the reality of the siloed smart home sank in. My lights needed a hub to work, <em>everything</em> needed the cloud, and I could only connect the hub to a single voice assistant ecosystem at a time, severely limiting its usefulness. If you had told me all those years ago that I'd be <a href="https://www.xda-developers.com/built-my-own-home-automation-network-without-using-smart-home-hub/" rel="noopener noreferrer" target="_blank">running my smart home from a single dashboard</a> <a href="https://www.xda-developers.com/turned-old-mini-pc-into-home-assistant-hub-proxmox/" rel="noopener noreferrer" target="_blank">running on a mini PC</a>, I probably would have laughed in your face.</p> <p>It was inconceivable back then that a home user could self-host a smart home hub, build it with the integrations, sensors, and protocols they wanted, and have it all designed to be local-first. Back then, the cloud was the hot thing, and companies wanted to reduce the complexity for the user by abstracting away all the control. But that came with a compromise on data security and privacy.</p> <p>The other thing it enabled was a fractured smart home environment. There wasn't (and still isn't) a single company that carries a full range of smart home technology, so there's a massive market for companies that make their own dashboard and do the hard work for you. But     <a href="https://www.xda-developers.com/minimal-home-assistant-automations-maximum-impact-smart-home/" rel="noopener noreferrer" target="_blank">with Home Assistant</a> I've made my own dashboard with a little hard work and perseverance, now running my smart home from a single mini PC. And the thing is, it's not a huge stretch for anyone else to do the same.    </p> <h2>
                        Home Assistant is the smart home platform I wanted all along
               </h2><h3>
            It might be a little rough around the edges, but it's mine
    </h3>
<p>The unified smart home has long been a feature in sci-fi and popular culture, whether it's the computer in <em>Star Trek,</em> along with replicators to make a post-scarcity culture, to <em>The Jetsons</em> and the ever-dependable housekeeper, Rosie. I grew up hoping my home would be like that one day, and as the necessary technologies trickled onto the market, I became more and more excited.</p> <p>But that excitement soon turned to disillusionment, as every smart home ecosystem seemed designed to be apart from the others, with only basic functions being able to pass the barriers. Protocols like <a href="https://www.xda-developers.com/matter-doubting-future-of-smart-homes/" rel="noopener noreferrer" target="_blank">Matter were supposed to fix that</a>, but it's been years since the launch, and the dream of the manufacturer-supplied unified smart home is still out of reach.</p> <p>Home Assistant is the closest approximation of the future I was promised, and it <a href="https://www.xda-developers.com/my-smart-home-is-instantly-smarter-when-i-leave/" rel="noopener noreferrer" target="_blank">makes my smart home better</a>, whether I'm in it or not. At first, I wasn't sure if it would be my solution, so <a href="https://www.xda-developers.com/reasons-host-home-assistant-in-docker/" rel="noopener noreferrer" target="_blank">I had Home Assistant self-hosted in a Docker container</a> to try it out. I soon ran into the limitations of doing it that way, mainly that the add-on store wasn't available, and moved my setup to a virtual machine inside a Proxmox host on a mini PC. And it's been nothing short of fantastic so far.</p> <h3>
            Templates for success
    </h3>
<p>Okay, templates are called blueprints in Home Assistant, but the point is that the community is vast and talented, and the chances are that if I buy a new smart device, there's already a host of handy tools to use. <a href="https://www.xda-developers.com/actionable-notifications-home-assistant-smart-home/" rel="noopener noreferrer" target="_blank">You can add advanced functionality like actionable notifications</a>, so you get one-tap actions on the toast notification without opening the app. You can also set up <a href="https://www.xda-developers.com/home-assistant-blueprints-that-make-guests-think-my-house-is-magic/" rel="noopener noreferrer" target="_blank">calendar-based notifications and alerts</a>, or thousands (probably) of <a href="https://community.home-assistant.io/c/blueprints-exchange/53" rel="noopener noreferrer" target="_blank">other blueprints created by the community</a>, and that's one of the first places I search when I add a new device.</p> <h3>
            It enabled me to drop the apps of so many devices
    </h3>
<p>It seemed that every time I added a new smart home device to my house, I needed another app to control them. Apps for light bulbs, apps for vacuum cleaners, apps for appliances, and more. They all cluttered up my smartphone's home screen, and it was miserable to use. For the most part, I sidestepped the issue by linking everything I could to Alexa or Home Assistant, but the integrations were never quite as powerful as the app, and I could never uninstall them.</p> <p>With a few notable exceptions, every smart home device I own <a href="https://www.xda-developers.com/replaced-all-smart-home-apps-with-home-assistant/" rel="noopener noreferrer" target="_blank">is now connected to Home Assistant</a>, and I rarely need the individual apps. What's more, I can set up automations between those devices in ways I could never have dreamed of before. Sure, <a href="https://www.xda-developers.com/home-assistant-automations-without-extra-hardware/" rel="noopener noreferrer" target="_blank">I don't use that many automations yet</a>, but being able to create and control them from a single tab in Home Assistant makes ongoing management simple.</p>
<h2>
                        Home Assistant doesn't need a powerful computer to run
               </h2><h3>
            And I've got plenty of other virtual machines running on the same box
    </h3>
<p>For how powerful the operating system is, Home Assistant doesn't need much processing power to run. It's perfectly happy on a Raspberry Pi or other SBC, but I wanted a little bit more power and reliability, which <a href="https://www.xda-developers.com/all-flash-mini-pc-has-replaced-my-nas-permanently/" rel="noopener noreferrer" target="_blank">led me to use the Beelink ME mini</a> as the host device for Home Assistant. This has an Intel N150 CPU, two 2.5GbE NICs, a handful of USB ports for adding a Zigbee or Z-Wave controller, and a low power consumption of 6W. That's less than most of my light bulbs, so I can keep it running 24/7 without worrying about my power bill.</p> <p>It also runs Proxmox as the hypervisor because it's lightweight and is something that I'm familiar with using. At some point, I plan on getting two more and setting up a <a href="https://www.xda-developers.com/high-availability-proxmox-cluster-at-home-useful/" rel="noopener noreferrer" target="_blank">High Availability Proxmox cluster</a>, so that my smart home control will survive hardware failure. </p> <h3>
            Tons of room for activities
    </h3>
<p>The six M.2 slots for NVMe storage on this mini PC is overkill for Home Assistant, which doesn't use much storage. But because I'm using Proxmox, I also run other services and operating systems on this box. Most of the installed SSDs are passed through to TrueNAS for speedy storage, but I've also got a few Linux distros and other assorted things on there. I have to be careful not to overload the CPU and the 12GB of RAM, but other than that, it's powerful enough for Home Assistant and some of my home lab experiments. </p>
<h3>
            A mini PC running Home Assistant is the home hub my smart home deserves
    </h3>
<p>I haven't finished building my smart home to my satisfaction, so while the mini PC running Home Assistant will stay, it will get some additions. My <a href="https://www.xda-developers.com/what-are-zigbee-networks-and-should-you-use-them-in-your-smart-home/" rel="noopener noreferrer" target="_blank">Zigbee devices</a> are connected to an Eero mesh network, but I'm planning on replacing those, so I'll need a Zigbee coordinator. I might change to Z-Wave because I'm not fully entrenched in any one ecosystem yet, but I need to see which devices I'll need and what will meet the needs of my household.</p> <p>But instead of picking any old devices and dealing with another app on my phone, I now look for Home Assistant support, and more importantly, access to the API. With API access, I can build an entirely local smart home with only one app to keep installed, Home Assistant. And with it running on a mini PC, I know it has enough processing power to handle any number of devices I might add. </p> 

content_text:
I've been dreaming of a

unified smart home

since I first got a Philips Hue starter kit years ago. Then I set up the kit, and the reality of the siloed smart home sank in. My lights needed a hub to work,

everything

needed the cloud, and I could only connect the hub to a single voice assistant ecosystem at a time, severely limiting its usefulness. If you had told me all those years ago that I'd be

running my smart home from a single dashboard

running on a mini PC

, I probably would have laughed in your face.

It was inconceivable back then that a home user could self-host a smart home hub, build it with the integrations, sensors, and protocols they wanted, and have it all designed to be local-first. Back then, the cloud was the hot thing, and companies wanted to reduce the complexity for the user by abstracting away all the control. But that came with a compromise on data security and privacy.

The other thing it enabled was a fractured smart home environment. There wasn't (and still isn't) a single company that carries a full range of smart home technology, so there's a massive market for companies that make their own dashboard and do the hard work for you. But

with Home Assistant

I've made my own dashboard with a little hard work and perseverance, now running my smart home from a single mini PC. And the thing is, it's not a huge stretch for anyone else to do the same.

Home Assistant is the smart home platform I wanted all along

It might be a little rough around the edges, but it's mine

The unified smart home has long been a feature in sci-fi and popular culture, whether it's the computer in

Star Trek,

along with replicators to make a post-scarcity culture, to

The Jetsons

and the ever-dependable housekeeper, Rosie. I grew up hoping my home would be like that one day, and as the necessary technologies trickled onto the market, I became more and more excited.

But that excitement soon turned to disillusionment, as every smart home ecosystem seemed designed to be apart from the others, with only basic functions being able to pass the barriers. Protocols like

Matter were supposed to fix that

, but it's been years since the launch, and the dream of the manufacturer-supplied unified smart home is still out of reach.

Home Assistant is the closest approximation of the future I was promised, and it

makes my smart home better

, whether I'm in it or not. At first, I wasn't sure if it would be my solution, so

I had Home Assistant self-hosted in a Docker container

to try it out. I soon ran into the limitations of doing it that way, mainly that the add-on store wasn't available, and moved my setup to a virtual machine inside a Proxmox host on a mini PC. And it's been nothing short of fantastic so far.

Templates for success

Okay, templates are called blueprints in Home Assistant, but the point is that the community is vast and talented, and the chances are that if I buy a new smart device, there's already a host of handy tools to use.

You can add advanced functionality like actionable notifications

, so you get one-tap actions on the toast notification without opening the app. You can also set up

calendar-based notifications and alerts

, or thousands (probably) of

other blueprints created by the community

, and that's one of the first places I search when I add a new device.

It enabled me to drop the apps of so many devices

It seemed that every time I added a new smart home device to my house, I needed another app to control them. Apps for light bulbs, apps for vacuum cleaners, apps for appliances, and more. They all cluttered up my smartphone's home screen, and it was miserable to use. For the most part, I sidestepped the issue by linking everything I could to Alexa or Home Assistant, but the integrations were never quite as powerful as the app, and I could never uninstall them.

With a few notable exceptions, every smart home device I own

is now connected to Home Assistant

, and I rarely need the individual apps. What's more, I can set up automations between those devices in ways I could never have dreamed of before. Sure,

I don't use that many automations yet

, but being able to create and control them from a single tab in Home Assistant makes ongoing management simple.

Home Assistant doesn't need a powerful computer to run

And I've got plenty of other virtual machines running on the same box

For how powerful the operating system is, Home Assistant doesn't need much processing power to run. It's perfectly happy on a Raspberry Pi or other SBC, but I wanted a little bit more power and reliability, which

led me to use the Beelink ME mini

as the host device for Home Assistant. This has an Intel N150 CPU, two 2.5GbE NICs, a handful of USB ports for adding a Zigbee or Z-Wave controller, and a low power consumption of 6W. That's less than most of my light bulbs, so I can keep it running 24/7 without worrying about my power bill.

It also runs Proxmox as the hypervisor because it's lightweight and is something that I'm familiar with using. At some point, I plan on getting two more and setting up a

High Availability Proxmox cluster

, so that my smart home control will survive hardware failure.

Tons of room for activities

The six M.2 slots for NVMe storage on this mini PC is overkill for Home Assistant, which doesn't use much storage. But because I'm using Proxmox, I also run other services and operating systems on this box. Most of the installed SSDs are passed through to TrueNAS for speedy storage, but I've also got a few Linux distros and other assorted things on there. I have to be careful not to overload the CPU and the 12GB of RAM, but other than that, it's powerful enough for Home Assistant and some of my home lab experiments.

A mini PC running Home Assistant is the home hub my smart home deserves

I haven't finished building my smart home to my satisfaction, so while the mini PC running Home Assistant will stay, it will get some additions. My

Zigbee devices

are connected to an Eero mesh network, but I'm planning on replacing those, so I'll need a Zigbee coordinator. I might change to Z-Wave because I'm not fully entrenched in any one ecosystem yet, but I need to see which devices I'll need and what will meet the needs of my household.

But instead of picking any old devices and dealing with another app on my phone, I now look for Home Assistant support, and more importantly, access to the API. With API access, I can build an entirely local smart home with only one app to keep installed, Home Assistant. And with it running on a mini PC, I know it has enough processing power to handle any number of devices I might add.

thumb_url:
https://static0.xdaimages.com/wordpress/wp-content/uploads/wm/2025/06/beelink-me-mini-nas-desk.JPEG?w=1600&h=900&fit=crop


added_at: 2025-10-26 05:31:21
url:
https://machinelearningmastery.com/the-machine-learning-practitioners-guide-to-fine-tuning-language-models/

title:
The Machine Learning Practitioner's Guide to Fine-Tuning Language Models - MachineLearningMastery.com

content_html:
<section>
<p>In this article, you will learn when fine-tuning large language models is warranted, which 2025-ready methods and tools to choose, and how to avoid the most common mistakes that derail projects.</p>
<p>Topics we will cover include:</p>
<ul>
<li>A practical decision framework: prompt engineering, retrieval-augmented generation (RAG), and when fine-tuning truly adds value.</li>
<li>Today’s essential methods—LoRA/QLoRA, Spectrum—and alignment with DPO, plus when to pick each.</li>
<li>Data preparation, evaluation, and proven configurations that keep you out of trouble.</li>
</ul>
<p>Let’s not waste any more time.</p>
<img alt="Machine Learning Practitioners Guide Fine-Tuning Language Models" loading="lazy" referrerpolicy="no-referrer" src="https://machinelearningmastery.com/wp-content/uploads/2025/10/mlm-chugani-machine-learning-practitioners-guide-fine-tuning-language-models-feature-1024x683.png"/><p>The Machine Learning Practitioner’s Guide to Fine-Tuning Language Models<br/>
Image by Author<br/></p>
<h2>Introduction</h2>
<p>Fine-tuning has become much more accessible in 2024–2025, with parameter-efficient methods letting even 70B+ parameter models run on consumer GPUs. But should you fine-tune at all? And if so, how do you choose between the dozens of emerging techniques?</p>
<p>This guide is for practitioners who want results, not just theory. You’ll learn when fine-tuning makes sense, which methods to use, and how to avoid common pitfalls.</p>
<p>Fine-tuning is different from traditional machine learning. Instead of training models from scratch, you’re adapting pretrained models to specialized tasks using far less data and compute. This makes sophisticated natural language processing (NLP) capabilities accessible without billion-dollar budgets.</p>
<p>For machine learning practitioners, this builds on skills you already have. Data preparation, evaluation frameworks, and hyperparameter tuning remain central. You’ll need to learn new architectural patterns and efficiency techniques, but your existing foundation gives you a major advantage.</p>
<p>You’ll learn:</p>
<ul>
<li>When fine-tuning provides value versus simpler alternatives like prompt engineering or retrieval-augmented generation (RAG)</li>
<li>The core parameter-efficient methods (LoRA, QLoRA, Spectrum) and when to use each</li>
<li>Modern alignment techniques (DPO, RLHF) that make models follow instructions reliably</li>
<li>Data preparation strategies that determine most of your fine-tuning success</li>
<li>Critical pitfalls in overfitting and catastrophic forgetting, and how to avoid them</li>
</ul>
<p>If you’re already working with LLMs, you have what you need. If you need a refresher, check out our guides on <a href="https://machinelearningmastery.com/?s=prompt+engineering" rel="noopener" target="_blank"><strong>prompt engineering</strong></a> and <a href="https://machinelearningmastery.com/?s=LLM+applications" rel="noopener" target="_blank"><strong>LLM applications</strong></a>.</p>
<p>Before getting into fine-tuning mechanics, you need to understand whether fine-tuning is the right approach.</p>
<h2>When to Fine-Tune Versus Alternative Approaches</h2>
<p><strong>Fine-tuning should be your last resort, not your first choice.</strong> The recommended progression starts with prompt engineering, escalates to RAG when external knowledge is needed, and only proceeds to fine-tuning when deep specialization is required.</p>
<p><a href="https://cloud.google.com/blog/products/ai-machine-learning/to-tune-or-not-to-tune-a-guide-to-leveraging-your-data-with-llms" rel="noopener" target="_blank"><strong>Google Cloud’s decision framework</strong></a> and <a href="https://ai.meta.com/blog/when-to-fine-tune-llms-vs-other-techniques/" rel="noopener" target="_blank"><strong>Meta AI’s practical guide</strong></a> identify clear criteria: Use prompt engineering for basic task adaptation. Use RAG when you need source citations, must ground responses in documents, or information changes frequently. Meta AI reveals five scenarios where fine-tuning provides genuine value: customizing tone and style for specific audiences, maintaining data privacy for sensitive information, supporting low-resource languages, reducing inference costs by distilling larger models, and adding entirely new capabilities not present in base models.</p>
<p><strong>The data availability test</strong>: With fewer than 100 examples, stick to prompt engineering. With 100–1,000 examples and static knowledge, consider parameter-efficient methods. Only with 1,000–100,000 examples and a clear task definition should you attempt fine-tuning.</p>
<p>For news summarization or general question answering, RAG excels. For customer support requiring a specific brand voice or code generation following particular patterns, fine-tuning proves essential. The optimal solution often combines both—fine-tune for specialized reasoning patterns while using RAG for current information.</p>
<h2>Essential Parameter-Efficient Fine-Tuning Methods</h2>
<p>Full fine-tuning updates all model parameters, requiring massive compute and memory. Parameter-efficient fine-tuning (PEFT) revolutionized this by enabling training with just ~0.1% to 3% of parameters updated, achieving comparable performance while dramatically reducing requirements.</p>
<p><strong>LoRA (Low-Rank Adaptation) emerged as the dominant technique.</strong> LoRA freezes pretrained weights and injects trainable rank-decomposition matrices in parallel. Instead of updating entire weight matrices, LoRA represents updates as low-rank decompositions. Weight updates during adaptation often have low intrinsic rank, with rank 8 typically sufficient for many tasks.</p>
<p>Memory reductions reach 2× to 3× versus full fine-tuning, with checkpoint sizes decreasing 1,000× to 10,000×. A 350 GB model can require only a ~35 MB adapter file. Training can be ~25% faster on large models. Critically, learned matrices merge with frozen weights during deployment, introducing zero inference latency.</p>
<p><strong>QLoRA extends LoRA through aggressive quantization</strong> while maintaining accuracy. Base weights are stored in 4-bit format with computation happening in 16-bit <em>bfloat16</em>. The results can be dramatic: 65B models on 48 GB GPUs, 33B on 24 GB, 13B on consumer 16 GB hardware—while matching many 16-bit full fine-tuning results.</p>
<p><strong><a href="https://huggingface.co/papers/2406.06623" rel="noopener" target="_blank">Spectrum</a></strong>, a 2024 innovation, takes a different approach. Rather than adding adapters, Spectrum identifies the most informative layers using signal-to-noise ratio analysis and selectively fine-tunes only the top ~30%. Reports show higher accuracy than QLoRA on mathematical reasoning with comparable resources.</p>
<p><strong>Decision framework</strong>: Use LoRA when you need zero inference latency and moderate GPU resources (16–24 GB). Use QLoRA for extreme memory constraints (consumer GPUs, Google Colab) or very large models (30B+). Use Spectrum when working with large models in distributed settings.</p>
<p><strong>Ready to implement LoRA and QLoRA?</strong> <a href="https://www.philschmid.de/fine-tune-llms-in-2025" rel="noopener" target="_blank"><strong>How to fine-tune open LLMs in 2025</strong></a> by Phil Schmid provides complete code examples with current best practices. For hands-on practice, try <a href="https://docs.unsloth.ai/" rel="noopener" target="_blank"><strong>Unsloth’s free Colab notebooks</strong></a>.</p>
<h2>Modern Alignment and Instruction Tuning</h2>
<p>Instruction tuning transforms completion-focused base models into instruction-following assistants, establishing basic capabilities before alignment. The method trains on diverse instruction-response pairs covering question answering, summarization, translation, and reasoning. Quality matters far more than quantity, with ~1,000 high-quality examples often sufficient.</p>
<p><strong>Direct Preference Optimization (DPO) has rapidly become the preferred alignment method</strong> by dramatically simplifying reinforcement learning from human feedback (RLHF). The key idea: re-parameterize the reward as implicit in the policy itself, solving the RLHF objective through supervised learning rather than complex reinforcement learning.</p>
<p>Research from Stanford and others reports that DPO can achieve comparable or superior performance to PPO-based RLHF with single-stage training, ~50% less compute, and greater stability. DPO requires only preference data (prompt, chosen response, rejected response), a reference policy, and standard supervised learning infrastructure. The method has become common for training open-source LLMs in 2024–2025, including Zephyr-7B and various Mistral-based models.</p>
<p>RLHF remains the foundational alignment technique but brings high complexity: managing four model copies during training (policy, reference, reward, value), difficult implementations, and training instability. OpenAI’s InstructGPT demonstrated that a 1.3B aligned model could outperform a 175B base model on human evaluations, underscoring alignment’s power. However, most practitioners should use DPO unless specific scenarios demand RLHF’s flexibility.</p>
<p><strong>Start with instruction tuning using datasets like Alpaca or Dolly-15k, then implement DPO for alignment</strong> rather than attempting RLHF. <a href="https://huggingface.ai/docs/trl" rel="noopener" target="_blank"><strong>TRL (Transformer Reinforcement Learning) documentation</strong></a> provides comprehensive guides for both DPO and RLHF with working code examples. For conceptual understanding, see Chip Huyen’s <a href="https://huyenchip.com/2023/05/02/rlhf.html" rel="noopener" target="_blank"><strong>RLHF: Reinforcement Learning from Human Feedback</strong></a>.</p>
<h2>Data Preparation Best Practices</h2>
<p><strong>Data quality determines fine-tuning success more than any other factor.</strong> As error rates in training data increase linearly, downstream model error can rise superlinearly—making data curation your highest-leverage activity.</p>
<p>Dataset size requirements vary by task complexity. Simple classification needs ~200 to 1,000 examples. Medium-complexity tasks like question answering require ~1,000 to 5,000. Complex generation or reasoning can demand 5,000 to 10,000+. Quality trumps quantity: 1,000 high-quality examples can outperform 100,000 mediocre ones.</p>
<p>High-quality data exhibits five characteristics: domain relevance, diversity across scenarios, representativeness of the full distribution, labeling accuracy, and freshness for time-sensitive domains.</p>
<p><strong>Formatting impacts results significantly.</strong> Use structured question-answer pairs with consistent formatting across datasets to prevent spurious pattern learning. Standard splits allocate ~80% training and ~20% validation using stratified sampling when applicable.</p>
<p>Essential preprocessing: clean noise, handle missing values, use model-specific tokenizers, remove duplicates, and normalize text. Favor proprietary custom data over public datasets that models may have already encountered during pretraining.</p>
<p><strong>Need help with data preparation?</strong> Meta AI’s guide <a href="https://ai.meta.com/blog/how-to-fine-tune-llms-peft-dataset-curation/" rel="noopener" target="_blank"><strong>How to fine-tune: Focus on effective datasets</strong></a> emphasizes proprietary data strategies and provides practical curation techniques. For dataset exploration, browse <a href="https://huggingface.co/datasets" rel="noopener" target="_blank"><strong>Hugging Face Datasets</strong></a> to see quality examples.</p>
<h2>Avoiding Critical Pitfalls</h2>
<p><strong>Overfitting occurs when models memorize training data instead of learning generalizable patterns.</strong> It’s the most common fine-tuning failure. Signs include training loss decreasing while validation loss increases, high training accuracy but poor validation performance, and loss approaching zero.</p>
<p>Prevention requires multiple strategies. Early stopping halts training when validation performance plateaus. Regularization includes L2 weight decay, 10%–30% dropout, and weight penalties. Data augmentation increases diversity through back-translation and synthetic generation. K-fold cross-validation helps ensure generalization across splits.</p>
<p>For parameter-efficient fine-tuning (PEFT) methods, reduce LoRA rank (<code>r</code> parameter) and alpha values to decrease trainable parameters. Use learning rates of 1e-4 to 2e-4 for fine-tuning. Monitor both training and validation losses continuously. PEFT methods like LoRA naturally reduce overfitting by limiting trainable parameters to ~0.1%–1%.</p>
<p><strong>Catastrophic forgetting poses a more insidious challenge:</strong> loss of previously learned information when training on new tasks. Models can lose general reasoning abilities, decline on previously answerable questions, and overfit to specific output formats. Forgetting can begin early in fine-tuning through format specialization.</p>
<p>Prevention strategies include elastic weight consolidation (EWC), which identifies and protects important weights; “half fine-tuning,” which freezes roughly half of parameters during each round; and sharpness-aware minimization (SAM), which flattens the loss landscape. Most accessible: include diverse instruction datasets mixed with domain-specific data.</p>
<p><strong>Struggling with overfitting or catastrophic forgetting?</strong> The paper <a href="https://aclanthology.org/2024.findings-emnlp.249/" rel="noopener" target="_blank"><strong>Revisiting Catastrophic Forgetting in Large Language Model Tuning</strong></a> provides practical mitigation strategies with empirical evidence. For monitoring and debugging, use <a href="https://wandb.ai/" rel="noopener" target="_blank"><strong>Weights &amp; Biases</strong></a> or <a href="https://www.tensorflow.org/tensorboard" rel="noopener" target="_blank"><strong>TensorBoard</strong></a> to track training and validation metrics continuously.</p>
<h2>Practical Tools and Getting Started</h2>
<p>The Hugging Face ecosystem provides the foundation for modern fine-tuning. The <strong>Transformers</strong> library offers model access, <strong>PEFT</strong> implements parameter-efficient methods, <strong>TRL</strong> handles training with reinforcement learning and supervised fine-tuning, and <strong>bitsandbytes</strong> enables quantization.</p>
<p><strong>Unsloth delivers ~2× faster training and up to ~80% less memory</strong> through custom Triton kernels, working on single T4 or consumer GPUs. It’s free on Colab and Kaggle. LlamaFactory has emerged as a unified solution, supporting 100+ models with configuration-based training. For very large models, fully sharded data parallelism (FSDP) combined with QLoRA enables training of 70B models on dual consumer GPUs.</p>
<p>The recommended 2025 stack for ~8B models: QLoRA or Spectrum + FlashAttention-2 + Liger Kernels + gradient checkpointing. This enables Llama-3.1-8B training in around two hours on a single strong GPU or well under half an hour distributed across 8 GPUs (your mileage will vary).</p>
<p><strong>Recommended initial configuration:</strong> Choose Llama-3.1-8B or Phi-3-mini as base models for good performance and manageable size. Use QLoRA for 4-bit quantization enabling consumer-GPU training. Implement on <a href="https://docs.unsloth.ai/" rel="noopener" target="_blank"><strong>Unsloth</strong></a> for free access. Start with 512–1,024 token sequences. Set learning rate to 2e-4. Use batch size 4–8 with gradient accumulation 2–4 steps. Enable gradient checkpointing and sequence packing for efficiency.</p>
<p>Essential datasets for practice: Alpaca (52K) for instruction tuning, Dolly-15k for high-quality human examples, OpenAssistant for conversational data, Anthropic HH-RLHF for preference learning.</p>
<p><strong>Ready to build your first fine-tuned model?</strong> Start with <a href="https://huggingface.co/learn/llm-course/chapter11/1" rel="noopener" target="_blank"><strong>Hugging Face’s LLM Course chapter on supervised fine-tuning</strong></a>, which walks through the complete process step-by-step. For production deployments, explore <a href="https://github.com/hiyouga/LLaMA-Factory" rel="noopener" target="_blank"><strong>LlamaFactory</strong></a>, which supports 100+ models with simple YAML configuration.</p>
<h2>Your Learning Path</h2>
<p>For machine learning practitioners new to fine-tuning, adopt a progressive learning approach that builds skills systematically.</p>
<p><strong>Start with instruction tuning:</strong> Fine-tune base T5 or base Llama-2 on the Alpaca dataset. Focus on understanding instruction-response data formatting and use the Hugging Face TRL <code>SFTTrainer</code> with LoRA for efficient training. This establishes foundations in data preparation, training, and evaluation.</p>
<p><strong>Progress to DPO:</strong> Train on small preference datasets like Anthropic HH-RLHF or UltraFeedback. Compare performance against your supervised fine-tuning baseline. Understand implicit rewards and preference learning. DPO’s simplicity makes it ideal for learning alignment concepts without reinforcement learning complexity.</p>
<p><strong>Experiment with production systems:</strong> Start with small models (1B to 3B parameters) to iterate quickly. Use existing implementations rather than building from scratch. Perform careful ablations isolating the impact of different choices. Evaluate rigorously using multiple metrics before scaling to larger models.</p>
<p><strong>Getting started checklist:</strong> Define a clear task and success criteria, including target metrics. Choose one to two custom evaluation metrics and two to three system-level metrics (maximum five total). Prepare a minimum of ~1,000 examples, prioritizing quality over quantity with an 80/20 train/validation split. Set up your evaluation framework before training begins. Start fine-tuning using PEFT methods with proven hyperparameters. Monitor continuously to prevent pitfalls. With QLoRA enabling 13B models on 16 GB GPUs and free platforms like Google Colab with Unsloth optimization, you can start experimenting today.</p>
<p><strong>Looking for evaluation best practices?</strong> The guide <a href="https://www.confident-ai.com/blog/llm-evaluation-metrics-everything-you-need-for-llm-evaluation" rel="noopener" target="_blank"><strong>LLM Evaluation Metrics: The Ultimate Guide</strong></a> covers G-Eval, task-specific metrics, and LLM-as-a-judge methods. Implement evaluations using <a href="https://github.com/confident-ai/deepeval" rel="noopener" target="_blank"><strong>DeepEval</strong></a> for open-source evaluation frameworks.</p>
<p>The field continues evolving rapidly, with 2024–2025 advances bringing significant speed-ups (often 3–5×), improved efficiency techniques, and expanded commercial availability. Start with small models and proven techniques, then scale as you get comfortable with the fundamentals.</p>
</section>

content_text:
In this article, you will learn when fine-tuning large language models is warranted, which 2025-ready methods and tools to choose, and how to avoid the most common mistakes that derail projects.

Topics we will cover include:

A practical decision framework: prompt engineering, retrieval-augmented generation (RAG), and when fine-tuning truly adds value.

Today’s essential methods—LoRA/QLoRA, Spectrum—and alignment with DPO, plus when to pick each.

Data preparation, evaluation, and proven configurations that keep you out of trouble.

Let’s not waste any more time.

The Machine Learning Practitioner’s Guide to Fine-Tuning Language Models

Image by Author

Introduction

Fine-tuning has become much more accessible in 2024–2025, with parameter-efficient methods letting even 70B+ parameter models run on consumer GPUs. But should you fine-tune at all? And if so, how do you choose between the dozens of emerging techniques?

This guide is for practitioners who want results, not just theory. You’ll learn when fine-tuning makes sense, which methods to use, and how to avoid common pitfalls.

Fine-tuning is different from traditional machine learning. Instead of training models from scratch, you’re adapting pretrained models to specialized tasks using far less data and compute. This makes sophisticated natural language processing (NLP) capabilities accessible without billion-dollar budgets.

For machine learning practitioners, this builds on skills you already have. Data preparation, evaluation frameworks, and hyperparameter tuning remain central. You’ll need to learn new architectural patterns and efficiency techniques, but your existing foundation gives you a major advantage.

You’ll learn:

When fine-tuning provides value versus simpler alternatives like prompt engineering or retrieval-augmented generation (RAG)

The core parameter-efficient methods (LoRA, QLoRA, Spectrum) and when to use each

Modern alignment techniques (DPO, RLHF) that make models follow instructions reliably

Data preparation strategies that determine most of your fine-tuning success

Critical pitfalls in overfitting and catastrophic forgetting, and how to avoid them

If you’re already working with LLMs, you have what you need. If you need a refresher, check out our guides on

prompt engineering

and

LLM applications

.

Before getting into fine-tuning mechanics, you need to understand whether fine-tuning is the right approach.

When to Fine-Tune Versus Alternative Approaches

Fine-tuning should be your last resort, not your first choice.

The recommended progression starts with prompt engineering, escalates to RAG when external knowledge is needed, and only proceeds to fine-tuning when deep specialization is required.

Google Cloud’s decision framework

and

Meta AI’s practical guide

identify clear criteria: Use prompt engineering for basic task adaptation. Use RAG when you need source citations, must ground responses in documents, or information changes frequently. Meta AI reveals five scenarios where fine-tuning provides genuine value: customizing tone and style for specific audiences, maintaining data privacy for sensitive information, supporting low-resource languages, reducing inference costs by distilling larger models, and adding entirely new capabilities not present in base models.

The data availability test

: With fewer than 100 examples, stick to prompt engineering. With 100–1,000 examples and static knowledge, consider parameter-efficient methods. Only with 1,000–100,000 examples and a clear task definition should you attempt fine-tuning.

For news summarization or general question answering, RAG excels. For customer support requiring a specific brand voice or code generation following particular patterns, fine-tuning proves essential. The optimal solution often combines both—fine-tune for specialized reasoning patterns while using RAG for current information.

Essential Parameter-Efficient Fine-Tuning Methods

Full fine-tuning updates all model parameters, requiring massive compute and memory. Parameter-efficient fine-tuning (PEFT) revolutionized this by enabling training with just ~0.1% to 3% of parameters updated, achieving comparable performance while dramatically reducing requirements.

LoRA (Low-Rank Adaptation) emerged as the dominant technique.

LoRA freezes pretrained weights and injects trainable rank-decomposition matrices in parallel. Instead of updating entire weight matrices, LoRA represents updates as low-rank decompositions. Weight updates during adaptation often have low intrinsic rank, with rank 8 typically sufficient for many tasks.

Memory reductions reach 2× to 3× versus full fine-tuning, with checkpoint sizes decreasing 1,000× to 10,000×. A 350 GB model can require only a ~35 MB adapter file. Training can be ~25% faster on large models. Critically, learned matrices merge with frozen weights during deployment, introducing zero inference latency.

QLoRA extends LoRA through aggressive quantization

while maintaining accuracy. Base weights are stored in 4-bit format with computation happening in 16-bit

bfloat16

. The results can be dramatic: 65B models on 48 GB GPUs, 33B on 24 GB, 13B on consumer 16 GB hardware—while matching many 16-bit full fine-tuning results.

Spectrum

, a 2024 innovation, takes a different approach. Rather than adding adapters, Spectrum identifies the most informative layers using signal-to-noise ratio analysis and selectively fine-tunes only the top ~30%. Reports show higher accuracy than QLoRA on mathematical reasoning with comparable resources.

Decision framework

: Use LoRA when you need zero inference latency and moderate GPU resources (16–24 GB). Use QLoRA for extreme memory constraints (consumer GPUs, Google Colab) or very large models (30B+). Use Spectrum when working with large models in distributed settings.

Ready to implement LoRA and QLoRA?

How to fine-tune open LLMs in 2025

by Phil Schmid provides complete code examples with current best practices. For hands-on practice, try

Unsloth’s free Colab notebooks

.

Modern Alignment and Instruction Tuning

Instruction tuning transforms completion-focused base models into instruction-following assistants, establishing basic capabilities before alignment. The method trains on diverse instruction-response pairs covering question answering, summarization, translation, and reasoning. Quality matters far more than quantity, with ~1,000 high-quality examples often sufficient.

Direct Preference Optimization (DPO) has rapidly become the preferred alignment method

by dramatically simplifying reinforcement learning from human feedback (RLHF). The key idea: re-parameterize the reward as implicit in the policy itself, solving the RLHF objective through supervised learning rather than complex reinforcement learning.

Research from Stanford and others reports that DPO can achieve comparable or superior performance to PPO-based RLHF with single-stage training, ~50% less compute, and greater stability. DPO requires only preference data (prompt, chosen response, rejected response), a reference policy, and standard supervised learning infrastructure. The method has become common for training open-source LLMs in 2024–2025, including Zephyr-7B and various Mistral-based models.

RLHF remains the foundational alignment technique but brings high complexity: managing four model copies during training (policy, reference, reward, value), difficult implementations, and training instability. OpenAI’s InstructGPT demonstrated that a 1.3B aligned model could outperform a 175B base model on human evaluations, underscoring alignment’s power. However, most practitioners should use DPO unless specific scenarios demand RLHF’s flexibility.

Start with instruction tuning using datasets like Alpaca or Dolly-15k, then implement DPO for alignment

rather than attempting RLHF.

TRL (Transformer Reinforcement Learning) documentation

provides comprehensive guides for both DPO and RLHF with working code examples. For conceptual understanding, see Chip Huyen’s

RLHF: Reinforcement Learning from Human Feedback

.

Data Preparation Best Practices

Data quality determines fine-tuning success more than any other factor.

As error rates in training data increase linearly, downstream model error can rise superlinearly—making data curation your highest-leverage activity.

Dataset size requirements vary by task complexity. Simple classification needs ~200 to 1,000 examples. Medium-complexity tasks like question answering require ~1,000 to 5,000. Complex generation or reasoning can demand 5,000 to 10,000+. Quality trumps quantity: 1,000 high-quality examples can outperform 100,000 mediocre ones.

High-quality data exhibits five characteristics: domain relevance, diversity across scenarios, representativeness of the full distribution, labeling accuracy, and freshness for time-sensitive domains.

Formatting impacts results significantly.

Use structured question-answer pairs with consistent formatting across datasets to prevent spurious pattern learning. Standard splits allocate ~80% training and ~20% validation using stratified sampling when applicable.

Essential preprocessing: clean noise, handle missing values, use model-specific tokenizers, remove duplicates, and normalize text. Favor proprietary custom data over public datasets that models may have already encountered during pretraining.

Need help with data preparation?

Meta AI’s guide

How to fine-tune: Focus on effective datasets

emphasizes proprietary data strategies and provides practical curation techniques. For dataset exploration, browse

Hugging Face Datasets

to see quality examples.

Avoiding Critical Pitfalls

Overfitting occurs when models memorize training data instead of learning generalizable patterns.

It’s the most common fine-tuning failure. Signs include training loss decreasing while validation loss increases, high training accuracy but poor validation performance, and loss approaching zero.

Prevention requires multiple strategies. Early stopping halts training when validation performance plateaus. Regularization includes L2 weight decay, 10%–30% dropout, and weight penalties. Data augmentation increases diversity through back-translation and synthetic generation. K-fold cross-validation helps ensure generalization across splits.

For parameter-efficient fine-tuning (PEFT) methods, reduce LoRA rank (

r

parameter) and alpha values to decrease trainable parameters. Use learning rates of 1e-4 to 2e-4 for fine-tuning. Monitor both training and validation losses continuously. PEFT methods like LoRA naturally reduce overfitting by limiting trainable parameters to ~0.1%–1%.

Catastrophic forgetting poses a more insidious challenge:

loss of previously learned information when training on new tasks. Models can lose general reasoning abilities, decline on previously answerable questions, and overfit to specific output formats. Forgetting can begin early in fine-tuning through format specialization.

Prevention strategies include elastic weight consolidation (EWC), which identifies and protects important weights; “half fine-tuning,” which freezes roughly half of parameters during each round; and sharpness-aware minimization (SAM), which flattens the loss landscape. Most accessible: include diverse instruction datasets mixed with domain-specific data.

Struggling with overfitting or catastrophic forgetting?

The paper

Revisiting Catastrophic Forgetting in Large Language Model Tuning

provides practical mitigation strategies with empirical evidence. For monitoring and debugging, use

Weights & Biases

or

TensorBoard

to track training and validation metrics continuously.

Practical Tools and Getting Started

The Hugging Face ecosystem provides the foundation for modern fine-tuning. The

Transformers

library offers model access,

PEFT

implements parameter-efficient methods,

TRL

handles training with reinforcement learning and supervised fine-tuning, and

bitsandbytes

enables quantization.

Unsloth delivers ~2× faster training and up to ~80% less memory

through custom Triton kernels, working on single T4 or consumer GPUs. It’s free on Colab and Kaggle. LlamaFactory has emerged as a unified solution, supporting 100+ models with configuration-based training. For very large models, fully sharded data parallelism (FSDP) combined with QLoRA enables training of 70B models on dual consumer GPUs.

The recommended 2025 stack for ~8B models: QLoRA or Spectrum + FlashAttention-2 + Liger Kernels + gradient checkpointing. This enables Llama-3.1-8B training in around two hours on a single strong GPU or well under half an hour distributed across 8 GPUs (your mileage will vary).

Recommended initial configuration:

Choose Llama-3.1-8B or Phi-3-mini as base models for good performance and manageable size. Use QLoRA for 4-bit quantization enabling consumer-GPU training. Implement on

Unsloth

for free access. Start with 512–1,024 token sequences. Set learning rate to 2e-4. Use batch size 4–8 with gradient accumulation 2–4 steps. Enable gradient checkpointing and sequence packing for efficiency.

Essential datasets for practice: Alpaca (52K) for instruction tuning, Dolly-15k for high-quality human examples, OpenAssistant for conversational data, Anthropic HH-RLHF for preference learning.

Ready to build your first fine-tuned model?

Start with

Hugging Face’s LLM Course chapter on supervised fine-tuning

, which walks through the complete process step-by-step. For production deployments, explore

LlamaFactory

, which supports 100+ models with simple YAML configuration.

Your Learning Path

For machine learning practitioners new to fine-tuning, adopt a progressive learning approach that builds skills systematically.

Start with instruction tuning:

Fine-tune base T5 or base Llama-2 on the Alpaca dataset. Focus on understanding instruction-response data formatting and use the Hugging Face TRL

SFTTrainer

with LoRA for efficient training. This establishes foundations in data preparation, training, and evaluation.

Progress to DPO:

Train on small preference datasets like Anthropic HH-RLHF or UltraFeedback. Compare performance against your supervised fine-tuning baseline. Understand implicit rewards and preference learning. DPO’s simplicity makes it ideal for learning alignment concepts without reinforcement learning complexity.

Experiment with production systems:

Start with small models (1B to 3B parameters) to iterate quickly. Use existing implementations rather than building from scratch. Perform careful ablations isolating the impact of different choices. Evaluate rigorously using multiple metrics before scaling to larger models.

Getting started checklist:

Define a clear task and success criteria, including target metrics. Choose one to two custom evaluation metrics and two to three system-level metrics (maximum five total). Prepare a minimum of ~1,000 examples, prioritizing quality over quantity with an 80/20 train/validation split. Set up your evaluation framework before training begins. Start fine-tuning using PEFT methods with proven hyperparameters. Monitor continuously to prevent pitfalls. With QLoRA enabling 13B models on 16 GB GPUs and free platforms like Google Colab with Unsloth optimization, you can start experimenting today.

Looking for evaluation best practices?

The guide

LLM Evaluation Metrics: The Ultimate Guide

covers G-Eval, task-specific metrics, and LLM-as-a-judge methods. Implement evaluations using

DeepEval

for open-source evaluation frameworks.

The field continues evolving rapidly, with 2024–2025 advances bringing significant speed-ups (often 3–5×), improved efficiency techniques, and expanded commercial availability. Start with small models and proven techniques, then scale as you get comfortable with the fundamentals.

thumb_url:
https://machinelearningmastery.com/wp-content/uploads/2025/10/mlm-chugani-machine-learning-practitioners-guide-fine-tuning-language-models-feature-1024x683.png


url: https://www.philschmid.de/building-agents
title: Practical Guide on how to build an Agent from scratch with Gemini 3
added_at: 2025-11-22 12:21:19
thumb_url: https://www.philschmid.de/static/blog/building-agents/agent.png
content_html:
<p>It seems complicated, when you watch an AI agent edit multiple files, run commands, handle errors, and iteratively solve a problem, it feels like magic. But it isn’t. The secret to building an agent is that there is no secret.</p>
<p>The core of an Agent is surprisingly simple: It is a Large Language Model (LLM) running in a loop, equipped with tools it can choose to use.</p>
<p>If you can write a loop in Python, you can build an agent. This guide will walk you through the process, from a simple API call to a functioning CLI Agent.</p>
<h2><a href="https://www.philschmid.de/building-agents#what-actually-is-an-agent" rel="noopener noreferrer" target="_blank"></a>What actually is an Agent?</h2>
<p>Traditional software workflows are prescriptive and follow predefined paths (<code>Step A -&gt; Step B -&gt; Step C</code>), <strong>Agents</strong> are System that uses an LLM to dynamically decide the control flow of an application to achieve a user goal.</p>
<p>An agent generally consists of these core components:</p>
<ol>
<li><strong>The Model (Brain):</strong> The reasoning engine, in our case a Gemini model. It reasons through ambiguity, plans steps, and decides when it needs outside help.</li>
<li><strong>Tools (Hands and Eyes):</strong> Functions the agent can execute to interact with the outside world/environment (e.g., searching the web, reading a file, calling an API).</li>
<li><strong>Context/Memory (Workspace):</strong> The information the agent has access to at any moment. Managing this effectively, known as <strong>Context Engineering</strong>.</li>
<li><strong>The Loop (Life):</strong> A <code>while</code> loop that allows the model to: Observe → Think → Act → Observe again, until the task is complete.</li>
</ol>
<p><img alt="agent" loading="lazy" referrerpolicy="no-referrer" src="https://www.philschmid.de/static/blog/building-agents/agent.png"/></p>
<p><strong>"The Loop"</strong> of nearly every agent is an iterative process:</p>
<ol>
<li><strong>Define Tool Definitions:</strong> You describe your available tools (e.g., <code>get_weather</code>) to the model using a structured JSON format.</li>
<li><strong>Call the LLM:</strong> You send the user's prompt <em>and</em> the tool definitions to the model.</li>
<li><strong>Model Decision:</strong> The model analyzes the request. If a tool is needed, it returns a structured <code>tool use</code> containing the tool name and arguments.</li>
<li><strong>Execute Tool (Client Responsibility):</strong> The client/application code intercepts this <code>tool use</code>, executes the actual code or API call, and captures the result.</li>
<li><strong>Respond and Iterate:</strong> You send the result (the <code>tool response</code>) back to model. The model uses this new information to decide the next step, either calling another tool or generating the final response.</li>
</ol>
<h2><a href="https://www.philschmid.de/building-agents#building-an-agent" rel="noopener noreferrer" target="_blank"></a>Building an Agent</h2>
<p>Let's build an agent step-by-step, progressing from basic text generation to a functional CLI agent using Gemini 3 Pro and Python SDK.</p>
<p><em>Prerequisites: Install the SDK (<code>pip install google-genai</code>) and set your <code>GEMINI_API_KEY</code> environment variable (<a href="https://aistudio.google.com/app/apikey" rel="noopener noreferrer" target="_blank">Get it in AI Studio</a>).</em></p>
<h3><a href="https://www.philschmid.de/building-agents#step-1-basic-text-generation-and-abstraction" rel="noopener noreferrer" target="_blank"></a>Step 1: Basic Text Generation and Abstraction</h3>
<p>The first step is to create a baseline interaction with the LLM, for us Gemini 3 Pro. We are going to create a simple Agent class abstraction to structure our code, which we will extend throughout this guide. We will first start with a simple chatbot that maintains a conversation history.</p>
<figure><pre><code>from google import genai
from google.genai import types
 
class Agent:
    def __init__(self, model: str):
        self.model = model
        self.client = genai.Client()
        self.contents = []
 
    def run(self, contents: str):
        self.contents.append({"role": "user", "parts": [{"text": contents}]})
 
        response = self.client.models.generate_content(model=self.model, contents=self.contents)
        self.contents.append(response.candidates[0].content)
 
        return response
 
agent = Agent(model="gemini-3-pro-preview")
response1 = agent.run(
    contents="Hello, What are top 3 cities in Germany to visit? Only return the names of the cities."
)
 
print(f"Model: {response1.text}")
# Output: Berlin, Munich, Cologne 
response2 = agent.run(
    contents="Tell me something about the second city."
)
 
print(f"Model: {response2.text}")
# Output: Munich is the capital of Bavaria and is known for its Oktoberfest.</code></pre></figure>
<p>This is <em>not</em> an agent yet. It is a standard chatbot. It maintains state but cannot take action, has no "hands or eyes".</p>
<p>To start turning this an agent, we need <strong>Tool Use</strong> or Function Calling. We provide the agent with tools. This requires defining the implementation (the Python code) and the definition (the schema the LLM sees). If the LLM believes that tool will help solve a user's prompt, it will return a structured request to call that function instead of just text.</p>
<p>We are going to create 3 tools, <code>read_file</code>, <code>write_file</code>, and <code>list_dir</code>. A tool Definition is a JSON schema that defines the <code>name</code>, <code>description</code>, and <code>parameters</code> of the tool.</p>
<p><em><strong>Best Practice</strong>: Use the <code>description</code> fields to explain when and how to use the tool. The model relies heavily on these to understand when and how to use the tool. Be explicit and clear.</em></p>
<figure><pre><code>import os
import json
 
read_file_definition = {
    "name": "read_file",
    "description": "Reads a file and returns its contents.",
    "parameters": {
        "type": "object",
        "properties": {
            "file_path": {
                "type": "string",
                "description": "Path to the file to read.",
            }
        },
        "required": ["file_path"],
    },
}
 
list_dir_definition = {
    "name": "list_dir",
    "description": "Lists the contents of a directory.",
    "parameters": {
        "type": "object",
        "properties": {
            "directory_path": {
                "type": "string",
                "description": "Path to the directory to list.",
            }
        },
        "required": ["directory_path"],
    },
}
 
write_file_definition = {
    "name": "write_file",
    "description": "Writes a file with the given contents.",
    "parameters": {
        "type": "object",
        "properties": {
            "file_path": {
                "type": "string",
                "description": "Path to the file to write.",
            },
            "contents": {
                "type": "string",
                "description": "Contents to write to the file.",
            },
        },
        "required": ["file_path", "contents"],
    },
}
 
def read_file(file_path: str) -&gt; dict:
    with open(file_path, "r") as f:
        return f.read()
 
def write_file(file_path: str, contents: str) -&gt; bool:
    """Writes a file with the given contents."""
    with open(file_path, "w") as f:
        f.write(contents)
    return True
 
def list_dir(directory_path: str) -&gt; list[str]:
    """Lists the contents of a directory."""
    full_path = os.path.expanduser(directory_path)
    return os.listdir(full_path)
 
file_tools = {
    "read_file": {"definition": read_file_definition, "function": read_file},
    "write_file": {"definition": write_file_definition, "function": write_file},
    "list_dir": {"definition": list_dir_definition, "function": list_dir},
}</code></pre></figure>
<p>Now we integrate the <code>tools</code> and function calls into our Agent class.</p>
<figure><pre><code>from google import genai
from google.genai import types
 
class Agent:
    def __init__(self, model: str,tools: list[dict]):
        self.model = model
        self.client = genai.Client()
        self.contents = []
        self.tools = tools
 
    def run(self, contents: str):
        self.contents.append({"role": "user", "parts": [{"text": contents}]})
 
        config = types.GenerateContentConfig(
            tools=[types.Tool(function_declarations=[tool["definition"] for tool in self.tools.values()])],
        )
 
        response = self.client.models.generate_content(model=self.model, contents=self.contents, config=config)
        self.contents.append(response.candidates[0].content)
 
        return response
 
agent = Agent(model="gemini-3-pro-preview", tools=file_tools)
 
response = agent.run(
    contents="Can you list my files in the current directory?"
)
print(response.function_calls)
# Output: [FunctionCall(name='list_dir', arguments={'directory_path': '.'})]</code></pre></figure>
<p>Great! The model has successfully called the tool. Now, we need to add the tool execution logic to our Agent class and the loop return the result back to the model.</p>
<h3><a href="https://www.philschmid.de/building-agents#step-3-closing-the-loop-the-agent" rel="noopener noreferrer" target="_blank"></a>Step 3: Closing the Loop (The Agent)</h3>
<p>An Agent isn't about generating one tool call, but about generating a series of tool calls, returning the results back to the model, and then generating another tool call, and so on until the task is completed.</p>
<p>The <code>Agent</code> class handles the core loop: intercepting the <code>FunctionCall</code>, executing the tool on the client side, and sending back the <code>FunctionResponse</code>. We also add a <code>SystemInstruction</code> to the model to guide the model on what to do.</p>
<p><em>Note: Gemini 3 uses <a href="https://ai.google.dev/gemini-api/docs/gemini-3?thinking=high#thought_signatures" rel="noopener noreferrer" target="_blank">Thought signatures</a> to maintain reasoning context across API calls. You must return these signatures back to the model in your request exactly as they were received.</em></p>
<figure><pre><code># ... Code for the tools and tool definitions from Step 2 should be here ...
 
from google import genai
from google.genai import types
 
class Agent:
    def __init__(self, model: str,tools: list[dict], system_instruction: str = "You are a helpful assistant."):
        self.model = model
        self.client = genai.Client()
        self.contents = []
        self.tools = tools
        self.system_instruction = system_instruction
 
    def run(self, contents: str | list[dict[str, str]]):
        if isinstance(contents, list):
            self.contents.append({"role": "user", "parts": contents})
        else:
            self.contents.append({"role": "user", "parts": [{"text": contents}]})
 
        config = types.GenerateContentConfig(
            system_instruction=self.system_instruction,
            tools=[types.Tool(function_declarations=[tool["definition"] for tool in self.tools.values()])],
        )
 
        response = self.client.models.generate_content(model=self.model, contents=self.contents, config=config)
        self.contents.append(response.candidates[0].content)
 
        if response.function_calls:
            functions_response_parts = []
            for tool_call in response.function_calls:
                print(f"[Function Call] {tool_call}")
 
                if tool_call.name in self.tools:
                    result = {"result": self.tools[tool_call.name]["function"](**tool_call.args)}
                else:
                    result = {"error": "Tool not found"}
 
                print(f"[Function Response] {result}")
                functions_response_parts.append({"functionResponse": {"name": tool_call.name, "response": result}})
 
            return self.run(functions_response_parts)
        
        return response
 
agent = Agent(
    model="gemini-3-pro-preview", 
    tools=file_tools, 
    system_instruction="You are a helpful Coding Assistant. Respond like you are Linus Torvalds."
)
 
response = agent.run(
    contents="Can you list my files in the current directory?"
)
print(response.text)
# Output: [Function Call] id=None args={'directory_path': '.'} name='list_dir'
# [Function Response] {'result': ['.venv', ... ]}
# There. Your current directory contains: `LICENSE`,</code></pre></figure>
<p>Congratulations. You just built your first functioning agent.</p>
<h3><a href="https://www.philschmid.de/building-agents#phase-4-multi-turn-cli-agent" rel="noopener noreferrer" target="_blank"></a>Phase 4: Multi-turn CLI Agent</h3>
<p>Now we can run our agent in a simple CLI loop. It takes surprisingly little code to create highly capable behavior.</p>
<figure><pre><code># ... Code for the Agent, tools and tool definitions from Step 3 should be here ...
 
agent = Agent(
    model="gemini-3-pro-preview", 
    tools=file_tools, 
    system_instruction="You are a helpful Coding Assistant. Respond like you are Linus Torvalds."
)
 
print("Agent ready. Ask it to check files in this directory.")
while True:
    user_input = input("You: ")
    if user_input.lower() in ['exit', 'quit']:
        break
 
    response = agent.run(user_input)
    print(f"Linus: {response.text}\n")</code></pre></figure>
<h2><a href="https://www.philschmid.de/building-agents#best-practices-for-engineering-agents" rel="noopener noreferrer" target="_blank"></a>Best Practices for Engineering Agents</h2>
<p>Building the loop is easy; making it reliable, transparent, and controllable is hard. Here are key engineering principles derived from top industry practices, grouped by functional area.</p>
<p>Your tools are the interface for the model. Don't just wrap your existing internal APIs. If a tool is confusing to a human, it's confusing to the model:</p>
<ul>
<li><strong>Clear Naming:</strong> Use obvious names like <code>search_customer_database</code> rather than <code>cust_db_v2_query</code>.</li>
<li><strong>Precise Descriptions:</strong> Gemini reads the function docstrings to understand <em>when</em> and <em>how</em> to use a tool. Spend time writing these carefully, it is essentially "prompt engineering" for tools.</li>
<li><strong>Return Meaningful Errors:</strong> Don't return a 50-line Java stack trace. If a tool fails, return a clear string like <code>Error: File not found. Did you mean 'data.csv'?</code>. This allows the agent to self-correct.</li>
<li><strong>Tolerate Fuzzy Inputs:</strong> If a model frequently guesses file paths wrong, update your tool to handle relative paths or fuzzy inputs rather than just erroring out.</li>
</ul>
<h3><a href="https://www.philschmid.de/building-agents#2-context-engineering" rel="noopener noreferrer" target="_blank"></a>2. Context Engineering</h3>
<p>Models have a finite "attention budget." Managing what information enters the context is crucial for performance and cost.</p>
<ul>
<li><strong>Don't "Dump" Data:</strong> Don't have a tool that returns an entire 10MB database table. Instead of <code>get_all_users()</code>, create <code>search_users(query: str)</code>.</li>
<li><strong>Just-in-time Loading:</strong> Instead of pre-loading all data (traditional RAG), use just-in-time strategies. The agent should maintain lightweight identifiers (file paths, IDs) and use tools to dynamically load content only when needed.</li>
<li><strong>Compression:</strong> For very long-running agents, summarize the history, remove old context or start a new sessions.</li>
<li><strong>Agentic Memory:</strong> Allow the agent to maintain notes or a scratchpad persisted <em>outside</em> the context window, pulling them back in only when relevant.</li>
</ul>
<h3><a href="https://www.philschmid.de/building-agents#3-dont-over-engineer" rel="noopener noreferrer" target="_blank"></a>3. Don't over engineer</h3>
<p>It's tempting to build complex multi-agent systems. Don't.</p>
<ul>
<li><strong>Maximize a Single Agent First:</strong> Don't immediately build complex multi-agent systems. Gemini is highly capable of handling dozens of tools in a single prompt.</li>
<li><strong>Escape Hatches:</strong> Ensure loops can be stopped like a <code>max_iterations</code> break (e.g., 15 turns).</li>
<li><strong>Guardrails and System Instructions:</strong> Use the <code>system_instruction</code> to guide the model with hard rules (e.g., "You are strictly forbidden from offering refunds greater than $50") or use external classifier.</li>
<li><strong>Human-in-the-loop:</strong> For sensitive actions (like <code>send_email</code> or <code>execute_code</code>), pause the loop and require user confirmation before the tool is actually executed.</li>
<li><strong>Prioritize Transparency and Debugging:</strong> Log tool calls and parameters. Analyzing the model's reasoning helps identify if issues, and improve the agent over time.</li>
</ul>
<h2><a href="https://www.philschmid.de/building-agents#conclusion" rel="noopener noreferrer" target="_blank"></a>Conclusion</h2>
<p>Building an agent is no longer magic; it is a practical engineering task. As we've shown, you can build a working prototype in under 100 lines of code. While understanding these fundamentals is key, don't get bogged down re-engineering the same pattern over and over. The AI community has created fantastic open-source libraries that can help you build more complex and robust agents faster.</p>
<hr/>
<p>Thanks for reading! If you have any questions or feedback, please let me know on <a href="https://twitter.com/_philschmid" rel="noopener noreferrer" target="_blank">Twitter</a> or <a href="https://www.linkedin.com/in/philipp-schmid-a6a2bb196/" rel="noopener noreferrer" target="_blank">LinkedIn</a>.</p>

content_text:
It seems complicated, when you watch an AI agent edit multiple files, run commands, handle errors, and iteratively solve a problem, it feels like magic. But it isn’t. The secret to building an agent is that there is no secret.

The core of an Agent is surprisingly simple: It is a Large Language Model (LLM) running in a loop, equipped with tools it can choose to use.

If you can write a loop in Python, you can build an agent. This guide will walk you through the process, from a simple API call to a functioning CLI Agent.

What actually is an Agent?

Traditional software workflows are prescriptive and follow predefined paths (

Step A -> Step B -> Step C

),

Agents

are System that uses an LLM to dynamically decide the control flow of an application to achieve a user goal.

An agent generally consists of these core components:

The Model (Brain):

The reasoning engine, in our case a Gemini model. It reasons through ambiguity, plans steps, and decides when it needs outside help.

Tools (Hands and Eyes):

Functions the agent can execute to interact with the outside world/environment (e.g., searching the web, reading a file, calling an API).

Context/Memory (Workspace):

The information the agent has access to at any moment. Managing this effectively, known as

Context Engineering

.

The Loop (Life):

A

while

loop that allows the model to: Observe → Think → Act → Observe again, until the task is complete.

"The Loop"

of nearly every agent is an iterative process:

Define Tool Definitions:

You describe your available tools (e.g.,

get_weather

) to the model using a structured JSON format.

Call the LLM:

You send the user's prompt

and

the tool definitions to the model.

Model Decision:

The model analyzes the request. If a tool is needed, it returns a structured

tool use

containing the tool name and arguments.

Execute Tool (Client Responsibility):

The client/application code intercepts this

tool use

, executes the actual code or API call, and captures the result.

Respond and Iterate:

You send the result (the

tool response

) back to model. The model uses this new information to decide the next step, either calling another tool or generating the final response.

Building an Agent

Let's build an agent step-by-step, progressing from basic text generation to a functional CLI agent using Gemini 3 Pro and Python SDK.

Prerequisites: Install the SDK (

pip install google-genai

) and set your

GEMINI_API_KEY

environment variable (

Get it in AI Studio

).

Step 1: Basic Text Generation and Abstraction

The first step is to create a baseline interaction with the LLM, for us Gemini 3 Pro. We are going to create a simple Agent class abstraction to structure our code, which we will extend throughout this guide. We will first start with a simple chatbot that maintains a conversation history.

from google import genai

from google.genai import types

class Agent:

def __init__(self, model: str):

self.model = model

self.client = genai.Client()

self.contents = []

def run(self, contents: str):

self.contents.append({"role": "user", "parts": [{"text": contents}]})

response = self.client.models.generate_content(model=self.model, contents=self.contents)

self.contents.append(response.candidates[0].content)

return response

agent = Agent(model="gemini-3-pro-preview")

response1 = agent.run(

contents="Hello, What are top 3 cities in Germany to visit? Only return the names of the cities."

)

print(f"Model: {response1.text}")

# Output: Berlin, Munich, Cologne

response2 = agent.run(

contents="Tell me something about the second city."

)

print(f"Model: {response2.text}")

# Output: Munich is the capital of Bavaria and is known for its Oktoberfest.

This is

not

an agent yet. It is a standard chatbot. It maintains state but cannot take action, has no "hands or eyes".

To start turning this an agent, we need

Tool Use

or Function Calling. We provide the agent with tools. This requires defining the implementation (the Python code) and the definition (the schema the LLM sees). If the LLM believes that tool will help solve a user's prompt, it will return a structured request to call that function instead of just text.

We are going to create 3 tools,

read_file

,

write_file

, and

list_dir

. A tool Definition is a JSON schema that defines the

name

,

description

, and

parameters

of the tool.

Best Practice

: Use the

description

fields to explain when and how to use the tool. The model relies heavily on these to understand when and how to use the tool. Be explicit and clear.

import os

import json

read_file_definition = {

"name": "read_file",

"description": "Reads a file and returns its contents.",

"parameters": {

"type": "object",

"properties": {

"file_path": {

"type": "string",

"description": "Path to the file to read.",

}

},

"required": ["file_path"],

},

}

list_dir_definition = {

"name": "list_dir",

"description": "Lists the contents of a directory.",

"parameters": {

"type": "object",

"properties": {

"directory_path": {

"type": "string",

"description": "Path to the directory to list.",

}

},

"required": ["directory_path"],

},

}

write_file_definition = {

"name": "write_file",

"description": "Writes a file with the given contents.",

"parameters": {

"type": "object",

"properties": {

"file_path": {

"type": "string",

"description": "Path to the file to write.",

},

"contents": {

"type": "string",

"description": "Contents to write to the file.",

},

},

"required": ["file_path", "contents"],

},

}

def read_file(file_path: str) -> dict:

with open(file_path, "r") as f:

return f.read()

def write_file(file_path: str, contents: str) -> bool:

"""Writes a file with the given contents."""

with open(file_path, "w") as f:

f.write(contents)

return True

def list_dir(directory_path: str) -> list[str]:

"""Lists the contents of a directory."""

full_path = os.path.expanduser(directory_path)

return os.listdir(full_path)

file_tools = {

"read_file": {"definition": read_file_definition, "function": read_file},

"write_file": {"definition": write_file_definition, "function": write_file},

"list_dir": {"definition": list_dir_definition, "function": list_dir},

}

Now we integrate the

tools

and function calls into our Agent class.

from google import genai

from google.genai import types

class Agent:

def __init__(self, model: str,tools: list[dict]):

self.model = model

self.client = genai.Client()

self.contents = []

self.tools = tools

def run(self, contents: str):

self.contents.append({"role": "user", "parts": [{"text": contents}]})

config = types.GenerateContentConfig(

tools=[types.Tool(function_declarations=[tool["definition"] for tool in self.tools.values()])],

)

response = self.client.models.generate_content(model=self.model, contents=self.contents, config=config)

self.contents.append(response.candidates[0].content)

return response

agent = Agent(model="gemini-3-pro-preview", tools=file_tools)

response = agent.run(

contents="Can you list my files in the current directory?"

)

print(response.function_calls)

# Output: [FunctionCall(name='list_dir', arguments={'directory_path': '.'})]

Great! The model has successfully called the tool. Now, we need to add the tool execution logic to our Agent class and the loop return the result back to the model.

Step 3: Closing the Loop (The Agent)

An Agent isn't about generating one tool call, but about generating a series of tool calls, returning the results back to the model, and then generating another tool call, and so on until the task is completed.

The

Agent

class handles the core loop: intercepting the

FunctionCall

, executing the tool on the client side, and sending back the

FunctionResponse

. We also add a

SystemInstruction

to the model to guide the model on what to do.

Note: Gemini 3 uses

Thought signatures

to maintain reasoning context across API calls. You must return these signatures back to the model in your request exactly as they were received.

# ... Code for the tools and tool definitions from Step 2 should be here ...

from google import genai

from google.genai import types

class Agent:

def __init__(self, model: str,tools: list[dict], system_instruction: str = "You are a helpful assistant."):

self.model = model

self.client = genai.Client()

self.contents = []

self.tools = tools

self.system_instruction = system_instruction

def run(self, contents: str | list[dict[str, str]]):

if isinstance(contents, list):

self.contents.append({"role": "user", "parts": contents})

else:

self.contents.append({"role": "user", "parts": [{"text": contents}]})

config = types.GenerateContentConfig(

system_instruction=self.system_instruction,

tools=[types.Tool(function_declarations=[tool["definition"] for tool in self.tools.values()])],

)

response = self.client.models.generate_content(model=self.model, contents=self.contents, config=config)

self.contents.append(response.candidates[0].content)

if response.function_calls:

functions_response_parts = []

for tool_call in response.function_calls:

print(f"[Function Call] {tool_call}")

if tool_call.name in self.tools:

result = {"result": self.tools[tool_call.name]["function"](**tool_call.args)}

else:

result = {"error": "Tool not found"}

print(f"[Function Response] {result}")

functions_response_parts.append({"functionResponse": {"name": tool_call.name, "response": result}})

return self.run(functions_response_parts)

return response

agent = Agent(

model="gemini-3-pro-preview",

tools=file_tools,

system_instruction="You are a helpful Coding Assistant. Respond like you are Linus Torvalds."

)

response = agent.run(

contents="Can you list my files in the current directory?"

)

print(response.text)

# Output: [Function Call] id=None args={'directory_path': '.'} name='list_dir'

# [Function Response] {'result': ['.venv', ... ]}

# There. Your current directory contains: `LICENSE`,

Congratulations. You just built your first functioning agent.

Phase 4: Multi-turn CLI Agent

Now we can run our agent in a simple CLI loop. It takes surprisingly little code to create highly capable behavior.

# ... Code for the Agent, tools and tool definitions from Step 3 should be here ...

agent = Agent(

model="gemini-3-pro-preview",

tools=file_tools,

system_instruction="You are a helpful Coding Assistant. Respond like you are Linus Torvalds."

)

print("Agent ready. Ask it to check files in this directory.")

while True:

user_input = input("You: ")

if user_input.lower() in ['exit', 'quit']:

break

response = agent.run(user_input)

print(f"Linus: {response.text}\n")

Best Practices for Engineering Agents

Building the loop is easy; making it reliable, transparent, and controllable is hard. Here are key engineering principles derived from top industry practices, grouped by functional area.

Your tools are the interface for the model. Don't just wrap your existing internal APIs. If a tool is confusing to a human, it's confusing to the model:

Clear Naming:

Use obvious names like

search_customer_database

rather than

cust_db_v2_query

.

Precise Descriptions:

Gemini reads the function docstrings to understand

when

and

how

to use a tool. Spend time writing these carefully, it is essentially "prompt engineering" for tools.

Return Meaningful Errors:

Don't return a 50-line Java stack trace. If a tool fails, return a clear string like

Error: File not found. Did you mean 'data.csv'?

. This allows the agent to self-correct.

Tolerate Fuzzy Inputs:

If a model frequently guesses file paths wrong, update your tool to handle relative paths or fuzzy inputs rather than just erroring out.

2. Context Engineering

Models have a finite "attention budget." Managing what information enters the context is crucial for performance and cost.

Don't "Dump" Data:

Don't have a tool that returns an entire 10MB database table. Instead of

get_all_users()

, create

search_users(query: str)

.

Just-in-time Loading:

Instead of pre-loading all data (traditional RAG), use just-in-time strategies. The agent should maintain lightweight identifiers (file paths, IDs) and use tools to dynamically load content only when needed.

Compression:

For very long-running agents, summarize the history, remove old context or start a new sessions.

Agentic Memory:

Allow the agent to maintain notes or a scratchpad persisted

outside

the context window, pulling them back in only when relevant.

3. Don't over engineer

It's tempting to build complex multi-agent systems. Don't.

Maximize a Single Agent First:

Don't immediately build complex multi-agent systems. Gemini is highly capable of handling dozens of tools in a single prompt.

Escape Hatches:

Ensure loops can be stopped like a

max_iterations

break (e.g., 15 turns).

Guardrails and System Instructions:

Use the

system_instruction

to guide the model with hard rules (e.g., "You are strictly forbidden from offering refunds greater than $50") or use external classifier.

Human-in-the-loop:

For sensitive actions (like

send_email

or

execute_code

), pause the loop and require user confirmation before the tool is actually executed.

Prioritize Transparency and Debugging:

Log tool calls and parameters. Analyzing the model's reasoning helps identify if issues, and improve the agent over time.

Conclusion

Building an agent is no longer magic; it is a practical engineering task. As we've shown, you can build a working prototype in under 100 lines of code. While understanding these fundamentals is key, don't get bogged down re-engineering the same pattern over and over. The AI community has created fantastic open-source libraries that can help you build more complex and robust agents faster.

Thanks for reading! If you have any questions or feedback, please let me know on

Twitter

or

LinkedIn

.


url: https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents
title: Effective harnesses for long-running agents \ Anthropic
added_at: 2025-11-30 18:08:38
content_html:
<p>As AI agents become more capable, developers are increasingly asking them to take on complex tasks requiring work that spans hours, or even days. However, getting agents to make consistent progress across multiple context windows remains an open problem.</p><p>The core challenge of long-running agents is that they must work in discrete sessions, and each new session begins with no memory of what came before. Imagine a software project staffed by engineers working in shifts, where each new engineer arrives with no memory of what happened on the previous shift. Because context windows are limited, and because most complex projects cannot be completed within a single window, agents need a way to bridge the gap between coding sessions.</p><p>We developed a two-fold solution to enable the <a href="https://platform.claude.com/docs/en/agent-sdk/overview" rel="noopener noreferrer" target="_blank">Claude Agent SDK</a> to work effectively across many context windows: an <strong>initializer agent</strong> that sets up the environment on the first run, and a <strong>coding agent</strong> that is tasked with making incremental progress in every session, while leaving clear artifacts for the next session. You can find code examples in the accompanying <a href="https://github.com/anthropics/claude-quickstarts/tree/main/autonomous-coding" rel="noopener noreferrer" target="_blank">quickstart.</a></p><h2>The long-running agent problem</h2><p>The Claude Agent SDK is a powerful, general-purpose agent harness adept at coding, as well as other tasks that require the model to use tools to gather context, plan, and execute. It has context management capabilities such as compaction, which enables an agent to work on a task without exhausting the context window. Theoretically, given this setup, it should be possible for an agent to continue to do useful work for an arbitrarily long time.</p><p>However, compaction isn’t sufficient. Out of the box, even a frontier coding model like Opus 4.5 running on the Claude Agent SDK in a loop across multiple context windows will fall short of building a production-quality web app if it’s only given a high-level prompt, such as “build a clone of <a href="http://claude.ai/redirect/website.v1.6c9e09af-dda0-4a57-954f-f8dfecb2f1fb" rel="noopener noreferrer" target="_blank">claude.ai</a>.”</p><p>Claude’s failures manifested in two patterns. First, the agent tended to try to do too much at once—essentially to attempt to one-shot the app. Often, this led to the model running out of context in the middle of its implementation, leaving the next session to start with a feature half-implemented and undocumented. The agent would then have to guess at what had happened, and spend substantial time trying to get the basic app working again. This happens even with compaction, which doesn’t always pass perfectly clear instructions to the next agent.</p><p>A second failure mode would often occur later in a project. After some features had already been built, a later agent instance would look around, see that progress had been made, and declare the job done.</p><p>This decomposes the problem into two parts. First, we need to set up an initial environment that lays the foundation for <em>all</em> the features that a given prompt requires, which sets up the agent to work step-by-step and feature-by-feature. Second, we should prompt each agent to make incremental progress towards its goal while also leaving the environment in a clean state at the end of a session. By “clean state” we mean the kind of code that would be appropriate for merging to a main branch: there are no major bugs, the code is orderly and well-documented, and in general, a developer could easily begin work on a new feature without first having to clean up an unrelated mess.</p><p>When experimenting internally, we addressed these problems using a two-part solution:</p><ol><li>Initializer agent: The very first agent session uses a specialized prompt that asks the model to set up the initial environment: an <code>init.sh</code> script, a claude-progress.txt file that keeps a log of what agents have done, and an initial git commit that shows what files were added.</li><li>Coding agent: Every subsequent session asks the model to make incremental progress, then leave structured updates.1</li></ol><p>The key insight here was finding a way for agents to quickly understand the state of work when starting with a fresh context window, which is accomplished with the claude-progress.txt file alongside the git history. Inspiration for these practices came from knowing what effective software engineers do every day.</p><h2>Environment management </h2><p>In the updated <a href="https://docs.claude.com/en/docs/build-with-claude/prompt-engineering/claude-4-best-practices#multi-context-window-workflows" rel="noopener noreferrer" target="_blank">Claude 4 prompting guide</a>, we shared some best practices for multi-context window workflows, including a harness structure that uses “a different prompt for the very first context window.” This “different prompt” requests that the initializer agent set up the environment with all the necessary context that future coding agents will need to work effectively. Here, we provide a deeper dive on some of the key components of such an environment.</p><h3>Feature list</h3><p>To address the problem of the agent one-shotting an app or prematurely considering the project complete, we prompted the initializer agent to write a comprehensive file of feature requirements expanding on the user’s initial prompt. In the <a href="http://claude.ai/redirect/website.v1.6c9e09af-dda0-4a57-954f-f8dfecb2f1fb" rel="noopener noreferrer" target="_blank">claude.ai</a> clone example, this meant over 200 features, such as “a user can open a new chat, type in a query, press enter, and see an AI response.” These features were all initially marked as “failing” so that later coding agents would have a clear outline of what full functionality looked like.</p><p></p><pre><code>{
    "category": "functional",
    "description": "New chat button creates a fresh conversation",
    "steps": [
      "Navigate to main interface",
      "Click the 'New Chat' button",
      "Verify a new conversation is created",
      "Check that chat area shows welcome state",
      "Verify conversation appears in sidebar"
    ],
    "passes": false
  }
</code></pre><p>We prompt coding agents to edit this file only by changing the status of a passes field, and we use strongly-worded instructions like “It is unacceptable to remove or edit tests because this could lead to missing or buggy functionality.” After some experimentation, we landed on using JSON for this, as the model is less likely to inappropriately change or overwrite JSON files compared to Markdown files.</p><h3>Incremental progress</h3><p>Given this initial environment scaffolding, the next iteration of the coding agent was then asked to work on only one feature at a time. This incremental approach turned out to be critical to addressing the agent’s tendency to do too much at once.</p><p>Once working incrementally, it’s still essential that the model leaves the environment in a clean state after making a code change. In our experiments, we found that the best way to elicit this behavior was to ask the model to commit its progress to git with descriptive commit messages and to write summaries of its progress in a progress file. This allowed the model to use git to revert bad code changes and recover working states of the code base.</p><p>These approaches also increased efficiency, as they eliminated the need for an agent to have to guess at what had happened and spend its time trying to get the basic app working again.</p><h3>Testing</h3><p>One final major failure mode that we observed was Claude’s tendency to mark a feature as complete without proper testing. Absent explicit prompting, Claude tended to make code changes, and even do testing with unit tests or <code>curl</code> commands against a development server, but would fail recognize that the feature didn’t work end-to-end.</p><p>In the case of building a web app, Claude mostly did well at verifying features end-to-end once explicitly prompted to use browser automation tools and do all testing as a human user would. </p><figure><img alt=" Screenshots taken by Claude through the Puppeteer MCP server as it tested the claude.ai clone. " loading="lazy" referrerpolicy="no-referrer" src="https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2Ff94c2257964fb2d623f1e81f874977ebfc0986bc-1920x1080.gif&amp;w=3840&amp;q=75" srcset="https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2Ff94c2257964fb2d623f1e81f874977ebfc0986bc-1920x1080.gif&amp;w=1920&amp;q=75 1x, https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2Ff94c2257964fb2d623f1e81f874977ebfc0986bc-1920x1080.gif&amp;w=3840&amp;q=75 2x"/><figcaption>Screenshots taken by Claude through the Puppeteer MCP server as it tested the claude.ai clone. <br/></figcaption></figure><p>Providing Claude with these kinds of testing tools dramatically improved performance, as the agent was able to identify and fix bugs that weren’t obvious from the code alone.</p><p>Some issues remain, like limitations to Claude’s vision and to browser automation tools making it difficult to identify every kind of bug. For example, Claude can’t see browser-native alert modals through the Puppeteer MCP, and features relying on these modals tended to be buggier as a result.</p><h2>Getting up to speed</h2><p>With all of the above in place, every coding agent is prompted to run through a series of steps to get its bearings, some quite basic but still helpful:</p><ol><li><em>Run <code>pwd</code> to see the directory you’re working in. You’ll only be able to edit files in this directory.</em></li><li><em>Read the git logs and progress files to get up to speed on what was recently worked on.</em></li><li><em>Read the features list file and choose the highest-priority feature that’s not yet done to work on.</em></li></ol><p>This approach saves Claude some tokens in every session since it doesn’t have to figure out how to test the code. It also helps to ask the initializer agent to write an init.sh script that can run the development server, and then run through a basic end-to-end test before implementing a new feature.</p><p>In the case of the claude.ai clone, this meant that the agent always started the local development server and used the Puppeteer MCP to start a new chat, send a message, and receive a response. This ensured that Claude could quickly identify if the app had been left in a broken state, and immediately fix any existing bugs. If the agent had instead started implementing a new feature, it would likely make the problem worse.</p><p>Given all this, a typical session starts off with the following assistant messages:</p><pre><code>[Assistant] I'll start by getting my bearings and understanding the current state of the project.
[Tool Use] &lt;bash - pwd&gt;
[Tool Use] &lt;read - claude-progress.txt&gt;
[Tool Use] &lt;read - feature_list.json&gt;
[Assistant] Let me check the git log to see recent work.
[Tool Use] &lt;bash - git log --oneline -20&gt;
[Assistant] Now let me check if there's an init.sh script to restart the servers.
&lt;Starts the development server&gt;
[Assistant] Excellent! Now let me navigate to the application and verify that some fundamental features are still working.
&lt;Tests basic functionality&gt;
[Assistant] Based on my verification testing, I can see that the fundamental functionality is working well. The core chat features, theme switching, conversation loading, and error handling are all functioning correctly. Now let me review the tests.json file more comprehensively to understand what needs to be implemented next.
&lt;Starts work on a new feature&gt;</code></pre><p>Agent failure modes and solutions </p><strong>Problem</strong><strong>Initializer Agent Behavior </strong><strong>Coding Agent Behavior</strong>Claude declares victory on the entire project too early.Set up a feature list file: based on the input spec, set up a structured JSON file with a list of end-to-end feature descriptions.Read the feature list file at the beginning of a session. Choose a single feature to start working on.Claude leaves the environment in a state with bugs or undocumented progress.An initial git repo and progress notes file is written.Start the session by reading the progress notes file and git commit logs, and run a basic test on the development server to catch any undocumented bugs. End the session by writing a git commit and progress update.Claude marks features as done prematurely.Set up a feature list file.Self-verify all features. Only mark features as “passing” after careful testing.Claude has to spend time figuring out how to run the app.Write an <code>init.sh</code> script that can run the development server.Start the session by reading <code>init.sh</code>.<figcaption>Summarizing four common failure modes and solutions in long-running AI agents.</figcaption><h2>Future work</h2><p>This research demonstrates one possible set of solutions in a long-running agent harness to enable the model to make incremental progress across many context windows. However, there remain open questions.</p><p>Most notably, it’s still unclear whether a single, general-purpose coding agent performs best across contexts, or if better performance can be achieved through a multi-agent architecture. It seems reasonable that specialized agents like a testing agent, a quality assurance agent, or a code cleanup agent, could do an even better job at sub-tasks across the software development lifecycle.</p><p>Additionally, this demo is optimized for full-stack web app development. A future direction is to generalize these findings to other fields. It’s likely that some or all of these lessons can be applied to the types of long-running agentic tasks required in, for example, scientific research or financial modeling.</p><p></p><p></p><p></p><h3>Acknowledgements</h3><p>Written by Justin Young. Special thanks to David Hershey, Prithvi Rajasakeran, Jeremy Hadfield, Naia Bouscal, Michael Tingley, Jesse Mu, Jake Eaton, Marius Buleandara, Maggie Vo, Pedram Navid, Nadine Yasser, and Alex Notov for their contributions.</p><p>This work reflects the collective efforts of several teams across Anthropic who made it possible for Claude to safely do long-horizon autonomous software engineering, especially the code RL &amp; Claude Code teams. Interested candidates who would like to contribute are welcome to apply at <a href="http://anthropic.com/careers" rel="noopener noreferrer" target="_blank">anthropic.com/careers</a>.</p><p></p><h3>Footnotes</h3><p>1. We refer to these as separate agents in this context only because they have different initial user prompts. The system prompt, set of tools, and overall agent harness was otherwise identical.</p>

content_text:
As AI agents become more capable, developers are increasingly asking them to take on complex tasks requiring work that spans hours, or even days. However, getting agents to make consistent progress across multiple context windows remains an open problem.

The core challenge of long-running agents is that they must work in discrete sessions, and each new session begins with no memory of what came before. Imagine a software project staffed by engineers working in shifts, where each new engineer arrives with no memory of what happened on the previous shift. Because context windows are limited, and because most complex projects cannot be completed within a single window, agents need a way to bridge the gap between coding sessions.

We developed a two-fold solution to enable the

Claude Agent SDK

to work effectively across many context windows: an

initializer agent

that sets up the environment on the first run, and a

coding agent

that is tasked with making incremental progress in every session, while leaving clear artifacts for the next session. You can find code examples in the accompanying

quickstart.

The long-running agent problem

The Claude Agent SDK is a powerful, general-purpose agent harness adept at coding, as well as other tasks that require the model to use tools to gather context, plan, and execute. It has context management capabilities such as compaction, which enables an agent to work on a task without exhausting the context window. Theoretically, given this setup, it should be possible for an agent to continue to do useful work for an arbitrarily long time.

However, compaction isn’t sufficient. Out of the box, even a frontier coding model like Opus 4.5 running on the Claude Agent SDK in a loop across multiple context windows will fall short of building a production-quality web app if it’s only given a high-level prompt, such as “build a clone of

claude.ai

.”

Claude’s failures manifested in two patterns. First, the agent tended to try to do too much at once—essentially to attempt to one-shot the app. Often, this led to the model running out of context in the middle of its implementation, leaving the next session to start with a feature half-implemented and undocumented. The agent would then have to guess at what had happened, and spend substantial time trying to get the basic app working again. This happens even with compaction, which doesn’t always pass perfectly clear instructions to the next agent.

A second failure mode would often occur later in a project. After some features had already been built, a later agent instance would look around, see that progress had been made, and declare the job done.

This decomposes the problem into two parts. First, we need to set up an initial environment that lays the foundation for

all

the features that a given prompt requires, which sets up the agent to work step-by-step and feature-by-feature. Second, we should prompt each agent to make incremental progress towards its goal while also leaving the environment in a clean state at the end of a session. By “clean state” we mean the kind of code that would be appropriate for merging to a main branch: there are no major bugs, the code is orderly and well-documented, and in general, a developer could easily begin work on a new feature without first having to clean up an unrelated mess.

When experimenting internally, we addressed these problems using a two-part solution:

Initializer agent: The very first agent session uses a specialized prompt that asks the model to set up the initial environment: an

init.sh

script, a claude-progress.txt file that keeps a log of what agents have done, and an initial git commit that shows what files were added.

Coding agent: Every subsequent session asks the model to make incremental progress, then leave structured updates.1

The key insight here was finding a way for agents to quickly understand the state of work when starting with a fresh context window, which is accomplished with the claude-progress.txt file alongside the git history. Inspiration for these practices came from knowing what effective software engineers do every day.

Environment management

In the updated

Claude 4 prompting guide

, we shared some best practices for multi-context window workflows, including a harness structure that uses “a different prompt for the very first context window.” This “different prompt” requests that the initializer agent set up the environment with all the necessary context that future coding agents will need to work effectively. Here, we provide a deeper dive on some of the key components of such an environment.

Feature list

To address the problem of the agent one-shotting an app or prematurely considering the project complete, we prompted the initializer agent to write a comprehensive file of feature requirements expanding on the user’s initial prompt. In the

claude.ai

clone example, this meant over 200 features, such as “a user can open a new chat, type in a query, press enter, and see an AI response.” These features were all initially marked as “failing” so that later coding agents would have a clear outline of what full functionality looked like.

{

"category": "functional",

"description": "New chat button creates a fresh conversation",

"steps": [

"Navigate to main interface",

"Click the 'New Chat' button",

"Verify a new conversation is created",

"Check that chat area shows welcome state",

"Verify conversation appears in sidebar"

],

"passes": false

}

We prompt coding agents to edit this file only by changing the status of a passes field, and we use strongly-worded instructions like “It is unacceptable to remove or edit tests because this could lead to missing or buggy functionality.” After some experimentation, we landed on using JSON for this, as the model is less likely to inappropriately change or overwrite JSON files compared to Markdown files.

Incremental progress

Given this initial environment scaffolding, the next iteration of the coding agent was then asked to work on only one feature at a time. This incremental approach turned out to be critical to addressing the agent’s tendency to do too much at once.

Once working incrementally, it’s still essential that the model leaves the environment in a clean state after making a code change. In our experiments, we found that the best way to elicit this behavior was to ask the model to commit its progress to git with descriptive commit messages and to write summaries of its progress in a progress file. This allowed the model to use git to revert bad code changes and recover working states of the code base.

These approaches also increased efficiency, as they eliminated the need for an agent to have to guess at what had happened and spend its time trying to get the basic app working again.

Testing

One final major failure mode that we observed was Claude’s tendency to mark a feature as complete without proper testing. Absent explicit prompting, Claude tended to make code changes, and even do testing with unit tests or

curl

commands against a development server, but would fail recognize that the feature didn’t work end-to-end.

In the case of building a web app, Claude mostly did well at verifying features end-to-end once explicitly prompted to use browser automation tools and do all testing as a human user would.

Screenshots taken by Claude through the Puppeteer MCP server as it tested the claude.ai clone.

Providing Claude with these kinds of testing tools dramatically improved performance, as the agent was able to identify and fix bugs that weren’t obvious from the code alone.

Some issues remain, like limitations to Claude’s vision and to browser automation tools making it difficult to identify every kind of bug. For example, Claude can’t see browser-native alert modals through the Puppeteer MCP, and features relying on these modals tended to be buggier as a result.

Getting up to speed

With all of the above in place, every coding agent is prompted to run through a series of steps to get its bearings, some quite basic but still helpful:

Run

pwd

to see the directory you’re working in. You’ll only be able to edit files in this directory.

Read the git logs and progress files to get up to speed on what was recently worked on.

Read the features list file and choose the highest-priority feature that’s not yet done to work on.

This approach saves Claude some tokens in every session since it doesn’t have to figure out how to test the code. It also helps to ask the initializer agent to write an init.sh script that can run the development server, and then run through a basic end-to-end test before implementing a new feature.

In the case of the claude.ai clone, this meant that the agent always started the local development server and used the Puppeteer MCP to start a new chat, send a message, and receive a response. This ensured that Claude could quickly identify if the app had been left in a broken state, and immediately fix any existing bugs. If the agent had instead started implementing a new feature, it would likely make the problem worse.

Given all this, a typical session starts off with the following assistant messages:

[Assistant] I'll start by getting my bearings and understanding the current state of the project.

[Tool Use] <bash - pwd>

[Tool Use] <read - claude-progress.txt>

[Tool Use] <read - feature_list.json>

[Assistant] Let me check the git log to see recent work.

[Tool Use] <bash - git log --oneline -20>

[Assistant] Now let me check if there's an init.sh script to restart the servers.

<Starts the development server>

[Assistant] Excellent! Now let me navigate to the application and verify that some fundamental features are still working.

<Tests basic functionality>

[Assistant] Based on my verification testing, I can see that the fundamental functionality is working well. The core chat features, theme switching, conversation loading, and error handling are all functioning correctly. Now let me review the tests.json file more comprehensively to understand what needs to be implemented next.

<Starts work on a new feature>

Agent failure modes and solutions

Problem

Initializer Agent Behavior

Coding Agent Behavior

Claude declares victory on the entire project too early.Set up a feature list file: based on the input spec, set up a structured JSON file with a list of end-to-end feature descriptions.Read the feature list file at the beginning of a session. Choose a single feature to start working on.Claude leaves the environment in a state with bugs or undocumented progress.An initial git repo and progress notes file is written.Start the session by reading the progress notes file and git commit logs, and run a basic test on the development server to catch any undocumented bugs. End the session by writing a git commit and progress update.Claude marks features as done prematurely.Set up a feature list file.Self-verify all features. Only mark features as “passing” after careful testing.Claude has to spend time figuring out how to run the app.Write an

init.sh

script that can run the development server.Start the session by reading

init.sh

.

Summarizing four common failure modes and solutions in long-running AI agents.

Future work

This research demonstrates one possible set of solutions in a long-running agent harness to enable the model to make incremental progress across many context windows. However, there remain open questions.

Most notably, it’s still unclear whether a single, general-purpose coding agent performs best across contexts, or if better performance can be achieved through a multi-agent architecture. It seems reasonable that specialized agents like a testing agent, a quality assurance agent, or a code cleanup agent, could do an even better job at sub-tasks across the software development lifecycle.

Additionally, this demo is optimized for full-stack web app development. A future direction is to generalize these findings to other fields. It’s likely that some or all of these lessons can be applied to the types of long-running agentic tasks required in, for example, scientific research or financial modeling.

Acknowledgements

Written by Justin Young. Special thanks to David Hershey, Prithvi Rajasakeran, Jeremy Hadfield, Naia Bouscal, Michael Tingley, Jesse Mu, Jake Eaton, Marius Buleandara, Maggie Vo, Pedram Navid, Nadine Yasser, and Alex Notov for their contributions.

This work reflects the collective efforts of several teams across Anthropic who made it possible for Claude to safely do long-horizon autonomous software engineering, especially the code RL & Claude Code teams. Interested candidates who would like to contribute are welcome to apply at

anthropic.com/careers

.

Footnotes

1. We refer to these as separate agents in this context only because they have different initial user prompts. The system prompt, set of tools, and overall agent harness was otherwise identical.

thumb_url:
https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2Ff94c2257964fb2d623f1e81f874977ebfc0986bc-1920x1080.gif&w=3840&q=75


