Running a Local LLM in Production with Ollama: Building an AI Listing Coach

On a TÜBİTAK 2209-A funded freelancer marketplace project, a real problem kept showing up: users didn't know what to write when creating a listing, and a meaningful share of people faced with a blank text box never finished the listing at all. The fix was an "AI listing coach" that offers suggestions. But for a budget-constrained venture, the per-call cost of the OpenAI API — especially with every draft revision triggering another call — wasn't sustainable. So I turned to a locally running LLM instead: Ollama running Qwen Coder.

Why "local" had to mean genuinely local

Running a local LLM didn't just mean a demo on a dev machine — it meant a service running 24/7 on the production server. That brought three concrete constraints:

1. Memory management: should the model stay loaded in server memory permanently, or load on demand per request? Keeping it loaded is faster but competes for memory with other services. When I measured it on the test server, the first load took ~4 seconds — not acceptable as user-facing latency. The fix was using Ollama's keep_alive parameter to keep the model in memory for a fixed window (30 minutes) and release it when idle.
2. Concurrency limits: when multiple users hit the endpoint at the same time, sharing a single CPU/GPU resource, Ollama queues the requests but response time grows linearly. I capped this at the API layer with a SemaphoreSlim(maxConcurrency: 2), showing the user a "generating your suggestion" state once the queue filled up.
3. Timeouts and graceful fallback: the model would occasionally take upwards of 15 seconds to respond. To avoid breaking the user experience, I set a 10-second timeout; if it expired, the user saw "couldn't generate a suggestion right now, you can finish your listing manually" instead of a blocked flow — AI should be a helper, never a hard requirement.

public async Task<string?> GenerateSuggestionAsync(string draftText, CancellationToken ct)
{
    using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
    cts.CancelAfter(TimeSpan.FromSeconds(10));
 
    await _concurrencyLimiter.WaitAsync(cts.Token);
    try
    {
        var response = await _ollamaClient.GenerateAsync(new()
        {
            Model = "qwen2.5-coder:7b",
            Prompt = BuildPrompt(draftText),
            KeepAlive = "30m",
        }, cts.Token);
 
        return response?.Text;
    }
    catch (OperationCanceledException)
    {
        return null; // Fallback: never block the user's flow.
    }
    finally
    {
        _concurrencyLimiter.Release();
    }
}

Prompt engineering: short and tightly constrained

A generic "write me a listing" prompt led the model to produce irrelevant or overly long text. What actually worked was constraining the prompt hard: a maximum word count, a target tone (professional but friendly), and an example format. I also ran the model's output through a simple length/profanity filter before showing it to the user — local model or not, there's no reason to skip the safety check.

Why the chat bot shares the same infrastructure

The listing coach and the platform's in-app Q&A chat bot share the same Ollama instance, differentiated only by their prompt templates. This avoided the extra memory/CPU cost of running a second model instance — one model, usable through multiple "personas."

Result

The hard part of putting a local LLM into production isn't running the model — it's making it play nicely with every other production constraint: memory, concurrency, timeouts, and a safe fallback path. You can find the rest of this project's production decisions — CSRF enforcement, retry policies — on the project page.