Open-Source LLMs for Text Analysis

Open-Source LLMs for Text Analysis#

This tutorial covers running open-source LLMs on KLC for text analysis research. Open-source models are preferable when your data is sensitive, when you need full reproducibility, or when cost at scale is a concern.

For a broader overview of why and when to use open-source models, see the Open Source LLMs guide.

Using Ollama on KLC#

Ollama provides a simple server/client interface for running open-source LLMs. On KLC, start the server with a shared launcher script.

Start the Server#

After logging onto KLC, start the server in a tmux session (the script runs in the foreground):

/kellogg/software/Modules/modulefiles/klc-main-ollama/bin/start_ollama_server.sh

When the server is ready, the script prints the node hostname and port — use those values for API calls below. For the full ready-banner reference, environment variables, and the GPU (sbatch) path, see Open Source LLMs on KLC.

Connect with Python#

module purge
module load mamba/24.3.0
eval "$('/hpc/software/mamba/24.3.0/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
source "/hpc/software/mamba/24.3.0/etc/profile.d/mamba.sh"
mamba create --prefix=./env python=3.13
mamba activate ./env
python -m pip install ollama

from ollama import Client

HOST = "localhost"  # same node as the server
PORT = 8234         # from the ready banner

client = Client(host=f"http://{HOST}:{PORT}")

Browse additional models at ollama.com/search .

Running a Prompt#

import logging

from ollama import Client

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
logger = logging.getLogger(__name__)


def ensure_model_available(client, model):
    available = {m.model for m in client.list().models}
    normalized = model if ":" in model else f"{model}:latest"
    if model in available or normalized in available:
        logger.info("Model already available: %s", model)
        return

    logger.info("Model not found locally: %s — pulling ...", model)
    last_status = None
    for progress in client.pull(model, stream=True):
        if progress.status != last_status:
            logger.info("pull %s: %s", model, progress.status)
            last_status = progress.status
    logger.info("Model pulled: %s", model)


HOST = "localhost"  # same node as the server
PORT = 8234         # from the ready banner

client = Client(host=f"http://{HOST}:{PORT}")
ensure_model_available(client, "llama3.2")

response = client.chat(
    model="llama3.2",
    messages=[
        {"role": "system", "content": "You are a helpful research assistant."},
        {"role": "user",   "content": "Summarize the key risks in this 10-K excerpt: ..."},
    ],
    options={"temperature": 0, "seed": 42},
)
print(response["message"]["content"])

Reproducibility Settings#

Open-source models give you full control over reproducibility:

Setting

What It Does

temperature=0

Deterministic output

seed=42

Fixes random state (where supported)

Model version pinning

Use model:tag (e.g., llama3.2:3b) to freeze the model version

Always record the model name and tag in your logs. Unlike API providers, you can ensure the exact same model weights are used across runs.

For the broader cost, privacy, and performance comparison between open-source and paid APIs, see Open Source LLMs.

Example: Batch Classification#

from ollama import Client

HOST = "localhost"  # same node as the server
PORT = 8234         # from the ready banner

client = Client(host=f"http://{HOST}:{PORT}")

def classify_text(text: str, model: str = "llama3.2") -> str:
    response = client.chat(
        model=model,
        messages=[
            {
                "role": "system",
                "content": (
                    "Classify the sentiment of the following text as one of "
                    "[Positive, Negative, Neutral]. Return only the label."
                ),
            },
            {"role": "user", "content": text},
        ],
        options={"temperature": 0, "seed": 42},
    )
    return response["message"]["content"].strip()

# Batch process a list of documents
documents = ["Revenue exceeded expectations.", "Significant litigation risk.", "Operations were stable."]
results   = [(doc, classify_text(doc)) for doc in documents]

for doc, label in results:
    print(f"{label:10} | {doc}")

Using Quest OnDemand#

An alternative to the command line: open the Quest OnDemand Jupyter interactive app and select “ollama” as the pre-installed kernel. This automatically starts the Ollama server alongside JupyterHub.

Next Steps#