Evaluating Small LLMs on Linux Troubleshooting Tasks with DeepEval

Introduction

This work evaluates large language models on Linux troubleshooting tasks for IT operators, using the DeepEval framework with the G-Eval metric to assess the correspondence between a given issue (input) and the model’s proposed solution (actual output). The evaluation targets small and cost-efficient models to identify which is best suited for this task. Building on prior work studying the integration of LLMs into infrastructure operations tasks, this study focuses specifically on error diagnosis and troubleshooting in Linux environments.

Can AI judge AI? : designing and building an automated LLM evaluation platform for enterprise generative AI systems

The thesis identifies that most quality evaluations in enterprise settings remain manual, subjective, and inconsistent, and while human evaluation is valuable and often treated as the most trusted form of assessment, it’s slow (2020). The thesis evaluate a protocol-driven LLM evaluation microservice that automatically scores RAG-enabled LLM outputs against standardized metrics as a non-blocking, event-driven (Kafka-native) downstream observer. Concretely, the work pursues three goals:

Design a modular, protocol-driven microservice architecture for automated evaluation.
Implement it as a working artifact integrating DeepEval metrics with a configurable LLM judge model.
Evaluate the artifact against design requirements metric behavior, evaluation consistency, and architectural properties like idempotency and portability.

It uses 2 metrics:

Built-in DeepEval metrics Faithfulness, Answer Relevancy, and Contextual Relevancy chosen to align with multi-dimensional evaluation approaches recommended in recent factuality/evaluation surveys.
Custom rubric-based metrics (via DeepEval’s GEval) Completeness (whether the output addresses all explicit requirements in the input) and Informativeness (whether the output is specific and useful rather than vague).

OpsEval: A Comprehensive Benchmark Suite

The work is a benchmark for evaluating LLM capabilities in IT operations (AIOps). IT operations tasks demand specialized technical knowledge, domain-specific terminology, and often involve proprietary or private information. The benchmark comprises 9,070 questions spanning 9 sub-domains, 8 tasks, and 3 core abilities. The question set is split between 7,334 multiple-choice questions and 1,736 open-ended QA items, with 20% of the total dataset created by domain experts and released publicly. Thay also evaluate 24 state-of-the-art LLMs under several distinct prompting techniques: Chain-of-thought (CoT), Self-consistency, In-context learning. The evaluation metric is FAE-Score designed to grade open-ended QA responses. The authors report that FAE-Score aligns substantially better with human expert judgments than traditional overlap-based metrics.

Accurate and Interpretable Log-Based Fault Diagnosis Using Large Language Models

Fault diagnosis typically requires reasoning over extensive log sequences, but LLMs’ context windows are too short to ingest all relevant logs at once.

The authors propose LogInsight, which combines two components:

Fine-tuning a medium-sized, open-source LLM to inject domain-specific fault-diagnosis knowledge rather than relying on a general-purpose model out-of-the-box.
A Fault-Oriented Log Summary (FOLS) module, which extracts the critical information from each log sequence before it reaches the LLM, mitigating the context-length bottleneck without requiring the model to process raw, full-length logs.

LogInsight is evaluated on three datasets two public and one production dataset, outperforming state-of-the-art baselines by 36.9%, 12.8%, and 7.3% on the respective datasets.

Methodology

LLM Test Cases

An LLMTestCase in DeepEval is used to unit test interactions within an LLM application, including RAG and LLM agent use cases (for individual components, nested agents, or the agent as a whole). It contains the information needed to evaluate the application for a given input (e.g., tools_called for agents, retrieval_context for RAG).

An LLMTestCase supports two evaluation levels:

End-to-end: represents the input and output of the “black-box” LLM application.
Component-level: multiple LLMTestCase instances represent multiple interactions across different components.

Creating an LLMTestCase requires specifying:

input
actual_output

Additional parameters, such as expected_output and context, are required only when the evaluation criteria depend on them. In the evaluation Parameters:

1

evaluation_params = [SingleTurnParams.INPUT, SingleTurnParams.ACTUAL_OUTPUT]

Each value corresponds to a component of the test case:

SingleTurnParams.INPUT: The original question or issue given to the model (e.g., “How do I install htop?”). SingleTurnParams.ACTUAL_OUTPUT: The response generated by the model being evaluated

Evaluation Dataset

Linux Troubleshooting. 2 basic steps apply to nearly any issue, following a systematic, methodical approach that tests one possibility at a time:

Gather the component and be concise. Avoid vague messages such as “the server is broken”; instead, be specific, e.g., “the web server returned a 503 on /api/users.”
Filter log files rather than reading them in full. For large log files, search them, e.g., grep -i error /var/log/syslog.

The evaluation dataset is structured as a CSV file with four fields: question, answer, context, and model. The question field contains the Linux troubleshooting query posed to the model. The answer field contains the model’s generated response, including any command-line instructions. The context field specifies the relevant system environment for the query. The context was used to obtain the answer field. The model field identifies which LLM generated the response.

4. Models

Model	Access	Input	Output
qwen/qwen-2.5-7b-instruct	API	0.04$	0.1$
meta-llama/llama-3.1-8b-instruct	API	0.02$	0.03$
meta-llama/llama-3.2-1b-instruct	API	0.027$	0.201$
meta-llama/llama-4-scout	API	0.1$	0.2$

Results

Result

The model with the smallest paramaters performed worst in Analytical Thinking. The Smallest model struggled to infer logical relationships in short, open-ended questions.

Within Meta’s model family (e.g., Llama 4), “Instruct” denotes the general-purpose, conversational fine-tuned version, while “Scout” is designed specifically for processing large documents. Scout performed well with 3 failed cases a small differences relative to the Instruct model with 5 failed cases, despite having a smaller size.

Qwen showed the best performance on Linux error scenarios at a size of 7B parameters with all 2 failed cases.

Conclusion

This work evaluated four LLMs on Linux troubleshooting sceneario questions using DeepEval’s G-Eval metric. Results show that model size alone does not determine performance in meta models, while the smallest model tested performed worst on Analytical Thinking, struggling to infer logical relationships in short, open-ended questions. Among the evaluated models, qwen-2.5-7b-instruct showed the best overall performance on Linux error diagnosis scenarios, suggesting it is a suitable candidate for cost-efficient deployment in IT operations support tasks.