Tema

LLMs

3009documentos

253últimos 7 días

Who Is the Influencer? Consensus Dynamics in Multi-Agent Systems

Multi-agent LLM systems are increasingly used in practice, but their behavior is complex and hard to understand. We design a simple task to ask which model has the most influence on the final consensus when several models are mixed in one system. Experimentally, across seven model families, we find that the opinions of the other models are absorbed into the opinion of the stronger model, the one with more parameters. This happens even though the task requires no pretraining knowledge and has no correct answer. To understand this phenomenon, we interpret the agents as weighted particles moving on a simplex. Larger models carry a larger weight and therefore occupy a deeper well in the energy landscape. A large weight means strong in-context learning: a model with a large weight amplifies the majority of its context. Larger models are generally better at in-context learning, and this is what makes the larger model the influencer. We hope that this work helps to interpret multi-agent systems, whose behavior is becoming too complex for humans to understand.

arXiv AI Papershace 22 días

MQAScore: Fine-Grained Text-Music Alignment Evaluation via Large Audio Language Model

Text-to-music alignment is commonly evaluated by the global similarity between text and audio embeddings, which collapses all described attributes into a single score and obscures which ones are missing. We instead introduce MQAScore, which verifies each atomic musical concept along five clip-level dimensions (genre, instrument, mood/theme, vocal, and usage) by the probability that a large audio language model (LALM) answers affirmatively to a binary question. To assess evaluator reliability, we construct verification labels from presence/absence annotations and LLM-guided concept substitutions, and propose the Sensitivity-Invariance Ratio, which measures how selectively an evaluator responds to meaning-changing substitutions. Experiments show that concept-level scoring improves both question-answering and similarity-based evaluators, and MQAScore can be a more reliable and interpretable fine-grained complement to global similarity for evaluating text-to-music alignment.

arXiv AI Papershace 22 días

SCOPE: Compiling Natural-Language Policies into Neurosymbolic Predicate Programs

Policy enforcement for LLM agents has shifted from single-turn content moderation to compound, application-specific policies evaluated over multi-turn transcripts and tool-call trajectories. Fine-tuned guardrail models are efficient but degrade on compound policies, and while LLM-as-a-judge approaches achieve higher accuracy, they incur high inference cost and tend to produce unstable predictions. Neurosymbolic methods offer a natural alternative, decomposing the policy into narrow neural decisions composed under explicit logic. However, existing symbolic layers lack the required expressiveness and are optimized against fixed data rather than the policy itself. To address this, we introduce SCOPE, which compiles a policy into an executable program of neural predicates executed by small models, deterministic extractors, and code that composes their outputs. The program is optimized jointly with its training data, using adversarial generation of counterexamples that target where the program deviates from the policy. We evaluate SCOPE on a benchmark of real-world policy tasks against prompt optimization, task decomposition, and workflow generation methods, achieving the highest average accuracy across tasks and models. Programs optimized with the weakest executor transfer to other models without further optimization. This allows them to be deployed in production on inexpensive models, where they reach accuracy comparable to frontier reasoning models. Unlike these monolithic models, SCOPE produces substantially more stable predictions, and each verdict is interpretable through the predicate outputs that produced it. Additionally, we validate the key components of our method through an ablation study. SCOPE thus enables reliable, interpretable, and efficient policy enforcement at production scale.

arXiv AI Papershace 22 días

Diagnosing the Explicit-to-Applied Theory of Mind Gap in LLMs

As large language models are deployed in high-stakes roles such as therapy, education, and personal assistance, their social cognition must be assessed reliably. Current Theory of Mind (ToM) benchmarks report aggregate scores that conflate two distinct competencies: explicit ToM (recognizing what someone believes) and applied ToM (using that belief to predict behavior). Building on this distinction, we investigate the gap and its recoverability using four structurally diverse benchmarks. Aggregate reporting obscures uneven competence: on SimpleToM, Llama-3.1-8B scores 51.5 percent overall while recognizing mental states at 99.4 percent and predicting behavior at only 28.9 percent. To assess recovery, we compare structured prompting, matched in-context learning, and our proposed Belief-State Test-Time Training (BSTTT), which adapts lightweight parameters on answered support questions through action reconstruction. Across Qwen2.5-7B, Llama-3.1-8B, and GPT-oss-20B, recovery depends on the model, task, and intervention. On SimpleToM, reminders combined with BSTTT improve Llama's behavior prediction by 40.8 percentage points; reminders account for most of the overall improvement, with BSTTT adding 1.3 to 5.5 points beyond reminders across models. Contextual examples often outperform weight updates, while BSTTT remains stronger in specific model-benchmark settings. Temporal belief tracking remains difficult despite partial recovery. These findings show that aggregate ToM scores can overstate applied competence while hiding performance recoverable through targeted assistance. We recommend decomposed reporting of explicit and applied accuracy alongside intervention comparisons that establish which failures are recoverable and under what conditions.

arXiv AI Papershace 22 días

Directing to Diversify: Multi-Directional Search for Automatic Heuristic Design

Large language models (LLMs) have demonstrated strong potential to automate heuristic design for NP-hard combinatorial optimization (CO) tasks without human intervention. Recent methods predominantly rely on evolutionary computation (EC) frameworks, where heuristic diversification is mainly induced through EC-like operators or downstream selection, while the direction of heuristic transformation remains largely unharnessed. In this paper, we propose a multi-directional Perturbation-Rewrite Search (PRS) framework for automatic heuristic design that factorizes heuristic transformation through directional search primitives at the generation level. Specifically, PRS operates on heuristic ideas through three complementary directions: expansion toward richer and more expressive designs, compression toward simpler and less specialized designs, and steering for fine-grained refinement, together enabling broader and more sustained exploration of diverse regions in the heuristic design space. PRS further organizes candidate exploration within a breadth-first search-based process with adaptive operator control, which ensures thorough candidate utilization and sustained exploration, and maintains strong search dynamics while preventing premature convergence. Extensive empirical results across diverse CO tasks show that PRS consistently discovers high-quality heuristics, demonstrating the effectiveness and generality of explicitly directional exploration design.

Hacker News (AI)hace 22 días

Achieving AI-style reasoning without an LLM

en▲ 7

NVIDIA AI Bloghace 22 días

NVIDIA Open Models Power a New Wave of European Public Sector Projects

Governments need modern solutions to serve the public while keeping sensitive information under their control. Open models provide a path forward, letting governments evaluate them, adapt them to local languages and public services, run them on infrastructure they control, apply their own safeguards and keep sensitive data within controlled environments. Some open models go further, […]

RunPod Bloghace 22 días

How to Run vLLM on Runpod Serverless (Beginner-Friendly Guide)

Learn how to run vLLM on Runpod’s serverless GPU platform. This guide walks you through fast, efficient LLM inference without complex setup.

RunPod Bloghace 22 días

NVIDIA's Llama 3.1 Nemotron 70B: Can It Solve Your LLM Bottlenecks?

Nemotron 70B is NVIDIA's latest open model and it's climbing the leaderboards. But how does it perform in the real world, and can it solve your toughest.

RunPod Bloghace 22 días

RAG vs. Fine-Tuning: Which Is Best for Your LLM?

Retrieval-Augmented Generation (RAG) and fine-tuning are powerful ways to adapt large language models. Learn the key differences, trade-offs, and when to.

RunPod Bloghace 22 días

Classifier-Free Guidance in LLMs: How It Works

Classifier-Free Guidance improves LLM output quality and control. Here’s how it works, where it came from, and why it matters for your AI generations

RunPod Bloghace 22 días

How to Work with GGUF Quantizations in KoboldCPP

GGUF quantizations make large language models faster and more efficient. This guide walks you through using KoboldCPP to load, run, and manage quantized.

RunPod Bloghace 22 días

Open Source Video & LLM Roundup: The Best of What’s New

Open-source AI is booming, and 2024 delivered an incredible wave of new LLMs and generative video models. Here's a quick roundup of the most exciting.

RunPod Bloghace 22 días

Run Very Large LLMs Securely with Runpod Serverless

Deploy large language models like LLaMA or Mixtral on Runpod Serverless with strong privacy controls and no infrastructure headaches. Here’s how.

RunPod Bloghace 22 días

Supercharge Your LLMs with SGLang: Boost Performance and Customization

Discover how to boost your LLM inference performance and customize responses using SGLang, an innovative framework for structured LLM workflows.

RunPod Bloghace 22 días

The Complete Guide to GPU Requirements for LLM Fine-Tuning

Fine-tuning large language models can require hours or days of runtime. This guide walks through how to choose the right GPU spec for cost and performance.

RunPod Bloghace 22 días

Why LLMs Can't Spell 'Strawberry' And Other Odd Use Cases

Large language models can write poetry and solve logic puzzles, but fail at tasks like counting letters or doing math. Here's why, and what it tells us.

Hacker News (AI)hace 22 días

LLM Agents Play Pokemon Showdown

en▲ 6

AWS Machine Learning Bloghace 22 días

5 production challenges in multi-agent systems and how Deep Insight solved them

Many teams build a multi-agent AI prototype in days but hit a wall in production. This post defines five production challenges (orchestration, per-agent model selection, reliable operation, secure code execution, and monitoring) and shows how Deep Insight, an open source multi-agent system on AWS, solves each.

AWS Machine Learning Bloghace 22 días

Adaptive agentic intelligence framework using Strands Agents

The Adaptive Agentic Intelligence Framework (AAIF), built with Strands Agents, handles routine tasks locally with small language models and routes complex requests to cloud models through Amazon Bedrock. This post walks through building AAIF with local SLMs through Ollama and Amazon Bedrock, using an Internet of Things monitoring scenario.