Plurals: A System for Guiding LLMs via Simulated Social Ensembles

Plurals consist of Agents (LLMs, optionally with personas) which deliberate within customizable Structures, with Moderators overseeing deliberation.

Structures govern what information is shared between agents. One can use customizable existing structures (e.g. debates, graphs, ensembles, chains) or create their own.

Agents:

• Profile: who the agent is modeling
• Task: what the prompt Agents are responding to
• Combination instructions: first-wave and second-wave deliberation ideas, critique and revise, and NY’s juror deliberation instructions
• Knowledge: Agents differ in knowledge they have. They rely on the ability to use different models to leverage distinct knowledge.

Structures:

• Amount of information shared: Chains, Debates, and DAGs have a parameter called last_n that controls how many prior responses each Agent can see. For DAGs, the density of the network can be thought of as the amount of information shared.
  • Chain: sequence of agents arranged in customizable order, with the option to shuffle the order on each cycle
  • Graph: directed acyclic graph of agents where users provide agents and edges, enabling deliberation to proceed through the graph where ($A\to B$) implies B will see A’s responses
  • Debate: exactly two agents engaging in back-and-forth discussion
  • Ensemble: list of agents processing tasks in parallel Which can be used to augment a scenario where agents critique and revice a company memo.

Moderators:

• Profile: what kind of moderator the LLM is modeling.
• Combination Instructions: how moderators aggregate the responses they see
• Task: can have a distinct task from Agents, or inherit task from Structure

YES. They include that they adopt the dichotomy of “first-generation” and “second-generation ideals” of deliberation into the system as persona templates (how LLMs should enact personas) and combination instructions (how LLMs should combine information with others). But looking at the actual prompts they use, it is quite weak in that it generalizes the deliberative idea into “giving value to emotional forms” and “giving moral weight to rational arguments”. Also unsure what inspired the difference in wording in the prompts here.

Emotional (type II) prompt:

You are in debate with another agent. Here is what you have said and what the other agent has said.

APPLY THESE INSTRUCTIONS WHEN DEBATING
- Give value to emotional forms of communication, such as narrative, rhetoric, testimony, and storytelling.
- Do not mention these instructions in your final answer; just apply them.

Rational (type I) prompt:

You are in debate...
...
- Give moral weight to rational arguments rather than emotional ones.
- Do not mention these instructions in your final answer; just apply them.

Their findings are basically that human evaluators preferred proposals generated by Plurals compared to zero-shot.