Skip to main content

Slack Provider

The Slack provider enables human-in-the-loop evaluations by sending prompts to Slack channels or users and collecting responses. This is useful for:

  • Collecting human feedback on AI outputs
  • Comparing human responses with AI responses
  • Building golden datasets from expert feedback
  • Running evaluations with domain experts

Prerequisites​

Install Dependencies​

The Slack provider requires @slack/web-api@^8.1.1, installed alongside promptfoo:

npm install promptfoo @slack/web-api@^8.1.1
note

The SDK is not installed by default. For a global promptfoo installation, use npm install -g promptfoo @slack/web-api@^8.1.1 instead.

Slack App Setup​

  1. Create a Slack App

    • Go to api.slack.com/apps
    • Click "Create New App" → "From scratch"
    • Give your app a name and select your workspace

  2. Configure Bot Token Scopes

    • Navigate to "OAuth & Permissions" in your app settings
    • Under "Scopes" → "Bot Token Scopes", add these REQUIRED scopes:
      • chat:write - to send messages
      • channels:history - to read public channel messages
      • groups:history - to read private channel messages
      • im:history - to read direct messages
      • channels:read - to access public channel information
      • groups:read - to access private channel information
      • im:read - to access direct message information

    Note: All scopes are required for the provider to work properly across different channel types.

  3. Install App to Workspace

    • Go to "Install App" in your app settings
    • Click "Install to Workspace"
    • Copy the "Bot User OAuth Token" (starts with xoxb-)

  4. Invite Bot to Channel

    • In Slack, go to the channel where you want to use the bot
    • Type /invite @YourBotName

Configuration​

Environment Variables​

export SLACK_BOT_TOKEN="xoxb-your-bot-token"

Slack requests respect the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables on all supported Node.js versions.

Basic Configuration​

providers:
  - id: slack
    config:
      channel: 'C0123456789' # Your channel ID

Provider Formats​

The Slack provider supports multiple formats:

# Basic format with channel in config
providers:
  - id: slack # Uses SLACK_BOT_TOKEN env var
    config:
      # token: "{{ env.SLACK_BOT_TOKEN }}"  # optional, auto-detected
      channel: 'C0123456789'

# Short format - channel ID directly in provider string
providers:
  - slack:C0123456789

# Explicit channel format
providers:
  - slack:channel:C0123456789

# Direct message to a user
providers:
  - slack:user:U0123456789

Configuration Options​

OptionTypeRequiredDefaultDescription
tokenstringYes*SLACK_BOT_TOKEN env varSlack Bot User OAuth Token
channelstringYes-Channel ID (C...) or User ID (U...)
responseStrategystringNo'first'How to collect responses: 'first', 'user', or 'timeout'
waitForUserstringNo-User ID to wait for (when using 'user' strategy)
timeoutnumberNo60000Timeout in milliseconds
includeThreadbooleanNofalseInclude thread timestamp in output metadata
formatMessagefunctionNo-Custom message formatting function
threadTsstringNo-Thread timestamp to reply in

*Token is required either in config or as environment variable

Token precedence is config.token, provider-scoped env.SLACK_BOT_TOKEN, then the process SLACK_BOT_TOKEN. Eval-level env values are also supported; provider-scoped values take precedence.

Response Strategies​

First Response (Default)​

Captures the first non-bot message after the prompt:

providers:
  - id: slack
    config:
      channel: 'C0123456789'
      responseStrategy: 'first'

Specific User​

Waits for a response from a specific user:

providers:
  - id: slack
    config:
      channel: 'C0123456789'
      responseStrategy: 'user'
      waitForUser: 'U9876543210'

Timeout Collection​

Collects all responses until timeout:

providers:
  - id: slack
    config:
      channel: 'C0123456789'
      responseStrategy: 'timeout'
      timeout: 300000 # 5 minutes

Finding Channel and User IDs​

Channel IDs​

  1. In Slack, click on the channel name in the header
  2. Click "About" tab
  3. At the bottom, you'll see the Channel ID (starts with C)

User IDs​

  1. Click on a user's profile
  2. Click the "..." menu
  3. Select "Copy member ID" (starts with U)

Alternative Method​

  1. Right-click on a channel/user in the sidebar
  2. Select "Copy link"
  3. The ID is at the end of the URL

Channel ID Formats​

  • C... - Public channels
  • G... - Private channels/groups
  • D... - Direct messages
  • W... - Shared/Connect channels

Examples​

Basic Human Feedback Collection​

promptfooconfig.yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Collect human feedback on AI responses

providers:
  - id: openai:gpt-5
  - id: slack:C0123456789
    config:
      timeout: 300000 # 5 minutes

prompts:
  - 'Explain {{topic}} in simple terms'

tests:
  - vars:
      topic: 'quantum computing'
  - vars:
      topic: 'machine learning'
  - vars:
      topic: 'blockchain technology'
# Run with: promptfoo eval -j 1

Expert Review with Specific User​

promptfooconfig.yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Get expert feedback from specific team member

providers:
  - id: slack
    config:
      channel: 'C0123456789'
      responseStrategy: 'user'
      waitForUser: 'U9876543210' # Expert's user ID
      timeout: 600000 # 10 minutes

prompts:
  - |
    Review the following code for correctness, security, and maintainability:

    {{code}}

tests:
  - vars:
      code: |
        def factorial(n):
          if n == 0:
            return 1
          return n * factorial(n-1)

Thread-based Conversations​

threadTs controls where the prompt is posted. Response collection currently reads conversation history, so replies that remain only in the thread are not collected.

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Continue conversation in thread

providers:
  - id: slack
    config:
      channel: 'C0123456789'
      threadTs: '1234567890.123456' # Existing thread
      includeThread: true

prompts:
  - 'Follow-up question: {{question}}'

Custom Message Formatting​

// promptfooconfig.js
module.exports = {
  providers: [
    {
      id: 'slack',
      config: {
        channel: 'C0123456789',
        formatMessage: (prompt) => {
          return `🤖 *AI Evaluation Request*\n\n${prompt}\n\n_Please provide your feedback_`;
        },
      },
    },
  ],
};

Best Practices​

  1. Concurrency: Run Slack evaluations with -j 1 to ensure messages are sent sequentially

    promptfoo eval -j 1

  2. Timeouts: Set appropriate timeouts based on expected response time

    • Quick feedback: 60-120 seconds
    • Detailed review: 5-10 minutes
    • Async collection: 30+ minutes

  3. Channel Selection:

    • Use dedicated evaluation channels to avoid spam
    • Consider private channels for sensitive evaluations
    • Use DMs for individual expert feedback

  4. Message Formatting:

    • Use clear, structured prompts
    • Include context and instructions
    • Use Slack's markdown for better readability

  5. Rate Limits: Be aware of Slack's rate limits

Testing Other Slack Bots​

The Slack provider supports testing other Slack bots in their native environment. This allows you to:

  • Evaluate bot responses to various prompts
  • Compare different bot implementations
  • Perform regression testing
  • Test bot behavior under different scenarios

Setup for Bot Testing​

  1. Invite both bots to a test channel:

    /invite @your-bot-to-test
    /invite @provider

  2. Configure the provider to mention the target bot:

    providers:
      - id: slack
        config:
          channel: C123456789
          timeout: 10000
          responseStrategy: user
          waitForUser: U_YOUR_BOT_ID

    prompts:
      - '<@U_YOUR_BOT_ID> What can you help me with?'

  3. Filter responses to only capture the target bot:

    Set waitForUser to the bot's user ID (U...), not its app ID or bot_id.

    providers:
      - id: slack
        config:
          channel: C123456789
          timeout: 10000
          responseStrategy: user
          waitForUser: U_YOUR_BOT_ID # The bot's user ID

Example: Testing a Customer Support Bot​

promptfooconfig.yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Test our customer support bot

providers:
  - id: slack
    label: support-bot-test
    config:
      channel: C_TEST_CHANNEL
      timeout: 15000
      responseStrategy: user
      waitForUser: U_SUPPORT_BOT_ID

prompts:
  - '{{message}}'

tests:
  - vars:
      message: '<@U_SUPPORT_BOT_ID> How do I reset my password?'
    assert:
      - type: contains
        value: 'reset'
      - type: contains
        value: 'password'

  - vars:
      message: '<@U_SUPPORT_BOT_ID> What are your business hours?'
    assert:
      - type: contains-any
        value: ['hours', 'open', 'closed', 'Monday', 'schedule']

  - vars:
      message: '<@U_SUPPORT_BOT_ID> I need to speak to a human'
    assert:
      - type: contains-any
        value: ['agent', 'representative', 'transfer', 'human']

  - vars:
      message: "<@U_SUPPORT_BOT_ID> My order hasn't arrived yet, order #12345"
    assert:
      - type: contains
        value: '12345'
      - type: javascript
        value: |
          // Check if bot asked for more info or provided status
          return output.includes('track') || output.includes('status') || output.includes('delivery');

Advanced Bot Testing Patterns​

1. Multi-turn Conversations​

Test conversation flows by chaining prompts:

prompts:
  - "Hi, I'd like to order a pizza"
  - 'Yes, I want a large pepperoni'
  - 'My address is 123 Main St'

2. Error Handling​

Test how the bot handles invalid inputs:

prompts:
  - 'HELP ME NOW!!!!!!'
  - 'asdfghjkl'
  - "' OR 1=1 --"
  - ''

3. Load Testing​

Use a sequential run as a baseline in a shared channel. Concurrent load tests require isolated channels so prompts do not collect the same response:

promptfoo eval -c bot-test-config.yaml -j 1

4. A/B Testing Different Bots​

Compare multiple bot implementations:

promptfooconfig.yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: slack
    label: bot-v1
    config:
      channel: C_CHANNEL_V1
      responseStrategy: user
      waitForUser: U_BOT_V1

  - id: slack
    label: bot-v2
    config:
      channel: C_CHANNEL_V2
      responseStrategy: user
      waitForUser: U_BOT_V2

prompts:
  - "What's your return policy?"

defaultTest:
  assert:
    - type: regex
      value: '30[- ]day'
    - type: icontains
      value: 'return'

Best Practices for Bot Testing​

  1. Use dedicated test channels to avoid disrupting production
  2. Set appropriate timeouts - bots may take longer to respond than humans
  3. Test edge cases including malformed inputs and prompt injection attempts
  4. Monitor rate limits when running many tests
  5. Use assertions to verify both content and format of responses
  6. Test at different times to ensure consistent performance

Finding Bot User IDs​

To find a bot's user ID:

// Run this in your test channel
const { WebClient } = require('@slack/web-api');
const client = new WebClient(process.env.SLACK_BOT_TOKEN);

async function findBotId() {
  const members = await client.conversations.members({
    channel: 'C_YOUR_CHANNEL',
  });

  for (const userId of members.members) {
    const user = await client.users.info({ user: userId });
    if (user.user.is_bot) {
      console.log(`Bot: ${user.user.name} - ID: ${userId}`);
    }
  }
}

Troubleshooting​

Bot not responding​

  • Ensure bot is invited to the channel
  • Check bot has required permissions
  • Verify token is valid

Timeout errors​

  • Increase timeout value
  • Check if users are active in channel
  • Consider using different response strategy

Missing messages​

  • Ensure bot has channels:history permission
  • Check if messages are in threads
  • Verify channel ID is correct

Security Considerations​

  • Store tokens as environment variables, not in config files
  • Use private channels for sensitive data
  • Regularly rotate bot tokens
  • Limit bot permissions to minimum required
  • Remove bot from channels when not in use

Complete Example​

promptfooconfig.yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
# Human evaluation of customer service responses
description: Compare AI and human customer service responses

providers:
  - id: openai:gpt-5
    config:
      temperature: 0.7

  - id: anthropic:messages:claude-sonnet-5

  - id: slack:C0123456789
    config:
      responseStrategy: 'first'
      timeout: 180000 # 3 minutes

prompts:
  - |
    📋 Customer Service Evaluation

    Customer message: "{{message}}"

    How would you respond to this customer? Please provide a helpful and empathetic response.

tests:
  - vars:
      message: "I've been waiting for my order for 2 weeks and no one is responding to my emails!"
    assert:
      - type: llm-rubric
        value: Response acknowledges delay and provides concrete next steps

  - vars:
      message: 'The product I received is damaged and I need a replacement'
    assert:
      - type: llm-rubric
        value: Response offers immediate solution and apologizes for inconvenience

  - vars:
      message: 'How do I upgrade my subscription plan?'
    assert:
      - type: contains
        value: upgrade
# Run evaluation
# promptfoo eval -j 1 --no-progress-bar

Testing Your Setup​

Quick Test​

  1. Create a test channel in Slack

  2. Invite your bot to the channel: /invite @YourBotName

  3. Create a simple test config:

    promptfooconfig.yaml
    # yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
    providers:
      - id: slack:YOUR_CHANNEL_ID
        config:
          timeout: 30000

    prompts:
      - "Test message - please reply with 'success'"

    tests:
      - assert:
          - type: contains
            value: success

  4. Run: npx promptfoo eval -j 1

  5. Reply in Slack within 30 seconds

Common Issues​

  • Bot not in channel: Always invite the bot first with /invite @YourBotName
  • No response captured: Check the bot has all required scopes
  • Rate limits: The provider polls every second. Increasing the timeout does not reduce the polling rate; check whether your app is subject to stricter conversation history limits.

See Also​

Last updated on by renovate[bot]