relay-tester

A comprehensive command-line tool for testing Nostr relay implementations against the NIP-01 specification and related NIPs. This tool validates relay compliance and helps developers ensure their implementations work correctly.

Features

• Comprehensive Test Coverage: Tests all major Nostr protocol features
• NIP Compliance Validation: Ensures relays follow Nostr Improvement Proposals
• Flexible Testing Options: Run all tests or focus on specific areas
• Multiple Output Formats: Human-readable or JSON output for automation
• Dependency-Aware Testing: Tests run in correct order with proper dependencies
• Integration with Build Pipeline: Suitable for CI/CD integration

Installation

From Source

# Clone the repository
git clone <repository-url>
cd next.orly.dev

# Build the relay-tester
go build -o relay-tester ./cmd/relay-tester

# Optionally install globally
sudo mv relay-tester /usr/local/bin/

Using the Install Script

# Use the provided installation script
./scripts/relaytester-install.sh

Usage

relay-tester -url <relay-url> [options]

Options

Option	Description	Default
-url	Required. Relay websocket URL (e.g., ws://127.0.0.1:3334 or wss://relay.example.com)	-
-test <name>	Run a specific test by name	Run all tests
-json	Output results in JSON format for automation	Human-readable
-v	Verbose output (shows additional info for each test)	false
-list	List all available tests and exit	false
-timeout <duration>	Timeout for individual test operations	30s

Examples

Basic Testing

Run all tests against a local relay:

relay-tester -url ws://127.0.0.1:3334

Run all tests with verbose output:

relay-tester -url ws://127.0.0.1:3334 -v

Specific Test Execution

Run a specific test:

relay-tester -url ws://127.0.0.1:3334 -test "Publishes basic event"

List all available tests:

relay-tester -list

Output Formats

Output results as JSON for automation:

relay-tester -url ws://127.0.0.1:3334 -json

Remote Relay Testing

Test a remote relay:

relay-tester -url wss://relay.damus.io

Test with custom timeout:

relay-tester -url ws://127.0.0.1:3334 -timeout 60s

Exit Codes

• 0: All required tests passed - relay is compliant
• 1: One or more required tests failed, or an error occurred
• 2: Invalid command-line arguments

Test Categories

The relay-tester runs comprehensive tests covering:

Core Protocol (NIP-01)

• Basic Event Operations:

- Publishing events - Finding events by ID, author, kind, and tags - Event retrieval and validation

• Filtering:

- Time range filters (since, until) - Limit and pagination - Multiple concurrent filters - Scrape queries for bulk data

• Event Types:

- Regular events (kind 1+) - Replaceable events (kinds 0, 3, etc.) - Parameterized replaceable events (addressable events with d tags) - Ephemeral events (kinds 20000+)

Extended Protocol Features

• Event Deletion (NIP-09): Testing deletion event handling
• EOSE Handling: Proper "end of stored events" signaling
• Event Validation: Signature verification and ID hash validation
• JSON Compliance: NIP-01 JSON escape sequences and formatting

Authentication & Access Control

• Authentication Testing: NIP-42 AUTH command support
• Access Control: Testing relay-specific access rules
• Rate Limiting: Basic rate limit validation

Test Results Interpretation

Successful Tests

✅ Publishes basic event
✅ Finds event by ID
✅ Filters events by time range

Failed Tests

❌ Publishes basic event: timeout waiting for OK
❌ Filters events by time range: unexpected EOSE timing

JSON Output Format

{
  "relay_url": "ws://127.0.0.1:3334",
  "timestamp": "2024-01-01T12:00:00Z",
  "tests_run": 25,
  "tests_passed": 23,
  "tests_failed": 2,
  "results": [
    {
      "name": "Publishes basic event",
      "status": "passed",
      "duration": "0.123s"
    },
    {
      "name": "Filters events by time range",
      "status": "failed",
      "error": "unexpected EOSE timing",
      "duration": "0.456s"
    }
  ]
}

Integration with Build Scripts

The relay-tester is integrated with the project's testing scripts:

# Test relay with default configuration
./scripts/relaytester-test.sh

# Test relay with policy enabled
ORLY_POLICY_ENABLED=true ./scripts/relaytester-test.sh

# Test relay with ACL enabled
ORLY_ACL_MODE=follows ./scripts/relaytester-test.sh

Testing Strategy

Development Testing

During development, run tests frequently:

# Quick test against local relay
go run ./cmd/relay-tester -url ws://127.0.0.1:3334

# Test specific functionality
go run ./cmd/relay-tester -url ws://127.0.0.1:3334 -test "EOSE handling"

CI/CD Integration

For automated testing in CI/CD pipelines:

# JSON output for parsing
relay-tester -url $RELAY_URL -json > test_results.json

# Check exit code
if [ $? -eq 0 ]; then
  echo "All tests passed!"
else
  echo "Some tests failed"
  cat test_results.json
  exit 1
fi

Performance Testing

The relay-tester can be combined with performance testing:

# Start relay
./orly &
RELAY_PID=$!

# Run compliance tests
relay-tester -url ws://127.0.0.1:3334

# Run performance tests
./scripts/runtests.sh

# Cleanup
kill $RELAY_PID

Troubleshooting

Common Issues

1. Connection Refused: Ensure relay is running and accessible
2. Timeout Errors: Increase timeout with -timeout flag
3. Authentication Required: Some relays require NIP-42 AUTH
4. WebSocket Errors: Check firewall and network configuration

Debug Output

Use verbose mode for detailed information:

relay-tester -url ws://127.0.0.1:3334 -v

Test Dependencies

Tests are run in dependency order. If a foundational test fails, subsequent tests may also fail. Always fix basic event publishing before debugging complex filtering.

Development

Running Tests

# Run relay-tester unit tests
go test ./cmd/relay-tester

# Run all tests including relay-tester
go test ./...

# Run with coverage
go test -cover ./cmd/relay-tester

Adding New Tests

1. Add test case to the test suite
2. Update test dependencies if needed
3. Ensure proper error handling
4. Update documentation

License

This tool is part of the next.orly.dev project and follows the same licensing terms.