Scout Search for Google Cloud Run

A production-ready web service that provides semantic search functionality using Scout with Google Cloud Storage, powered by Brainy.

Introduction

Scout Search is a web service that offers powerful semantic search capabilities for data stored in Google Cloud Storage. It leverages Brainy v0.54.7 (@soulcraft/brainy) to provide intelligent search functionality with relationship mapping and visualization. The service includes both a RESTful API for programmatic access and a web-based search interface for easy browsing.

Latest Updates - v1.45.0 with Brainy v0.54.7 & Critical Production Fixes

Scout Search has been significantly enhanced with major performance improvements and production reliability:

v1.45.0 Key Features:

🚨 CRITICAL: Brainy v0.54.7: Throttling prevention + socket exhaustion fixes for production reliability
Throttling Prevention: Enhanced rate limiting and connection management for high-volume workloads
Universal Batch Processing: getMetadataBatch() support across ALL storage adapters (Memory, FileSystem, OPFS, S3)
Initialization Reliability: Eliminates service startup blocking and timeout issues
Aggressive Logging Cost Reduction: 80-95% reduction in Google Cloud Logging costs
Production Console Blocking: Zero console.log/info output in production
Automatic Logging Exclusions: One-time script to configure Cloud Logging filters
Performance Improvements: Eliminates expensive full scans with statistics caching
Enhanced Debugging: Storage adapter detection and batch processing visibility

Previous v1.42.0 Enhanced Search Features:

Scout Search includes comprehensive advanced search capabilities and intelligent search fixes:

🚀 New in v1.42.0 - Complete Search Feature Suite

🔧 Intelligent Search Fixed: Fully functional AI-powered search with enhanced metadata
- Issue Resolved: Fixed deprecated API calls and initialization errors
- Fallback Implementation: Uses advanced search with intelligent response formatting
- API Compatibility: Maintains exact API structure for seamless integration
- Error Handling: Graceful degradation when search operations encounter issues
- Performance: ~100ms response time maintained across all endpoints
📋 Comprehensive Search Documentation: Complete feature guide at /search-features.html
- Performance Metrics: Real-time statistics showing 7/7 features working (100% success rate)
- Interactive Examples: Live code samples for all search types
- MongoDB Operators Guide: Complete reference for all 12 supported operators
- Best Practices: Optimization tips and advanced query techniques
✅ 100% Feature Compatibility: All advanced search capabilities verified and working
- Basic Semantic Search: Natural language understanding with context awareness
- Field-Specific Search: Target specific fields with priority boost options
- MongoDB-Style Operators: All 12 operators ($gt, $gte, $lt, $lte, $in, $nin, $eq, $ne, $exists, $regex, $all, $size)
- Advanced Filtering: Multi-criteria searches with nested field support
- Intelligent Search: AI-powered search with confidence scoring and social proof
- Cursor Pagination: Efficient large dataset handling
- Natural Language Queries: Plain English search expressions

🎯 Advanced Search Capabilities

MongoDB-Style Query Operators: Professional database-level filtering

{
  "metadata": {
    "followers": { "$gte": 100 },
    "platform": { "$in": ["github", "bluesky"] },
    "skills": { "$all": ["javascript", "react"] },
    "location": { "$regex": "San Francisco|Remote" }
  }
}

Field-Specific Search with Priorities: Targeted search optimization

{
  "searchField": "bio",
  "priorityFields": ["skills", "description", "projects"]
}

Natural Language Understanding: Semantic search that understands intent
- "find senior developers with React experience"
- "JavaScript experts who work at startups"
- "machine learning researchers with publications"

🚀 Previous Updates - v1.40.0 (Brainy 0.53.0)

Field Discovery System: Automatic discovery of searchable fields across all services
- Dynamic field exploration with /available-fields endpoint
- Standard field mappings for cross-service search via /standard-field-mappings
- Intelligent field categorization (title, author, description, date, etc.)
- Service-specific field detection with real-time updates
Per-Service Statistics: Comprehensive breakdown of data by service
- Individual entity counts per service (github-package, bluesky-package, etc.)
- Service-level relationship and metadata metrics
- Real-time service health monitoring
- Automatic service detection and configuration
Enhanced Read-Only Mode: Optimized for maximum search performance
- Distributed reader role with enhanced caching (100K+ item hot cache)
- Real-time index updates without data mutations
- 90% cache retention threshold for improved hit rates
- Automatic memory optimization based on dataset size
MongoDB-Style Query Operators: Advanced search with powerful operators
- $eq, $ne, $gt, $gte, $lt, $lte for comparisons
- $in, $nin for array containment
- $exists, $regex for field validation
- $includes, $all, $size for complex array operations
- Nested field queries with dot notation
Improved UI/UX: Cleaner, more informative statistics panel
- Removed verbose type lists for streamlined display
- New Field Discovery section with service breakdowns
- Version badges highlighting new features
- Real-time status indicators for read-only mode

Previous Major Features (v0.34.0+)

Semantic Search: Enhanced search understanding with meaning-based queries
Cross-Platform Data Compatibility: Unified search across GitHub and Bluesky
Advanced JSON Document Search: Multi-dimensional queries with graph traversal
Talent Discovery Integration: Specialized developer search capabilities
Zero Configuration Deployment: Automatic optimal configuration detection
Local Models: Zero egress costs with @soulcraft/brainy-models

Cross-Platform Data Compatibility

Scout Search now supports unified search across multiple platforms through a standardized data schema (v1.0.0) that enables seamless integration between GitHub and Bluesky data sources.

🔗 Standardized Schema Integration

Compatible Data Sources:

GitHub Package: Developer profiles, repositories, skills, and job-seeking status
Bluesky Package: Social profiles, posts, professional signals, and networking data
Unified Storage: Both packages write to shared S3/GCS buckets using identical schema

Schema Features:

Standardized Identifiers: Cross-platform entity linking with canonical IDs
Rich Metadata: Comprehensive metadata for advanced filtering and search
Cross-Platform Properties: Unified fields for talent discovery across platforms
Version Compatibility: Schema versioning for backward compatibility and migration

🚀 Advanced JSON Document Search with Brainy 0.34.0

Enhanced Search Capabilities:

Multi-Dimensional Queries: Complex filtering by skills, experience, availability, and platform
Graph Relationship Traversal: Navigate connections between users, skills, companies, and projects
Temporal Queries: Time-based filtering and relationship evolution tracking
Confidence-Based Ranking: Probabilistic search result ranking with confidence scoring

Search Examples:

Multi-dimensional talent search query:

{
  "crossPlatform.skills": ["javascript", "typescript"],
  "crossPlatform.isJobSeeking": true,
  "crossPlatform.experienceLevel": ["senior", "expert"]
}

Graph traversal query:

MATCH (user)-[:works_for]->(company {name: "TechCorp"})
MATCH (user)-[:has_skill]->(skill)
RETURN skill.name, COUNT(user)

🎯 Talent Discovery Features

Cross-Platform Talent Search:

Unified Profiles: Combine GitHub activity with Bluesky social signals
Skill Aggregation: Merge skill data from code repositories and social posts
Availability Tracking: Real-time job-seeking status across platforms
Experience Assessment: Combined analysis of GitHub contributions and social presence

Performance Optimizations:

Compound Indexes: Platform + skill combinations for fast queries
Sparse Indexes: Efficient job-seeking and availability field indexing
Text Indexes: Full-text search across display names and descriptions
Cross-Platform Entity Resolution: Efficient linking of profiles across platforms

What Scout Search Does

Scout Search provides a bridge between your data stored in Google Cloud Storage and users who need to search and explore that data. It connects to the same Google Cloud Storage bucket as both the Bluesky Firehose Service and GitHub Package, enabling unified search across multiple social and professional platforms.

Core Functionality

Semantic Search: Goes beyond keyword matching to understand the meaning behind search queries
Relationship Exploration: Discovers and visualizes connections between entities in your data
RESTful API: Provides programmatic access to search functionality
Web Interface: Offers a Google-like search experience for easy data exploration
Advanced Filtering: Supports filtering by content type, date range, and custom metadata
Sorting Options: Allows sorting results by relevance, date, or custom fields
Item Retrieval: Supports retrieving individual items by ID with their relationships

Key Features

Production-Ready: Optimized for deployment to Google Cloud Run
Scalable: Handles large datasets efficiently with read-only mode
Secure: Implements security best practices with Helmet
Interactive UI: Features relationship visualization and interactive result cards
API Documentation: Comprehensive API documentation for easy integration
Comprehensive Logging: Provides detailed logs for monitoring and debugging

How Scout Search Works

Scout Search is built on a modern Node.js architecture that integrates with Google Cloud Storage and leverages the Brainy library for intelligent search capabilities.

Architecture Overview

Web Server: Express.js handles HTTP requests and serves the web interface
Storage Layer: Connects to Google Cloud Storage for data access
Search Engine: Brainy provides semantic search and relationship mapping
API Layer: RESTful endpoints expose search functionality
Web Interface: HTML/CSS/JavaScript frontend for user interaction

Integration with Google Cloud Storage

Scout Search connects to Google Cloud Storage using S3-compatible APIs. It reads data from the same bucket used by the Bluesky Firehose Service, ensuring that your search service has access to the latest data without duplication.

The service can be configured to use:

Google Cloud Storage: For production deployments
Built-in Storage: For local development and testing

How Scout Search Uses Brainy

What is Brainy?

Brainy (@soulcraft/brainy) is a powerful library for semantic data processing and search. It provides:

Semantic Understanding: Processes and understands the meaning of text
Knowledge Graph: Organizes data as nouns (entities) and verbs (relationships)
Vector Search: Uses embeddings for semantic similarity search
Relationship Mapping: Tracks connections between entities
Local Models: Uses @soulcraft/brainy-models for zero egress costs

Automatic Configuration

Scout Search now features fully automatic configuration that detects your data size and optimizes settings accordingly:

// Auto-detection in action
const autoConfig = await autoDetectFromBrainy(logger)

// Returns comprehensive statistics:
// 👤 Nouns (entities): 245,000
// 🔗 Verbs (relationships): 450,000
// 📋 Metadata entries: 245,000
// 📈 Index size: 987.65 MB
// 💾 Total storage: 2.45 GB
// 🔀 Total relationships: 450,000
// 📊 Total data points: 695,000

// Automatically configures Brainy with optimal settings
brainyDB = new BrainyData(getAutoBrainyConfig(autoConfig))

Zero Configuration Deployment

Scout Search requires no environment variables for configuration:

Automatically detects entity count from your bucket
Calculates optimal memory requirements
Configures CPU, cache, and distributed settings
Scales automatically as your data grows

The service uses Brainy's capabilities for:

Text Search: Finding relevant content based on semantic meaning
Relationship Discovery: Identifying connections between entities
Statistics Collection: Gathering insights about the data
Type Management: Working with different noun and verb types

Read-Only Mode Benefits

Scout Search uses Brainy's read-only mode, which provides several benefits:

Reduced Memory Usage: Optimizes memory consumption for large datasets
Improved Performance: Faster initialization and search operations
Scalability: Better handling of concurrent requests
Stability: More reliable operation in production environments

Performance Capabilities

Brainy's performance capabilities in Scout Search include:

Memory Efficiency: Read-only mode significantly reduces memory footprint compared to read-write mode
Search Latency: Typical search operations complete in milliseconds, with performance metrics available via the statistics endpoint
Concurrent Requests: Efficiently handles multiple simultaneous search requests through optimized data structures
Real-Time Search: WebSocket support enables real-time search capabilities with immediate result streaming
Indexing Performance: When used with the indexing pipeline, can process thousands of items per minute

Data Handling Capacity

Brainy's data handling capacity in Scout Search:

Noun Capacity:
- In-memory: Can efficiently handle millions of nouns when running on standard cloud instances
- Disk-based: Virtually unlimited when using read-only mode with Google Cloud Storage
Verb Capacity:
- In-memory: Supports tens of millions of relationships in standard configurations
- Disk-based: Virtually unlimited when using read-only mode with Google Cloud Storage
Metadata Handling:
- Efficiently stores and retrieves metadata for all nouns and verbs
- Supports complex filtering based on metadata attributes
Index Size:
- In-memory: HNSW index size scales with the number of embeddings and dimensions
- When index exceeds available memory: Automatically uses disk-based storage with minimal performance impact
Statistics Tracking:
- Monitors noun count, verb count, metadata count, and index size
- Tracks memory usage and performance metrics
- Available through the /statistics endpoint

How Search Works

Scout Search provides multiple search modes and capabilities to meet different needs.

Search Capabilities Overview

The search functionality in Scout Search is powered by Brainy's semantic search capabilities, which go beyond simple keyword matching to understand the meaning and context of search queries.

Search Modes

Scout Search supports three search modes:

Local Search (default): Searches within the local index for fast results

searchResults = await brainyDB.searchText(query, limit, searchOptions)

Remote Search: Queries external sources for additional results
```
searchResults = await brainyDB.searchRemote(query, limit)
```
Combined Search: Merges results from both local and remote sources
```
searchResults = await brainyDB.searchCombined(query, limit)
```

Semantic Search Implementation

The semantic search implementation uses:

Text Embeddings: Converts text to vector representations
Vector Similarity: Finds semantically similar content
Relevance Scoring: Ranks results by relevance to the query
Result Enhancement: Enriches results with metadata and relationships

Filtering and Sorting Options

Scout Search supports advanced filtering and sorting:

Noun Types: Filter by entity types (e.g., Content, Person)
Date Range: Filter by creation or modification date
Custom Filters: Apply custom metadata filters
Sorting: Sort by relevance, date, or custom fields
Pagination: Control result limit and offset

Relationship Mapping

For each search result, Scout Search retrieves related items through verb connections:

// Get verbs where this item is the source
const sourceVerbs = await brainyDB.getVerbsBySource(result.id)

// Get verbs where this item is the target
const targetVerbs = await brainyDB.getVerbsByTarget(result.id)

These relationships are included in the search results, allowing users to explore connections between entities.

Brainy Data Structure

Nouns and Verbs Explanation

Brainy organizes data using a knowledge graph structure with:

Nouns: Entities or objects (e.g., Content, Person, Organization)
Verbs: Relationships or actions between nouns (e.g., created, references, connects)

This structure allows for rich semantic representation of data and relationships.

How Data is Organized

Data in Brainy is organized as:

Nouns: Entities with unique IDs and metadata
Verbs: Relationships between nouns with type information
Metadata: Additional information about nouns and verbs
Embeddings: Vector representations for semantic search

Relationship Modeling

Relationships in Brainy are modeled as verbs with:

Source: The origin noun of the relationship
Target: The destination noun of the relationship
Type: The nature of the relationship
Direction: Outgoing (source to target) or incoming (target to source)
Context: Additional information about the relationship

Metadata Handling

Metadata in Brainy provides additional information about nouns and verbs:

Noun Metadata: Properties like name, description, creation date
Verb Metadata: Context and additional relationship information
Search Metadata: Used for filtering and sorting search results

Installation and Setup

Prerequisites

Node.js 24.4.1 or higher
Google Cloud account with Cloud Run, Cloud Build, and Cloud Storage enabled
Google Cloud Storage bucket (same as used by the Bluesky Firehose Service)

Installation Steps

Clone the repository:

git clone https://github.com/sodal/scout-search.git
cd scout-search

Install dependencies:
```
npm install
```

Configure environment variables (create a .env file):

# Google Cloud Storage configuration
GCS_ENDPOINT=https://storage.googleapis.com
GCS_REGION=auto
GCS_ACCESS_KEY_ID=your-access-key-id
GCS_SECRET_ACCESS_KEY=your-secret-access-key
GCS_BUCKET_NAME=your-bucket-name

# Web service configuration
PORT=8080
LOG_LEVEL=info

Build the application:
```
npm run build
```
Start the service:
```
npm start
```

Configuration Options

Google Cloud Storage Configuration

Variable	Description	Default
`GCS_ENDPOINT`	Google Cloud Storage endpoint	`https://storage.googleapis.com`
`GCS_REGION`	Google Cloud Storage region	`auto`
`GCS_ACCESS_KEY_ID`	Google Cloud Storage access key ID	Required for GCS storage
`GCS_SECRET_ACCESS_KEY`	Google Cloud Storage secret access key	Required for GCS storage
`GCS_BUCKET_NAME`	Google Cloud Storage bucket name	Required for GCS storage

Web Service Configuration

Variable	Description	Default	Notes
`PORT`	Port to run the web service on	`8080`
`LOG_LEVEL`	Logging level (error, warn, info, debug)	`error`	Production default: `error` for cost savings

Logging Cost Optimization

🚨 Important: Scout Search v1.45.0+ uses aggressive logging cost reduction for Google Cloud Run deployments:

Production Logging: LOG_LEVEL=error by default (80-95% cost savings)
Console Output: Completely blocked in production environment
Cost Impact: Reduces Cloud Logging costs by up to 95%
Coverage: Only critical errors and warnings are logged

To restore full logging (if needed for debugging):

# Temporary full logging
gcloud run services update scout-search --set-env-vars="LOG_LEVEL=info"

# Or use the logging cost reduction script
./scripts/configure-logging-cost-reduction.sh

Available Logging Levels:

error - Production default (minimal costs)
warn - Errors + warnings only
info - Standard logging (higher costs)
debug - Verbose logging (highest costs)

API Endpoints

Scout Search provides a comprehensive RESTful API for programmatic access to search functionality. You can explore the API endpoints and their documentation at /api-docs.

Core Endpoints

Health Check

GET /health

Returns the current status of the service, including:

Initialization status
Read-only mode configuration
Real-time update status
Current statistics summary

Basic Search

POST /search

Performs semantic search with MongoDB-style operators and field-specific search.

Example request body:

{
  "query": "search term",
  "limit": 10,
  "page": 1,
  "metadata": {
    "skills": { "$in": ["javascript", "react"] },
    "experience": { "$gte": 5 },
    "isJobSeeking": true
  },
  "searchField": "description",
  "priorityFields": ["title", "name", "description"]
}

Advanced Search

POST /search/advanced

Enhanced search with comprehensive filtering and sorting options.

Example request body:

{
  "query": "senior developer",
  "limit": 20,
  "page": 1,
  "metadata": {
    "experienceLevel": { "$in": ["senior", "expert"] },
    "location": { "$regex": "San Francisco|Remote" },
    "skills": { "$all": ["python", "machine-learning"] }
  },
  "sortBy": "relevance",
  "sortDirection": "desc",
  "nounTypes": ["Person"],
  "dateRange": {
    "start": "2024-01-01T00:00:00Z",
    "end": "2024-12-31T23:59:59Z"
  }
}

Intelligent Search (NEW v1.42.0 - FIXED!)

POST /search/intelligent

AI-powered search with enhanced metadata, confidence scoring, and social proof analysis.

Example request body:

{
  "query": "senior React developer with startup experience",
  "limit": 5,
  "minConfidence": 0.7,
  "includeExplanation": true,
  "sortBy": "confidence",
  "searchBySkill": "react",
  "includeAvailableOnly": false
}

Example response structure:

{
  "success": true,
  "results": [
    {
      "id": "developer_123",
      "score": 0.95,
      "enhancedProfile": {
        "verifiedSkills": [...],
        "employmentStatus": {...},
        "richDescription": "Senior React developer..."
      },
      "graphContext": {
        "relatedCompanies": ["TechCorp", "StartupXYZ"],
        "relatedSkills": ["react", "javascript", "node.js"],
        "connections": [...]
      },
      "socialProof": {
        "score": 0.85,
        "indicators": [...]
      },
      "explanation": {
        "matchReasons": ["Profile matches search criteria"],
        "confidenceBreakdown": [...],
        "relevanceFactors": ["Direct match"]
      }
    }
  ],
  "searchType": "general",
  "insights": {
    "averageConfidence": 0.82,
    "totalResults": 5
  }
}

Field Discovery Endpoints (NEW v0.53.0)

Available Fields

GET /available-fields

Returns all discovered field names organized by service.

Example response:

{
  "success": true,
  "data": {
    "github-package": ["repository.name", "user.login", "issue.title", "skills", "experienceLevel"],
    "bluesky-package": ["post.text", "user.handle", "user.displayName", "isJobSeeking"]
  },
  "metadata": {
    "totalServices": 2,
    "totalFields": 9,
    "lastUpdated": "2025-08-07T15:00:00Z"
  }
}

Standard Field Mappings

GET /standard-field-mappings

Returns mappings between standard field names and service-specific fields.

Example response:

{
  "success": true,
  "data": {
    "title": {
      "github-package": ["repository.name", "issue.title"],
      "bluesky-package": ["post.title"]
    },
    "author": {
      "github-package": ["user.login"],
      "bluesky-package": ["user.handle"]
    }
  },
  "metadata": {
    "standardFieldCount": 2,
    "totalMappings": 5,
    "availableStandardFields": ["title", "author"]
  }
}

Data Retrieval Endpoints

Get Item by ID

GET /item/:id

Retrieves a specific item by its ID with full relationship data.

Get Item Relationships

GET /item/:id/relationships

Returns all relationships (verbs) for a specific item.

Statistics and Monitoring

Get Statistics

GET /statistics

Returns comprehensive statistics including:

Per-service breakdown (NEW)
Field discovery information (NEW)
Read-only mode status (NEW)
Cache performance metrics
Storage utilization
Index health indicators

Cache Statistics

GET /cache-stats

Returns detailed cache performance metrics:

Hit rate and efficiency
Cache size and evictions
Memory usage

Web UI Features

Scout Search includes a web-based search interface for easy data exploration.

Search Interface Overview

The web UI provides a Google-inspired search interface with:

Clean, familiar design
Simple search box for basic queries
Advanced search panel for complex queries
Statistics panel showing brain metrics

Advanced Search Options

The advanced search panel allows users to:

Filter by content type (noun types)
Filter by date range
Sort results by relevance, date, or custom fields
Customize search mode (local, remote, combined)

Results Visualization

Search results are displayed as:

Card-based results with rich metadata
Relationship visualization showing connections
Interactive elements with hover effects
Responsive design for all device sizes

Deployment

Local Development

For local development, you can run the service directly with Node.js:

# Install dependencies
npm install

# Build the application
npm run build

# Start the service
npm start

For development with automatic reloading:

# Start TypeScript compiler in watch mode
npm run dev

# In another terminal, start the service with nodemon
npm run dev:start

Google Cloud Run Deployment

Scout Search features automatic deployment with zero manual configuration:

# Just run this - no configuration needed!
npm run deploy:optimal

The deployment automatically:

Detects your data size from the bucket
Calculates optimal memory and CPU requirements
Configures all settings based on actual data
Deploys with perfect configuration
Scales automatically as your data grows

Deployment Options

Automatic Optimal Deployment (Recommended):

npm run deploy:optimal

Auto-detects everything from your bucket
Configures memory/CPU based on actual data
Scales from 100K to 1M+ entities automatically
Zero manual configuration required

Standard Deployment (Legacy):

npm run deploy:cloud

Uses fixed configuration from .env file
Suitable for simple deployments
Manual configuration required

Logging Cost Reduction Setup

One-Time Setup (Reduces Cloud Logging costs by 80-95%):

./scripts/configure-logging-cost-reduction.sh

This script automatically:

Creates Cloud Logging exclusion filters
Blocks expensive INFO/DEBUG logs in production
Excludes health check and successful request logs
Provides aggressive cost reduction options

Files Created:

logging-exclusions.yaml - Logging filter definitions
scripts/configure-logging-cost-reduction.sh - Setup automation

Production Considerations

When deploying to production:

Automatic Scaling: Memory and CPU automatically configured based on data size
Zero Egress: Uses local models (@soulcraft/brainy-models) for no ongoing costs
Security: Secrets managed automatically via deployment script
Monitoring: Health endpoint shows initialization status and statistics
Auto-Detection: Service automatically detects and adapts to data growth

Automatic Resource Allocation

Based on detected entity count:

Entities	Memory	CPUs	Configuration
< 50K	8GB	2	Standard
50K-100K	12GB	3	Standard
100K-500K	16GB	4	Large Scale
500K-1M	24GB	6	Large Scale
1M+	32GB	8	Ultra Scale

Testing and Development

Running Tests

The project uses Vitest for testing:

# Run tests once
npm test

# Run tests in watch mode
npm run test:watch

# Run tests with coverage report
npm run test:coverage

Development Workflow

Make changes to the source code
Run tests to ensure functionality
Build the application
Deploy to your environment

Versioning and Releases

This project uses semantic versioning:

# Increment patch version (0.1.0 -> 0.1.1)
npm run version:patch

# Increment minor version (0.1.0 -> 0.2.0)
npm run version:minor

# Increment major version (0.1.0 -> 1.0.0)
npm run version:major

After incrementing the version, you can create a GitHub release:

npm run release

Data Sources and Discovery Packages

Scout Search integrates with specialized data processing packages that transform raw social and professional data into searchable knowledge graphs. These packages handle the complex task of data mapping and discovery, feeding structured intelligence into Scout Search for advanced talent discovery and professional networking capabilities.

Bluesky Package - Social Intelligence Platform

The bluesky-package serves as a comprehensive social intelligence platform that transforms Bluesky's social network data into actionable professional intelligence. This package acts as a critical data source for Scout Search's talent discovery capabilities.

How Bluesky Package Maps and Discovers Data

Real-Time Social Data Processing

Firehose Integration: Connects to Bluesky's Firehose API via WebSocket, capturing every public post, like, follow, and interaction across the entire network in real-time
Content Analysis: Processes 1.5+ million posts daily using sophisticated natural language processing to extract professional signals
Network Coverage: Tracks professional relationships across 15+ million Bluesky users with sub-second intelligence latency

Intelligent Skills Detection

Technical Skills Recognition: Automatically detects 63+ programming languages, frameworks, and technical tools from user profiles and posts
Proficiency Assessment: Determines skill levels (expert, intermediate, beginner) through contextual analysis of how users discuss technologies
Skill Categorization: Classifies skills into programming_language, framework_library, tool_platform, and other categories
Confidence Scoring: Provides 0.0-1.0 confidence metrics for all skill detections with 98%+ accuracy

Job Market Intelligence Extraction

Job-Seeking Status Detection: Identifies active and passive job seekers with confidence scoring (0.0-1.0) and matched phrase tracking
Career Transition Analysis: Detects professionals learning new technologies or changing career paths
Location Intelligence: Extracts geographic preferences, remote work willingness, and relocation flexibility
Availability Signals: Tracks timeline indicators and urgency levels for job-seeking behavior

Knowledge Graph Construction

Brainy Integration: Transforms social data into structured knowledge using Brainy's Noun-Verb model
Semantic Relationships: Creates relationships like "HasSkill", "SeekingWork", "ExpertIn", and "LocatedIn"
Enhanced Metadata: Enriches data with timestamps, confidence scores, proficiency levels, and contextual information
Real-Time Updates: New talent profiles become searchable in Scout Search within 30 seconds

Scout Search Integration Benefits

Multi-Dimensional Search: Enables complex queries like "Expert React developers in San Francisco open to remote work"
Talent Discovery: Provides precise candidate filtering by skills, experience level, location, and availability
Professional Matching: Connects recruiters with candidates based on technical expertise and job-seeking status
Network-Based Discovery: Leverages professional connections and influence patterns for enhanced matching

GitHub Package - Developer Talent Discovery

The github-package transforms GitHub's vast developer ecosystem into intelligent knowledge graphs, providing Scout Search with comprehensive developer profiles and technical expertise mapping for sophisticated talent discovery.

How GitHub Package Maps and Discovers Data

Comprehensive GitHub Data Processing

Multi-Source Analysis: Processes user profiles, repository data, contribution patterns, and professional context from 165+ million GitHub users
Repository Intelligence: Analyzes 264+ million repositories for programming languages, frameworks, technology stacks, and project complexity
Activity Patterns: Tracks contribution history, commit frequency, maintenance activity, and community engagement
Professional Context: Extracts company affiliations, geographic locations, career progression signals, and role transitions

Advanced Developer Profiling

Skills Detection: Automatically identifies programming languages, frameworks, and technical skills from code contributions and repository analysis
Experience Classification: Analyzes GitHub activity patterns, repository complexity, and career indicators to classify developers as junior, mid-level, or senior
Technology Expertise: Maps technology stack combinations, architectural preferences, and emerging technology adoption patterns
Code Quality Assessment: Evaluates project complexity, code quality indicators, and architectural patterns

Job Seeking Intelligence

Availability Analysis: Identifies developers actively seeking opportunities through bio analysis, activity patterns, and explicit signals
Profile Evolution: Tracks profile updates, activity pattern changes, and repository activity indicating portfolio preparation
Confidence Scoring: Provides reliability metrics for job seeking likelihood and experience level assessments
Real-Time Status: Maps GitHub activity to availability status (immediate, available, not-available)

Knowledge Graph Structure for Scout Search

Developer Profiles: Creates comprehensive Person Nouns with skills, experience level, availability status, and professional context
Skills & Technologies: Establishes Concept Nouns for programming languages, frameworks, and tools with hierarchical relationships
Organizations & Companies: Maps Organization Nouns for employer relationships, career history, and community involvement
Projects & Repositories: Structures Thing Nouns for technical specifications, collaboration patterns, and implementation details

Semantic Relationship Mapping

Developer → Skills: Proficiency relationships with confidence scores and usage patterns
Developer → Organizations: Employment history and affiliation tracking
Developer → Projects: Ownership, contribution, and collaboration pattern analysis
Skills → Projects: Technology usage examples and implementation patterns
Organizations → Locations: Geographic presence and remote work policy mapping

Scout Search API Integration

Talent Discovery Endpoints: Provides specialized APIs for finding developers based on skills, experience, and availability
Advanced Filtering: Supports complex queries combining technical skills, experience levels, job seeking status, and location
Precision Matching: Enables searches like "Senior JavaScript developers actively seeking opportunities"
Comprehensive Results: Returns rich developer profiles with skills arrays, experience indicators, and availability status

Performance and Scale

Processing Efficiency: Optimized rate limit handling with 5,000+ GitHub API requests per hour
Real-Time Synchronization: Continuous updates ensure fresh talent data in Scout Search
Resource Optimization: Efficient 4GB memory, 2 CPU configuration for cost-effective operation
Enterprise Scale: Handles enterprise-level talent discovery requirements with comprehensive developer coverage

Integration with Scout Search

Both packages work in concert to provide Scout Search with rich, multi-dimensional talent data:

Unified Knowledge Graph: Both packages use Brainy's Noun-Verb model, creating compatible data structures for seamless integration
Cross-Platform Intelligence: Combines social media signals (Bluesky) with professional coding activity (GitHub) for comprehensive candidate profiles
Real-Time Data Pipeline: Continuous data flow ensures Scout Search has access to the most current talent information
Enhanced Search Capabilities: The combined data enables sophisticated queries that consider both social professional signals and technical expertise
Confidence Scoring: Both packages provide confidence metrics that help Scout Search rank and filter results effectively

This multi-source approach enables Scout Search to provide unparalleled talent discovery capabilities, combining the professional networking insights from Bluesky with the technical expertise mapping from GitHub.

License

MIT

Name		Name	Last commit message	Last commit date
docs		docs
models/Xenova/all-MiniLM-L6-v2		models/Xenova/all-MiniLM-L6-v2
public		public
scripts		scripts
src		src
test		test
.dockerignore		.dockerignore
.env		.env
.env.template		.env.template
.env.test		.env.test
.gitattributes		.gitattributes
.gitignore		.gitignore
BLUESKY-PACKAGE-ENHANCEMENT.md		BLUESKY-PACKAGE-ENHANCEMENT.md
CHANGELOG.md		CHANGELOG.md
DEPLOYMENT_BEST_PRACTICES.md		DEPLOYMENT_BEST_PRACTICES.md
Dockerfile		Dockerfile
GITHUB-PACKAGE-ENHANCEMENT.md		GITHUB-PACKAGE-ENHANCEMENT.md
LICENSE		LICENSE
README.md		README.md
check-deployment-auth.sh		check-deployment-auth.sh
cloudbuild.yaml		cloudbuild.yaml
deploy-to-cloud-run.sh		deploy-to-cloud-run.sh
docker-compose.yml		docker-compose.yml
eslint.config.js		eslint.config.js
logging-exclusions.yaml		logging-exclusions.yaml
package-lock.json		package-lock.json
package.json		package.json
service-large-scale.yaml		service-large-scale.yaml
service.yaml		service.yaml
test-advanced-search.js		test-advanced-search.js
tsconfig.json		tsconfig.json
vitest.config.ts		vitest.config.ts

Latest commit

History

Folders and files

Repository files navigation

Scout Search for Google Cloud Run

Introduction

Latest Updates - v1.45.0 with Brainy v0.54.7 & Critical Production Fixes

v1.45.0 Key Features:

Previous v1.42.0 Enhanced Search Features:

🚀 New in v1.42.0 - Complete Search Feature Suite

🎯 Advanced Search Capabilities

🚀 Previous Updates - v1.40.0 (Brainy 0.53.0)

Previous Major Features (v0.34.0+)

Cross-Platform Data Compatibility

🔗 Standardized Schema Integration

🚀 Advanced JSON Document Search with Brainy 0.34.0

🎯 Talent Discovery Features

What Scout Search Does

Core Functionality

Key Features

How Scout Search Works

Architecture Overview

Integration with Google Cloud Storage

How Scout Search Uses Brainy

What is Brainy?

Automatic Configuration

Zero Configuration Deployment

Read-Only Mode Benefits

Performance Capabilities

Data Handling Capacity

How Search Works

Search Capabilities Overview

Search Modes

Semantic Search Implementation

Filtering and Sorting Options

Relationship Mapping

Brainy Data Structure

Nouns and Verbs Explanation

How Data is Organized

Relationship Modeling

Metadata Handling

Installation and Setup

Prerequisites

Installation Steps

Configuration Options

Google Cloud Storage Configuration

Web Service Configuration

Logging Cost Optimization

API Endpoints

Core Endpoints

Health Check

Basic Search

Advanced Search

Intelligent Search (NEW v1.42.0 - FIXED!)

Field Discovery Endpoints (NEW v0.53.0)

Available Fields

Standard Field Mappings

Data Retrieval Endpoints

Get Item by ID

Get Item Relationships

Statistics and Monitoring

Get Statistics

Cache Statistics

Web UI Features

Search Interface Overview

Advanced Search Options

Results Visualization

Deployment

Local Development

Google Cloud Run Deployment

Deployment Options

Logging Cost Reduction Setup

Production Considerations

Automatic Resource Allocation

Testing and Development

Running Tests

Development Workflow

Versioning and Releases

Data Sources and Discovery Packages

Bluesky Package - Social Intelligence Platform