yoink-core

Telegram bot platform with a plugin system, REST API, and React WebApp. Handles auth, RBAC, groups, settings, and provides a plugin protocol that each plugin implements.

The default bot/API runtime is Rust with native Turso storage. Download, Music, Stats, Insight and Inbox share current authorization and bounded lifecycle ownership. See native deployment for configuration, cutover, backups and rollback. Python sources and PostgreSQL remain legacy reference/rollback assets; plugin frontends remain under plugins/.

Legacy Python entrypoints

Script	Module	Description
`yoink-combined`	`yoink.combined_main:main`	Legacy deployment. Bot + API in a single asyncio loop; shared session_factory and bot_data. Used by docker entrypoint.
`yoink`	`yoink.main:main`	Bot only (no API). Useful when API runs separately.
`yoink-api`	`yoink.api_main_cli:main`	API only via uvicorn (programmatic).
`yoink.api_main`	ASGI app object	For external ASGI servers: `uvicorn yoink.api_main:app`.

For legacy split-deploy scenarios run yoink + yoink-api in separate containers. For single-container deploys use yoink-combined.

Quick start

Clone with git clone --recurse-submodules https://github.com/AdamsGH/yoink-core.git. Use Docker with Compose 2.24.4+, Node.js 22+, the pinned Rust 1.99.1 toolchain, just, and the host ACL utilities. Configure a reachable SOCKS proxy in .env; the default points to the host's port 1080. Do not use the production bot token while another poller is active.

cp .env.example .env        # fill in required vars
just data-dirs              # create private data directories (sudo required)
just build all              # build runtime, YouTube, frontend and nginx
just up                     # start services
just build backup           # build the native archive operator
# Native schemas migrate and verify automatically at startup.

The first runtime build initializes private release state and updates only the runtime image pin in .env. It does not start a container. If a yoink container already exists without release state, the build refuses automatic initialization. Use the adoption procedure in backend releases instead.

For a published vMAJOR.MINOR.PATCH checkout, replace just build all with just pull-images followed by just build yoink. The latter validates and adopts the downloaded runtime without rebuilding it. just pull-images also downloads the archive operator, so omit just build backup. Images are available under ghcr.io/adamsgh/yoink-core/{backend,frontend,nginx,youtube-services,importer,operations}:MAJOR.MINOR.PATCH. These images target Linux amd64. Plugin code and frontends are included in the runtime and SPA; plugin repositories do not define independent services.

Provision the Bot API dependency before just up. The supplied Compose configuration still expects yoink/tg-bot-api:latest; existing installations keep that image unchanged. On a fresh host, the legacy compatibility option is an explicit just build tg, which compiles patched TDLight. Neither just build all, just pull-images, nor release CI builds or downloads it. A different Bot API image requires its own compatible service configuration; the release helpers do not replace this dependency automatically.

GitHub CI runs just verify, builds the SPA, scans all five repository histories, and builds the six release images. The optional custom TDLight build is excluded from the release matrix. Master publishes commit-addressed sha-COMMIT tags. Release tags must match backend/VERSION and publish SemVer image tags. Publication never deploys or modifies production data. Existing image tags cannot move to another source commit.

Use just ci-local before remote CI on Linux amd64. It requires a local Docker daemon with Buildx and a configured Git commit identity. Keep 40 GiB free on the project and Docker storage filesystems. The containerd image store must use its standard /var/lib/containerd location.

The default runs the actual workflow's browser and verify jobs as pull_request. Use just ci-local browser for the browser preflight alone. The runner pins act 0.2.89, Ubuntu 24.04, Chrome 155.0.8059.39 and Node 24 for actions. The workflow selects Node 22 for tests.

The disposable snapshot includes full histories, tracked edits and the named helper files. Dirty submodules, missing tracked files and source symlinks stop the run. Live environment files, tokens, user configuration and unrelated untracked files stay outside the snapshot. act skips checkout to preserve local edits. Only source verification runs locally. Image publication remains a final GitHub check.

Run this helper only for trusted workflows. Its privileged controller uses its own Docker daemon, not the production Docker socket. It mounts only the snapshot and uses bridge networks and bounded resources. Cleanup removes owned containers, volumes, images, caches and temporary files on success, failure or cancellation. A 512 KiB log tail remains in backend/target/ci-local-logs/. Cleanup failures are errors.

GHCR initially creates private packages, even for public repositories. After the first publication, the package owner must set each package's visibility to public in GitHub Package settings and verify anonymous pulls. Until then, registry authentication is required.

Services

Service	Description	Port
`yoink`	Native Rust bot + API	8003
`yoink-frontend`	React SPA (nginx)	3010
`yoink-nginx`	HTTP edge	8081
`tg-bot-api`	Custom tdlight Bot API server	8082
`yoink-proxy`	HTTP facade over the shared host proxy on 1080	internal 8080
`native-youtube-auth-gateway`	Combined youtubei, bgutil and Unix adapter	Unix socket only

PostgreSQL, Redis, ARQ and the old standalone YouTube helpers are excluded from default startup. Use the legacy override only for an explicit rollback. Their data volumes are preserved.

Optional service	Description	Port
`yoink-backup`	Legacy pg_dump + S3 (profile `backup`)	-
`yoink-browser`	Kasmweb Chromium (profile `cookies`)	6902

Just recipes

just build [yoink|gateway|frontend|tg|backup|all]
just pull-images                # download images for the checked-out release
just up [service]
just down
just restart <service>
just logs [service]
just ps
just migrate [up|down|current|history|create "msg"]
just psql
just shell [service]
just test [path]
just data-dirs
just clean-pyc
just reset
just tg login +<phone>
just tg status
just tg logout
just browser [up|down|logs]
just proxy-init
just backup
just backup restore [list|latest|FILE]
just backup [up|down|logs]
just lint                       # native workspace Clippy
just verify-py [test-path]       # explicit legacy container lint + pytest
just verify-fe                   # i18n, Inbox/Insight regressions and frontend tsc
just verify                      # Rust, deployment, publication and frontend checks

Environment variables

Core variables (all lowercase):

Variable	Required	Default	Description
`bot_token`	yes	-	Telegram bot token
`owner_id`	yes	-	Telegram user ID of the owner
`api_id` / `api_hash`	yes	-	Telegram API credentials (for tg-bot-api)
`api_secret_key`	yes	-	JWT signing secret
`yoink_plugins`	no	-	Comma-separated plugin names (default: empty)
`database_url`	no	-	Legacy PostgreSQL URL; unused by the native runtime
`telegram_base_url`	no	-	Bot API base URL (default: official Telegram)
`data_dir`	no	-	Host path for cookies, sessions, browser profile
`json_logs`	no	-	Enable JSON log format
`DEV_AUTH_ENABLED`	no	`false`	Enable /auth/dev endpoint (restricted to DEV_ALLOWED_CIDR by nginx)
`DEV_ALLOWED_CIDR`	no	`192.168.0.0/16`	CIDR allowed to access /auth/dev (nginx-enforced)

See .env.example for the full list with defaults. Plugin-specific variables are documented in each plugin's README.

REST API

Base path: /api/v1. Docs: http://localhost:8003/docs.

Auth: Authorization: Bearer <JWT> obtained via POST /api/v1/auth/token (Telegram WebApp initData).

Core endpoints

Method	Path	Auth	Description
POST	/auth/token	-	Exchange Telegram initData for JWT
POST	/auth/dev	-	Dev token (local network only, requires DEV_AUTH_ENABLED=true)
GET	/users/me	user	Current user profile
GET	/users	admin	List all users (sortable: created_at, updated_at, name, role, dl_count, dl_last_at)
PATCH	/users/{id}	admin	Update role
GET	/users/{id}/stats	admin	Activity stats (dl + music + ai aggregated)
GET	/users/{user_id}/photo	public	Proxy user avatar from Bot API
POST	/users/photos/sync	owner	Mass-backfill user avatars from Bot API
GET	/groups	admin	List groups
POST	/groups	admin	Add group
PATCH	/groups/{id}	admin	Update group settings
GET	/groups/{id}/photo	public	Proxy group chat photo from Bot API
GET	/groups/{id}/threads	admin	List thread policies
POST	/groups/{id}/threads	admin	Add/toggle thread policy
DELETE	/groups/{id}/threads/{pid}	admin	Delete thread policy
GET	/threads/status	admin	Check if user-mode session is available
POST	/threads/scan/{group_id}	admin	Scan forum topics via user-mode session
GET	/settings	user	Personal settings
PATCH	/settings	user	Update settings
GET	/bot-settings	admin	Global bot settings
PATCH	/bot-settings	owner	Update global settings
GET	/bot-settings/tag-map	admin	Tag-to-feature map
PUT	/bot-settings/tag-map	owner	Update tag-to-feature map
GET	/bot-settings/available-features	admin	All registered features
GET	/permissions/all	admin	List all user permissions
POST	/users/{id}/permissions	admin	Grant a feature
DELETE	/users/{id}/permissions/{plugin}/{feature}	admin	Revoke a feature
GET	/users/{id}/feature-access	admin	Effective feature access for a user
GET	/features	user	List all registered features

M2M API

Auth: X-Api-Key header. Base path: /api/internal/v1.

Method	Path	Scope	Description
GET	/status	health:r	Bot status
GET	/users	users:r	List users
GET	/groups	groups:r	List groups
POST	/events	events:w	Create event

API key management: GET/POST /api/v1/api-keys, DELETE /api/v1/api-keys/{id} (owner only).

Health

Method	Path	Description
GET	/health	Health check (DB / bot / disk)
GET	/metrics	In-process counters

Plugin routes are mounted at /api/v1/{plugin_name}/.

Dev authentication

POST /api/v1/auth/dev generates a JWT for any user_id without Telegram verification.

Requires DEV_AUTH_ENABLED=true in env
Restricted to DEV_ALLOWED_CIDR at nginx level - never reachable from the internet
Accepts user_id and role query params (default role: user)
Used by the frontend when ?dev_token=<user_id>:<role> is in the URL or VITE_DEV_TOKEN is set in .env.local

Plugin system

Plugins are Python packages declared via entry points:

[project.entry-points."yoink.plugins"]
dl = "yoink_dl:DlPlugin"

Each plugin implements the YoinkPlugin protocol:

Method	Returns	Description
`get_handlers()`	`list[HandlerSpec]`	PTB message/command handlers
`get_inline_handlers()`	`list[InlineHandlerSpec]`	Inline query handlers
`get_routes()`	`APIRouter \| None`	FastAPI routes
`get_models()`	`list`	SQLAlchemy models (concrete classes, not base aliases)
`get_locale_dir()`	`Path \| None`	Directory with `en.yml`, `ru.yml`
`get_jobs()`	`list[JobSpec] \| None`	Scheduled background jobs
`get_web_manifest()`	`WebManifest \| None`	Frontend pages and sidebar entries
`get_commands()`	`list[CommandSpec]`	Bot commands for BotFather menu
`get_features()`	`list[FeatureSpec]`	RBAC features declared by this plugin
`get_help_section()`	`str`	HTML fragment for `/help`
`setup(ctx)`	-	Async startup: init services, populate `bot_data`, register `ActivityProvider`

Activity providers

Plugins register an ActivityProvider callable in setup() via register_activity_provider(). Core collects activity from all providers in collect_activity() to build /users/{id}/stats responses. This keeps core decoupled from plugin internals.

# in plugin setup():
from yoink.core.activity import register_activity_provider
register_activity_provider("dl", dl_activity_provider)

Inline dispatcher

Core registers a single InlineQueryHandler that routes queries to plugins by priority, then prefix, then pattern, then catch-all. If access_policy is set on an InlineHandlerSpec, access is checked before calling the handler - denied users are silently skipped.

Plugin	Priority	Description
yoink-music	10	Music URLs and empty-query hint; returns `False` for non-music
yoink-dl	0	YouTube/URL search catch-all

RBAC

Role hierarchy: banned < restricted < user < moderator < admin < owner

Owner is set via owner_id env var at startup and always has full access.

Features

Plugins declare FeatureSpec objects that describe access-gated capabilities:

FeatureSpec(
    plugin="insight",
    feature="summary",
    label="AI Summary",
    description="Access to /summary and /about commands",
    default_min_role=None,  # None = explicit grant required; "user" = all users by default
)

Access is granted if any of the following is true:

An explicit grant exists in user_permissions (non-expired row).
user.role >= feature.default_min_role (role threshold).
A plugin-registered feature provider returns True for the user.

Owner always passes regardless.

yoink.core.auth.effective_features.EffectiveFeatureResolver is the single source of truth for all three resolution paths. Plugins register additional grant sources at startup via register_feature_provider(plugin, feature, provider). Example from yoink-insight:

from yoink.core.auth.effective_features import register_feature_provider
from yoink_insight.services.byok_access import byok_tldr_provider
register_feature_provider("insight", "tldr", byok_tldr_provider)

All command-menu refresh (refresh_user_commands), /help generation, and PermissionChecker.check calls go through EffectiveFeatureResolver so provider grants are reflected everywhere without additional wiring.

Callers that need to distinguish a real RBAC grant from a side-channel provider grant (e.g. a settings UI that gates gateway-only controls behind an explicit grant, while still letting BYOK-only users edit their language) use resolver.grant_source(user_id, plugin, feature). It returns a GrantSource enum (owner, explicit, role, provider) or None; the boolean is_allowed() is a thin wrapper over grant_source() is not None.

Commands

CommandSpec supports required_feature="plugin:feature" to hide a command from users who don't have access:

CommandSpec(
    command="summary",
    description="Summarize a YouTube video",
    required_feature="insight:summary",
)

Bot command menus are refreshed automatically on role change, grant/revoke, language change, and /start.

Web dashboard

The frontend is a React SPA served by nginx. Built with Vite + Tailwind + shadcn/ui components.

Admin pages

/admin/users - user list; Item list + bottom Drawer with tabs (Stats / Access / Edit); sortable by role, name, dl_count, dl_last_at; user avatars via photo proxy
/admin/groups - group list; Item list + Dialog for editing; group photo via proxy; thread policies via Settings2 icon + ThreadPoliciesDialog; scan button (user-session only)
/admin/permissions - per-feature access matrix (grant/revoke per user)
/admin/bot-settings - accepts plugin-contributed sections via PluginManifest.botSettingsSections

Both list pages support dynamic search with 300 ms debounce (opacity fade, no skeleton flash).

Frontend architecture

frontend/
  src/
    components/
      ui/            # shadcn components (never edited directly); index.ts barrel -> @ui alias
      app/           # app-level components (UserPanel, SettingRow, InlineSelect, StatusBadge); index.ts barrel -> @app alias
      charts/        # StatCard, PeriodToggle, chartColors; index.ts barrel -> @core/components/charts
    lib/
      api/           # typed API modules: users, groups, bot-settings, permissions, user-settings, threads
      api-client.ts  # axios instance with auth interceptor
      user-utils.ts  # userInitials, userPhotoUrl, GRADIENT/RING/roleMediaColor, openProfileLink
      utils.ts       # cn, formatDate, formatDateMonth, formatDateDay, formatBytes
    pages/
      admin/
        users/       # AdminUsersPage + UserDrawer (extracted)
        groups/      # AdminGroupsPage + useAdminGroups hook (extracted)
        ...
    types/
      api.ts         # core API types (User, Group, Feature, Permission, ...)
      plugin.ts      # PluginManifest, UserStats, NavGroup, ...

Aliases: @core/* = frontend/src/*, @ui = frontend/src/components/ui, @app = frontend/src/components/app, @dl/* = plugins/yoink-dl/frontend/src/*, @stats/* = plugins/yoink-stats/frontend/src/*, @insight/* = plugins/yoink-insight/frontend/src/*, @inbox/* = plugins/yoink-inbox/frontend/src/*.

@app components: UserPanel, SettingRow, InlineSelect, StatusBadge, CompactCardHeader, EmptyState, MarkdownBody, PageContainer (max-w-4xl prose wrapper; omit in full-width pages like StarsPage).

Right sidebar slot: AppLayout exposes useRightSidebar() (from @core/layout/AppLayout) that returns setContent(node). Pages that need a page-level right panel (e.g. StarsPage folder navigator) call setContent(<Panel />) in useEffect and setContent(null) in the cleanup. The slot renders a shadcn <Sidebar side="right"> inside the shared SidebarProvider (sidebar-15 pattern); setContent is a stable useCallback to prevent render-loop issues.

Alias resolution: vite.config.ts resolve.alias is the sole runtime resolver (no vite-tsconfig-paths plugin). tsconfig.json paths remain in sync for tsc/editor only.

Page naming: all page files use PascalCase (AdminGroupsPage.tsx, GroupPage.tsx, etc.). Index barrels (index.ts) export the public surface of each feature directory.

Logic hooks: heavy pages extract state/logic into co-located usePage.ts hooks (useAdminGroups, useAdminCookies) keeping JSX thin.

API layer: all apiClient calls go through typed modules in lib/api/ and plugin api/ directories. Pages and hooks import from those modules, not from apiClient directly.

SQL query layer

Raw SQL lives in .sql files, never inline in Python. Loaded at import time via yoink.core.db.query.load_sql().

src/yoink/core/db/
  query.py                  # load_sql(base, name), date_condition(), date_params()
  base.py                   # DeclarativeBase, AuditMixin, SoftDeleteMixin, utcnow()
core/api/routers/queries/
  list_users.sql            # paginated user list with dl stats (LATERAL)
  count_users.sql           # count for the same filtered query

load_sql(base, name) - reads base/name.sql once at module import; result is a plain string passed to text().
date_condition(col) - returns CAST(:since AS timestamptz) IS NULL OR {col} >= :since; eliminates the date_filter = "AND ..." if since else "" pattern.
date_params(since, **kw) - builds {"since": since, ...}; since=None disables the filter without changing query text (stable query plan).

Plugin SQL files: plugins/yoink-stats/src/yoink_stats/queries/*.sql (23 files).

DB view: stats_user_latest_name (migration 0031) - DISTINCT ON (user_id) over stats_user_names ORDER BY date DESC. Replaces repeated LEFT JOIN LATERAL (SELECT ... ORDER BY date DESC LIMIT 1) in ~10 analytics queries.

Database migrations

Single Alembic chain covering core and all plugins:

Migration	Description
0001	users, groups, thread_policies, bot_settings, events
0002	dl settings, file cache, download log, rate limits, cookies
0003	message log, stats tables
0004	stats tsvector full-text index
0005	inline storage settings for groups
0006	user is_premium flag
0007	dl DM topic thread ID
0008	M2M API keys
0009	insight_access table
0010	file_cache multi-file (file_ids JSON)
0011	gallery_zip flag in download_log
0012	unified user_permissions table
0013	insight_user_settings table
0014	clip_start, clip_end, group_title in download_log
0015	cookies.inherited flag
0016	music download_log fields (user_id, group_id, thread_id)
0017	stats ranked list and period fields
0018	users.photo_url
0019	file_cache.cache_key String(80)
0020	cookies.is_pool flag + index
0021	cookies.label
0022	cookies partial unique index (personal)
0023	cookies.avatar_url
0024	cookies.content_hash + index
0025	cookies.session_key + index
0026	dl_user_settings.use_pool_cookies
0027	groups.photo_url
0028	stats_reactions table
0029	stats_group_members table
0030	stats_chat_admins table
0031	stats_user_latest_name view (live, not materialised - source table is write-hot)
0032	stats_messages (chat_id, message_id) index + message_thread_id column
0033	insight_summary_cache table
0034	youtube_auth_mode column in dl_user_settings
0035	insight_summary_cache: video_id renamed to content_key (VARCHAR 512, supports non-YouTube URLs)
0036	insight_user_settings: tldr_model column (per-user LLM override for /tldr)
0037	insight_user_settings: github_token column (GitHub URL support in /tldr)
0038	dl_user_settings: audio_codec column
0039	insight_tldr_aliases table (initial: user-defined /tldr prompt aliases)
0040	insight_tldr_aliases: rename alias to aliases column
0041	insight_tldr_aliases: domain bindings and binding-to-built-in rows
0042	insight_usage_log: content_chars, video_seconds, alias_key columns (TLDR metrics)
0043	insight_user_prompts table (per-user default prompt overrides per command)
0044	insight_user_settings: use_search column (AI Search toggle)
0045	insight_user_byok table (per-user Bring-Your-Own-Key configuration)

Legacy custom Bot API server

The explicit compatibility build uses tdlight-telegram-bot-api with docker/patches/tdlight-forum-extras.patch. It adds methods for forum topics, message viewers and chat history. This dependency is not part of the Rust release build or GHCR image set.

just build tg   # explicit legacy TDLight source build

Current server version: 9.5.

User-mode session

The bot API server runs with --allow-users, enabling both bot and user accounts. The user session unlocks API methods unavailable to bots (forum topic listing, chat history, etc). It is strictly optional - all core functionality works without it; user-session features are only shown in the UI when GET /threads/status returns available: true.

just tg login +79001234567
just tg status
just tg logout

Token stored in data/tg-bot-api/user.token.

YouTube TV OAuth

The native YouTube container combines youtubei.js, bgutil and the Unix authentication adapter. The youtubei provider handles requests from users authorized through the Google TV device flow. Build the container with just build gateway; yoink_youtube_image pins its immutable SemVer image. See native deployment for lifecycle and isolation details.

The youtubei provider is only used when a user has completed OAuth authorization and switched their YouTube auth method to "YouTube TV OAuth" in Settings. All other users continue to use yt-dlp with Netscape cookies by default. See plugins/yoink-dl/README.md for the full setup and flow description.

Cookie extraction

just browser up   # start Kasmweb Chromium on port 6902
# log in to sites in the browser
just browser down

Cookies written to data/cookies/ and picked up by yt-dlp. Alternatively, use the browser extension to sync cookies directly from your own browser.

Backup

Requires backup_s3_* env vars. pg_dump with custom format, uploaded to S3 via mc.

just backup           # one-shot
just backup restore list  # list available backups
just backup restore       # restore latest
just backup up        # start cron (daily at 03:00)

Retention: 7 daily + 4 weekly.

Development

Each plugin pyproject has a [tool.pyrefly] section with python-interpreter-path pointing at ../../.venv and a search-path listing sibling plugin src/ directories. This lets Pyrefly resolve cross-plugin imports without a monorepo-wide config. Run pyrefly check from any plugin directory to get project-scoped type errors.

Submodule heads: run just submodules (wraps git submodule status). Refresh the core repo and every submodule with just pull (runs git pull then git submodule update --remote --merge).

Tests

just test                        # core tests
cd plugins/yoink-dl && just test
cd plugins/yoink-stats && just test

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
backend		backend
browser-extension		browser-extension
docker		docker
docs		docs
frontend		frontend
plugins		plugins
scripts		scripts
src		src
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
.gitleaksignore		.gitleaksignore
.gitmodules		.gitmodules
README.md		README.md
docker-compose.legacy.yaml		docker-compose.legacy.yaml
docker-compose.yaml		docker-compose.yaml
justfile		justfile
pyproject.toml		pyproject.toml

Navigation Menu

Repository navigation

About

Repository files navigation

yoink-core

Legacy Python entrypoints

Quick start

Services

Just recipes

Environment variables

REST API

Core endpoints

M2M API

Health

Dev authentication

Plugin system

Activity providers

Inline dispatcher

RBAC

Features

Commands

Web dashboard

Admin pages

Frontend architecture

SQL query layer

Database migrations

Legacy custom Bot API server

User-mode session

YouTube TV OAuth

Cookie extraction

Backup

Development

Tests

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

yoink-core

Legacy Python entrypoints

Quick start

Services

Just recipes

Environment variables

REST API

Core endpoints

M2M API

Health

Dev authentication

Plugin system

Activity providers

Inline dispatcher

RBAC

Features

Commands

Web dashboard

Admin pages

Frontend architecture

SQL query layer

Database migrations

Legacy custom Bot API server

User-mode session

YouTube TV OAuth

Cookie extraction

Backup

Development

Tests

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages