External Services & Dependencies¶

Every service, queue, and RPC channel the Evospin stack talks to — infrastructure, third-party, and internal microservice boundaries. Use this page when onboarding, debugging connectivity, or planning environment provisioning.

Quick-reference table¶

Service	Protocol	Port (local)	Local mode	Required for sign-in?	Required for betting?
Postgres	TCP (libpq)	5555 → 5432	Real	Yes	Yes
Redis "cache"	RESP	6379	Real	Yes (session)	Yes (BullMQ)
Redis "bot"	RESP	6380	Real	No	No (bots only)
RabbitMQ	AMQP 0-9-1	5672 / 15672 (UI)	Stubbed (zero traffic)	No	No
reCAPTCHA v3	HTTPS	—	Bypass (`"pass"`)	No (bypass active)	No
Fast Track CRM	AMQP (stub)	—	Stubbed (11 silent drops)	No	No
EOS blockchain	HTTPS (JSON-RPC)	—	Real (public nodes)	No	Yes (speed-roulette RNG)
EVO wallet	Redis pub/sub RPC	—	Real (internal)	No	Yes (bj / speed-roulette)
Skindeck	HTTPS + webhook	—	Real (needs API key)	No	No (deposits only)
Sentry	HTTPS	—	Disabled (`NODE_ENV=local`)	No	No
SendGrid	HTTPS (API)	—	Bypass (`isLocal`)	No	No
OTel Collector	OTLP/HTTP + gRPC	4318 / 4317	Real (self-hosted)	No	No
Redis pub/sub RPC	RESP (pub/sub)	6379	Real	Yes (auth gateway)	Yes (wallet RPC)
Socket.IO (rt)	WebSocket	4001	Real	No	Yes (live events)
BullMQ (13 queues)	RESP (via ioredis)	6379 / 6380	Real	Yes (session queue)	Yes (bet settlement)

Infrastructure (local + production)¶

Postgres 15¶

Purpose. Single relational database for the entire stack. Hosts three Prisma schemas (public, blackjack, speed_roulette) in one database named ebit. All five NestJS apps connect via Prisma Client.

Wire protocol. TCP (libpq), port 5432 inside Docker, host-mapped to 5555.

Integration points.

Prisma schema files: ebit-api/libs/_prisma/src/schema/{api,blackjack,speed_roulette}.prisma
Connection URL env var: DATABASE_URL
Docker service: ebit-db (docker-compose.yml:7–28)
Healthcheck: pg_isready -U ebit -d ebit every 5 s, 10 retries

Local dev mode. Real — runs in Docker. Seeded via npm run db:seed from libs/_prisma/src/seed/index.ts.

Production credentials. Managed in Doppler. Format: postgresql://<user>:<pass>@<host>:5432/ebit?schema=public.

Failure handling. Prisma throws PrismaClientKnownRequestError on connection loss. NestJS health module checks DB status at /health. No circuit breaker — app crashes on sustained Postgres unavailability (Nest health probe fails → container restart via restart: unless-stopped).

Observability. Prisma instrumentation emits prisma:client:operation, prisma:engine:db_query, prisma:engine:serialize spans. The OTel Collector's spanmetrics connector derives calls_total and duration_milliseconds_bucket histograms. Dashboard: Evospin · Prisma / Postgres in Grafana.

Redis "cache" (port 6379)¶

Purpose. Primary data-plane Redis used for BullMQ job queues (10 queues), session caching, live-bets feed (LPUSH/LRANGE), exchange-rate cache, challenge state, and chat data. Also backs the @ExternalControllerClient Redis pub/sub RPC transport between NestJS apps.

Wire protocol. RESP (Redis Serialization Protocol), port 6379. Password: cache.

Integration points.

Connection env var: REDISCLOUD_URL (e.g., redis://:cache@ebit-redis:6379/0)
Fallback config: ebit-api/libs/shared/src/common/utils/utils.ts:49–60 (getRedisConfig()) reads REDIS_HOST, REDIS_PORT, REDIS_PASSWORD, REDIS_DB
BullMQ registration: ebit-api/apps/api/src/app.module.ts:137–143 (BullModule.forRoot(getRedisConfig()))
Docker service: ebit-redis (docker-compose.yml:30–46, image redis/redis-stack:latest)
Healthcheck: redis-cli -a cache --no-auth-warning ping | grep -q PONG

Local dev mode. Real — runs in Docker. Pre-populated by npm run db:seed (some seed scripts write cache entries).

Caching subsystems.

Subsystem	Key pattern	File
Auth sessions	`USER_AUTH_SESSION_CACHE_PATTERN`	`apps/api/src/auth/session/session.service.ts`
Live bets feed	`CACHE_FEEDS_PREFIX` (LPUSH/LRANGE)	`apps/api/src/bet/live/live-bets-cache.repository.ts`
Exchange rates	key per currency pair	`apps/api/src/exchange-rates/exchange-rates.repository.ts`
Challenge state	challenge scan keys	`apps/api/src/challenge/repository/cache/challenge-scan.cache.ts`
Chat data	room/message keys	`apps/api/src/chat/chat-data.service.ts`

Failure handling. ioredis auto-reconnects with exponential backoff. BullMQ workers pause on Redis disconnect and resume on reconnect. Session cache miss falls through to Postgres lookup.

Observability. ioredis auto-instrumentation emits per-command spans (get, set, expire, evalsha, etc.). Spanmetrics connector derives histograms. Dashboard: Evospin · Redis (ioredis) in Grafana.

Redis "bot" (port 6380)¶

Purpose. Dedicated Redis instance for bot simulation infrastructure. Isolates bot BullMQ queues and bot session state from production traffic on the cache instance.

Wire protocol. RESP, port 6380 on host (6379 inside container). Password: bot.

Integration points.

Connection env var: REDISCLOUD_BOT_URL (e.g., redis://:bot@ebit-redis-bot:6379)
Fallback config: ebit-api/libs/shared/src/common/utils/utils.ts:62–73 (getRedisBotConfig()) reads REDIS_BOT_HOST, REDIS_BOT_PORT, REDIS_BOT_PASSWORD, REDIS_BOT_DB
Docker service: ebit-redis-bot (docker-compose.yml:48–64)
Healthcheck: redis-cli -a bot --no-auth-warning ping | grep -q PONG

BullMQ queues on bot Redis.

Queue	Purpose	Schedule
`bots-session-scheduler`	Daily cron — enqueues bot sessions	`0 0 * * *` UTC
`bots-start-session`	Initiates a bot gaming session	On-demand
`bots-bet`	Executes individual bot bets	Delayed per strategy
`challenges`	Scans active challenges every 15 min	`/15 * * *` UTC

Direct ioredis clients (not BullMQ).

BotRedisClient (apps/api/src/bots/system/bot-session/redis/redis.client.ts) — session storage via HSET/HGETALL/EXPIRE/DEL
ValKeyRedisClient (libs/shared/src/redis/valkey-redis.client.ts:8–28) — challenge cache scans

Failure handling. Same ioredis reconnect behaviour as cache Redis. Bot queues are non-critical — failures do not affect real user traffic.

Observability. Same ioredis span instrumentation as cache Redis. Spans carry net.peer.port=6380 to distinguish from cache.

RabbitMQ (AMQP)¶

Purpose. Originally provisioned for the Fast Track CRM integration. Currently receives zero traffic — the Fast Track module is hardcoded to disabled = true.

Wire protocol. AMQP 0-9-1, port 5672. Management UI on 15672. Credentials: rabbitmq:rabbitmq. Vhost: ft.

Integration points.

Connection env var: BROKER_URI (e.g., amqp://rabbitmq:rabbitmq@ebit-rabbitmq:5672)
Fast Track stub: apps/api/src/fast-track/rabbitmq/fast-track.rmq.module.ts:8 — const disabled = true
Docker service: ebit-rabbitmq (docker-compose.yml:91–115, image rabbitmq:3.7-management)
Healthcheck: rabbitmq-diagnostics -q ping

Local dev mode. Runs in Docker but receives zero traffic. Management UI at http://localhost:15672 shows empty queues.

Stub behaviour. When disabled = true, the module registers a stub provider: - publish() → undefined - emitEvent() → undefined - sendMessage() → { success: false }

11 call sites silently drop into the stub (bet settlement events in bet.service.ts lines 398/464/500/591, promo bonus events in promo-effect.service.ts lines 133/221/277/350/420/461/487).

Failure handling. N/A — stub swallows all calls. If enabled, errors are caught and logged via EvoLogger.error without propagating.

Observability. None — no traffic flows. If enabled, @golevelup/nestjs-rabbitmq would emit ioredis-style connection spans but no AMQP-specific instrumentation exists.

Third-party services¶

reCAPTCHA v3 (Google)¶

Purpose. Anti-bot protection on auth endpoints. Validates a client-side token on rate-limit violation.

Wire protocol. HTTPS POST to https://www.google.com/recaptcha/api/siteverify.

Integration points.

Service: apps/api/src/captcha/google/recaptcha.service.ts
Guard: apps/api/src/captcha/google/recaptcha.guard.ts — triggers on rate-limit via @ThrottleRecaptcha() decorator
Token header: x-captcha-token
Idempotency lock: recaptcha:${sha256(token)} with 5 s TTL (prevents replay)
Feature flag: disable_captcha can disable globally

Local dev mode. Bypass active — recaptcha.service.ts:28: if isLocal && token === 'pass', validation returns immediately.

Production credentials. Google Cloud Console → reCAPTCHA Enterprise. Env var: RECAPTCHA_SECRET.

Failure handling.

Missing token → ApiException(AUTH_RECAPTCHA_TOKEN_MISSING)
Failed validation → ApiException(AUTH_INVALID_CAPTCHA)
Google API unreachable → exception propagates as 500

Observability. HTTP client span for the siteverify POST. Logged via EvoLogger on failure.

Fast Track CRM¶

Purpose. Real-time player engagement CRM. Designed to emit casino transaction and bonus events for player lifecycle tracking. Currently fully stubbed — no data leaves the application.

Wire protocol. AMQP via RabbitMQ (when enabled). Queue: rtevents, vhost: ft.

Integration points.

Module: apps/api/src/fast-track/ (26+ files)
Disable flag: fast-track.rmq.module.ts:8 — const disabled = true
Casino service: FastTrackCasinoService.emitTransaction() — called from bet settlement
Bonus service: FastTrackBonusService.emitBonus() — called from promo effects
Config: fast-track.config-service.ts reads FAST_TRACK_API_KEY, FAST_TRACK_RABBITMQ_URL, FAST_TRACK_RABBITMQ_QUEUE_NAME, FAST_TRACK_REGISTER_CONSUMERS

Local dev mode. Stubbed. 11 call sites silently return undefined or { success: false }.

Production credentials. TBD — product input needed. Fast Track provides an API key and RabbitMQ broker endpoint for production environments.

Failure handling. Errors caught in service methods, logged via EvoLogger.error, not propagated to callers. Casino/bonus event emission is fire-and-forget.

Observability. None — stub produces no spans or logs beyond the initial module-load message.

EOS blockchain¶

Purpose. Provides block IDs used as verifiable entropy for speed-roulette RNG. Each game round commits to a future EOS block number; when that block is produced, its ID seeds RngGames.getRandomSpeedRoulette().

Wire protocol. HTTPS JSON-RPC to EOS API nodes.

Integration points.

State service: apps/speed-roulette/src/roulette/state/roulette-state.service.ts:85 — eosBlock = await this.eosService.waitForBlock(model.game.eosBlockNum, ...)
RNG call: roulette-state.service.ts:141 — RngGames.getRandomSpeedRoulette({secret, eosBlockId})
EOS service (injected as DIEosService): methods getCurrentBlock(), getFutureBlockNum(delaySeconds), waitForBlock(blockNum, timeoutMs)
Cache keys (apps/bj/src/shared/constants.ts): eos:current_block, eos:future_block_num, eos:block_history

Local dev mode. Real — hits public EOS nodes. No stub; speed-roulette rounds require live block data.

Production credentials. Env vars: - EOS_NODE_HOST_1 — primary node (default: https://eos.nownodes.io) - EOS_NODE_AUTH_HEADER_1 — auth header for primary (format: login:pass) - EOS_NODE_HOST_2 — fallback node (default: https://eos.greymass.com) - EOS_NODE_AUTH_HEADER_2 — auth header for fallback (typically empty)

Failure handling. waitForBlock() throws on timeout — the game round cannot settle without a valid block ID. Speed-roulette state machine transitions to an error state and refunds pending bets via the RollbackBetsJob queue.

Observability. HTTP client spans for each EOS API call. Block wait duration visible in speed-roulette trace waterfalls.

EVO wallet RPC¶

Purpose. Internal wallet/accounting gateway for game sessions. Handles bet deductions and win credits for blackjack and speed-roulette. Despite the name, this is not an external API — it's an internal NestJS service called via @ExternalControllerClient Redis pub/sub RPC.

Wire protocol. Redis pub/sub (via @ExternalControllerClient proxy). Events: GATEWAY_API_EVENTS.Private.EvoGamesPlay, GATEWAY_API_EVENTS.Private.EvoGamesRollback.

Integration points.

Wallet service: apps/api/src/casino/slots/providers/evogames/wallet/evogames.wallet.service.ts
Gateway controller: apps/api/src/casino/slots/providers/evogames/wallet/evogames.wallet.gateway.controller.ts
Callers:
apps/speed-roulette/src/bet/bet.service.ts — @ExternalControllerClient(EvoGamesWalletGatewayController)
apps/bj/src/bets/bets.service.ts — same decorator

Request/response shape.

PlayRequestDto — user_id, currency, game, game_id, finished, actions[] (bet/win with amount + action_id)
RollbackRequestDto — user_id, currency, game, game_id, actions[]
Response: { transactions[], game_id, balance }

Local dev mode. Real — runs as part of the ebit-api process. No external dependency.

Failure handling.

BET_IS_NOT_FOUND — silently ignored (idempotent retry)
ACCOUNTING_BALANCE_INSUFFICIENT — throws EvoGamesException with PLAYER_HAS_NOT_ENOUGH_FUNDS, HTTP 412

Observability. Redis pub/sub RPC calls produce ioredis spans but lack W3C traceparent propagation — the Redis transport does not inject trace context into the message envelope. This is a known gap (see docs/architecture.md).

Skindeck (skin deposits)¶

Purpose. Skin-trading deposit provider. Users deposit CS2 skins via Skindeck's marketplace; the platform credits their balance after trade confirmation.

Wire protocol. HTTPS REST API + inbound webhook.

Integration points.

API host: https://api.skindeck.com (constant in skindeck.const.ts)
Endpoints called:
GET /client/inventory — user's skin inventory
POST /auth/authenticate-client — client auth
POST /client/trading/deposit — create deposit trade
Controller: apps/api/src/payment/provider/integration/skindeck/skindeck.controller.ts
Webhook controller: skindeck-webhook.controller.ts — guarded by SkinDeckCallbackGuard (HMAC signature validation using SKINDECK_API_SECRET)
BullMQ queue: SKINDECK_DEPOSIT — polls trade status every 30 s, concurrency 10, 3 attempts with 10 s backoff

Local dev mode. Real — requires valid SKINDECK_API_KEY and SKINDECK_API_SECRET. Without keys, deposit endpoints return errors but don't crash the app.

Production credentials. Skindeck partner dashboard. Env vars: SKINDECK_API_KEY (HTTP header api-key), SKINDECK_API_SECRET (webhook HMAC).

Failure handling.

Webhook signature mismatch → 403 "Invalid hash for SkinDeck webhook payload"
Job failure after 3 retries → job kept for audit (removeOnFail: false)
API errors logged via EvoLogger

Observability. HTTP client spans for Skindeck API calls. BullMQ job lifecycle visible in bullmq_queue_jobs{queue="SKINDECK_DEPOSIT"}.

Sentry (error tracking)¶

Purpose. Application performance monitoring and error tracking. Captures unhandled exceptions and HTTP 5xx errors.

Wire protocol. HTTPS to Sentry ingest endpoint (SaaS or self-hosted).

Integration points.

Init: libs/shared/src/basic/pre/pre-sentry.main.ts:10–17
Enable gate: isSentryEnabled = !!sentryDsn && env.NODE_ENV !== 'local' (line 9)
Error filter: libs/shared/src/api/filters/sentry.filter.ts — captures status >= 500 and all non-HTTP exceptions
Integrations: nodeProfilingIntegration(), Sentry.prismaIntegration()
Trace sample rate: 1.0, profile sample rate: 1.0

Local dev mode. Disabled — NODE_ENV=local short-circuits initialization regardless of DSN presence.

Production credentials. Per-service DSN env vars (pattern: SENTRY_DSN_${APP}): - SENTRY_DSN_API, SENTRY_DSN_BJ, SENTRY_DSN_BO, SENTRY_DSN_RT, SENTRY_DSN_SPEED_ROULETTE

Source-map upload via SENTRY_AUTH_TOKEN, SENTRY_ORG, SENTRY_PROJECT at build time.

Failure handling. Sentry SDK swallows its own transport errors — if Sentry is unreachable, the app continues without error reporting.

Observability. Sentry traces are independent from the OTel pipeline. Both run concurrently — Sentry for error aggregation + alerting, OTel for distributed tracing + metrics.

SendGrid (email)¶

Purpose. Transactional email delivery for email verification, password reset, welcome messages, and deposit confirmations.

Wire protocol. HTTPS (SendGrid REST API via @sendgrid/mail library).

Integration points.

Engine: apps/api/src/external-notification-sender/mail_engine/sendgrid/engine.service.ts — this.engine.setApiKey(config.SENDGRID_API_KEY)
Dispatcher: apps/api/src/external-notification-sender/email-sender.service.ts:63–65 — isLocal bypass
Profile service: profile.email.service.ts — resetPassword(), verificationEmail()
Call sites in apps/api/src/user/user.service.ts:
sendEmailVerificationLink() — verification email
resetUserPassword() — password reset link (${fe_url}?modal=restore-password&token=${token})
notifyUserRegistration() — welcome email

Local dev mode. Bypass — email-sender.service.ts:63: if isLocal, sendEmail() returns immediately without calling SendGrid.

Production credentials. SendGrid dashboard. Env vars: - SENDGRID_API_KEY — API authentication - SENDGRID_FROM_EMAIL — sender address (default: no-reply@playebit.com) - SENDGRID_VERIFY_EMAIL_TEMPLATE_ID, SENDGRID_WELCOME_EMAIL_TEMPLATE_ID, SENDGRID_RESET_PASSWORD_EMAIL_TEMPLATE_ID, SENDGRID_DEPOSIT_SUCCESSFUL_EMAIL_TEMPLATE_ID

Failure handling. SendGrid errors throw ApiException(MAILER_MAIL_NOT_SENT). Callers catch and log — email failure does not block the user action (fire-and-forget for verification/welcome; password reset propagates to show a user-facing error).

Observability. HTTP client span for each SendGrid API call. No custom metrics.

OTel Collector (self-hosted)¶

Purpose. Central telemetry gateway. Receives OTLP from all NestJS apps and ebit-fe browser, exports traces to Jaeger, metrics to Prometheus, and logs to Loki. Runs the spanmetrics connector to derive RED histograms from spans.

Wire protocol. OTLP/HTTP on port 4318, OTLP/gRPC on 4317. Prometheus scrape on 8889.

Integration points.

Config: observability/otel-collector.yml
Docker service: otel-collector in docker-compose.yml:522–548 (image otel/opentelemetry-collector-contrib:0.96.0, runs as user: "0:0")
App env vars: OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318, OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
Browser env vars: NEXT_PUBLIC_OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

Pipeline summary.

Signal	Receivers	Processors	Exporters
Traces	otlp	memory_limiter, batch	otlphttp/jaeger, spanmetrics
Metrics	otlp, spanmetrics	memory_limiter, batch	prometheus
Logs	otlp, filelog/docker	memory_limiter, batch	loki

Local dev mode. Real — runs in Docker. CORS configured for localhost:3000/3001/3003.

Failure handling. Healthcheck at / on port 13133. Memory limiter (512 MiB limit, 128 MiB spike) drops telemetry under memory pressure.

Observability. Self-metrics on port 8888 (internal). Prometheus scrapes the external metrics port 8889.

Internal microservice communication¶

Redis pub/sub RPC (`@ExternalControllerClient`)¶

Purpose. Synchronous request-response RPC between NestJS apps over Redis pub/sub. Used for cross-app method calls (e.g., speed-roulette calling ebit-api's wallet service).

Wire protocol. Redis pub/sub via NestJS ClientProxy with Transport.REDIS. Default timeout: 5000 ms.

Integration points.

Decorator: libs/gateway/src/ms-controller/external-controller-client.decorator.ts
Proxy factory: libs/gateway/src/ms-controller/gateway-client.factory.ts:27–34
Client module: libs/gateway/src/gateway-client.module.ts:18 — transport: Transport.REDIS
Server module: libs/gateway/src/gateway-server.module.ts:7–24
Events: apps/rt/src/gateway/events.ts — CORE_CLIENT_EVENTS (ServerError, TimeoutError, AuthError, AuthSuccess, Unauthorized) and GATEWAY_METHODS (Authorization)

Known limitation. Redis pub/sub transport does not propagate W3C traceparent headers. Cross-app RPC calls break the distributed trace — the callee starts a new trace root. This is documented in docs/architecture.md.

Socket.IO WebSocket gateway (rt service)¶

Purpose. Real-time event delivery to player browsers. Pushes game state updates, live bets, chat messages, notifications, and speed-roulette round events.

Wire protocol. WebSocket (Socket.IO, namespace /events, transport websocket only — HTTP long-polling disabled).

Integration points.

Gateway: apps/rt/src/gateway/client.gateway.ts:48–52
Auth: socket socket_token query param → AuthService.authorizeConnection()
Port: 4001 (env var PORT_RT)
CORS origins: APP_FE_ORIGIN (default http://localhost:3000) + https://admin.socket.io in dev

Client connection (ebit-fe).

socket.io-client with transports: ['websocket']
Auth flow: connect → send socket_token → receive AuthSuccess or AuthError + disconnect

Channel events (from ebit-fe subscribe).

Speed-roulette join/leave, live drops, chat, live bets, notifications — full list in client.gateway.ts handleConnection().

BullMQ queues (13 total)¶

All queues use BullMQ (Redis-backed via @nestjs/bullmq). Enqueue operations appear in OTel traces as ioredis EVALSHA spans (Lua scripts). The bullmq_queue_jobs{queue, state} gauge in Prometheus tracks queue depth; no per-job duration histograms exist today (@opentelemetry/instrumentation-bullmq is not in the auto-instrumentation set).

Cache Redis queues (port 6379)¶

Queue name	Owner app	Purpose	Concurrency	Retries	Backoff
`update-session`	api	Update user session metadata (geo, IP, UA)	1	3	1 s exp
`bet_settled_queue`	api	Bet settlement side-effects (leaderboard, rakeback, affiliate, fast-track emit)	2	10	500 ms exp
`update-user-stats`	api	Increment user statistics counters	1	8	1 s exp
`migrate-user-stats`	api	Batch migration of historical user stats	1	1	—
`leaderboard_queue`	api	Update leaderboard rankings and prize payouts	3	10	500 ms exp
`SKINDECK_DEPOSIT`	api	Poll Skindeck trade status every 30 s	10	3	10 s fixed
`promo-expired`	api	Handle promo code expiration	1	3	1 s exp
`speed-roulette-bet-queue`	speed-roulette	Settle/rollback speed-roulette bets	3	10	2 s exp
`speed-roulette-state-queue`	speed-roulette	Game state machine transitions	1	—	—

Bot Redis queues (port 6380)¶

Queue name	Owner app	Purpose	Schedule
`bots-session-scheduler`	api	Daily cron — enqueue bot sessions	`0 0 * * *` UTC
`bots-start-session`	api	Start a bot gaming session, schedule first bet	On-demand
`bots-bet`	api	Execute individual bot bet actions	Delayed per strategy
`challenges`	api	Scan active challenges for progress	`/15 * * *` UTC

Key files.

Queue	Producer	Processor
`update-session`	`apps/api/src/auth/session/session.queue-producer.ts`	`session.update.queue-processor.ts`
`bet_settled_queue`	`apps/api/src/bet/queue/bet.queue-producer.ts`	`bet.queue-processor.ts`
`update-user-stats`	`apps/api/src/user/stats/user-stats.queue-producer.ts`	`user-stats.queue-processor.ts`
`migrate-user-stats`	`apps/api/src/user/stats/migrate/user-stats-migrate.queue-producer.ts`	`user-stats-migrate.queue-processor.ts`
`leaderboard_queue`	`apps/api/src/leaderboard/leaderboard.queue-producer.ts`	`leaderboard.queue-processor.ts`
`SKINDECK_DEPOSIT`	`apps/api/src/payment/provider/integration/skindeck/skindeck.module.ts`	`skindeck-deposits.service.ts`
`promo-expired`	`apps/api/src/promo/tasks/task.module.ts`	`expired-promo.worker.ts`
`speed-roulette-bet-queue`	`apps/speed-roulette/src/bet/bet.module.ts`	`bet-queue.processor.ts`
`speed-roulette-state-queue`	`apps/speed-roulette/src/roulette/roulette.module.ts`	`roulette-state.processor.ts`
`bots-*` (3 queues)	`apps/api/src/bots/system/bull/bull.module.ts`	`bots-*.processor.ts`
`challenges`	`apps/api/src/challenge/bullmq/challenge-bull.module.ts`	`challenge-scan.processor.ts`

Debugging stuck jobs. Connect to the relevant Redis instance and inspect BullMQ keys:

redis-cli -a cache -p 6379   # cache queues
redis-cli -a bot -p 6380     # bot queues
KEYS bull:*                   # list all BullMQ key namespaces
LRANGE bull:<queue>:wait 0 -1 # waiting job IDs
HGETALL bull:<queue>:<jobId>  # job payload + metadata

See also: docs/runbooks/bullmq-stuck-job.md for the full troubleshooting runbook.

External Services & Dependencies¶

Quick-reference table¶

Infrastructure (local + production)¶

Postgres 15¶

Redis "cache" (port 6379)¶

Redis "bot" (port 6380)¶

RabbitMQ (AMQP)¶

Third-party services¶

reCAPTCHA v3 (Google)¶

Fast Track CRM¶

EOS blockchain¶

EVO wallet RPC¶

Skindeck (skin deposits)¶

Sentry (error tracking)¶

SendGrid (email)¶

OTel Collector (self-hosted)¶

Internal microservice communication¶

Redis pub/sub RPC (@ExternalControllerClient)¶

Socket.IO WebSocket gateway (rt service)¶

BullMQ queues (13 total)¶

Cache Redis queues (port 6379)¶

Bot Redis queues (port 6380)¶

Redis pub/sub RPC (`@ExternalControllerClient`)¶