Skip to content

ADR-0003 — BullMQ for production async; RabbitMQ kept but stubbed

Status: Accepted Date: 2026-04-16 (original); expanded 2026-04-25 Author(s): Platform engineering

Context

ebit-api/ ships with two message-broker integrations:

  • BullMQ (@nestjs/bullmq) — Redis-backed task queues, in production use across the entire codebase.
  • RabbitMQ (@golevelup/nestjs-rabbitmq) — AMQP broker, present in docker-compose.yml, present in module imports, but wired to a stub (disabled = true at apps/api/src/fast-track/rabbitmq/fast-track.rmq.module.ts:8).

The duality is historical, not by design. RabbitMQ was introduced for a planned Fast Track CRM integration that would emit transactional and bonus events to an external CRM via AMQP. The integration was never completed; the producer was stubbed pending product decision; the broker stayed in the compose so the import graph wouldn't fail. Eight months on, the stub is unchanged and BullMQ has absorbed every async use case that emerged in the meantime.

This ADR codifies the decision so future engineers — including new ones who see two brokers and assume both carry traffic — don't burn time debugging an empty RabbitMQ vhost.

Inventory of async work in the codebase

All of the following ride BullMQ on the cache Redis instance (port 6379):

Domain Producer Consumer
Auth session updates apps/api/src/auth/session/session.queue-producer.ts session worker
Bets settlement apps/api/src/bet/queue/bet.queue-producer.ts bet.queue-processor.ts
Bot activity apps/api/src/bots/system/bull/ bot scheduler
Challenge progress apps/api/src/challenge/... challenge worker
Leaderboard updates apps/api/src/leaderboard/... leaderboard worker
Promo tasks apps/api/src/promo/... promo worker
User stats migration apps/api/src/user/... one-shot
Skindeck deposits apps/api/src/payment/provider/integration/skindeck/... settlement worker
Speed-roulette state apps/speed-roulette/src/... (state queue) per-table worker
Speed-roulette bets apps/speed-roulette/src/... (bet queue) settlement

All of the following ride the RabbitMQ stub (no traffic ever leaves the process):

  • apps/api/src/bet/bet.service.ts × 4 call sites — emitTransaction(...).
  • apps/api/src/promo/promo-effect.service.ts × 7 call sites — emitBonus(...).

Both classes of call site invoke methods on FastTrackRabbitMQProducer. The provider is replaced with a Stub at module-register time; every call resolves to a no-op promise.

Decision

  1. BullMQ is the canonical production async transport. New async work goes here; do not add to the RabbitMQ producer.
  2. RabbitMQ stays in compose because removing it would require coordinated source changes across the FastTrackRabbitMQModule import + the 11 call sites + BROKER_URI env handling. That refactor is reserved for the day product confirms either (a) Fast Track is permanently abandoned, or (b) Fast Track is re-enabled and the stub flips to real.
  3. The stub is documented, not hidden. The disabled = true flag at the top of fast-track.rmq.module.ts:8 is read on every container start. A comment block above ebit-rabbitmq: in docker-compose.yml explains "this broker boots empty by design."
  4. OTel traces show BullMQ enqueue as ioredis EVALSHA spans — they do not appear as a separate broker hop. Operators chasing a "missing broker hop" are tracing this correctly; BullMQ rides Redis Lua scripts.
  5. Cross-job traceparent is not propagated. The BullMQ bet-settled consumer (and others) start an orphan trace because the producer doesn't write traceparent into the job payload. Documented limitation; correlate by user/bet id when needed. See docs/audits/perf-trace-coverage-audit.md and project_otel_microservice_transport_gap memory.

Considered alternatives

A. Remove RabbitMQ from compose entirely

The simplest cleanup. Rejected because:

  • The FastTrackRabbitMQModule is still imported by apps/api/src/app.module.ts.
  • Removing the broker without also removing the module would surface as Nest startup failures (the module reads BROKER_URI even when disabled — the URI must resolve to a host that responds, even if the producer is stubbed).
  • 11 call sites would need refactoring to drop the emitTransaction / emitBonus invocations or replace them with no-ops not gated by the module.
  • This is a real refactor, not a compose tweak; it stays on the backlog until a product owner explicitly retires Fast Track.

B. Wire up a real Fast Track sandbox in dev

Replace the stub with the real producer pointing at a Fast Track-supplied sandbox. Rejected for now because:

  • Requires external coordination (Fast Track-issued sandbox URL + JWT keypair). The compose env values FASTTRACK_JWT_PRIVATE_KEY / FASTTRACK_JWT_PUBLIC_KEY are placeholders ("local-stub-not-used").
  • No business case to do this absent a product decision to re-launch the integration.
  • Re-evaluate if product confirms Fast Track returns; until then the stub is correct behaviour.

C. Delete the Fast Track module entirely

Tear out the 11 call sites + the module + the broker. Rejected because:

  • It's a destructive change. Restoring Fast Track later means re-introducing 11 call sites in their correct semantic positions across bet and promo flows. Risk of getting the semantics wrong on re-introduction.
  • The cost of the stub today is ~300 MB idle RAM and a small amount of confusion mitigated by this ADR. Not worth a destructive change.

D. Migrate the Fast Track stub to BullMQ

Use BullMQ for what the stubbed RabbitMQ producer would have emitted. Rejected because:

  • Fast Track expects AMQP semantics (topic exchanges, headers exchanges, durable bindings) that BullMQ doesn't model natively. A migration would mean re-encoding the routing logic in the BullMQ worker.
  • If Fast Track is ever turned on, AMQP-shaped routing is the native fit. Pre-migrating to BullMQ would either need a second migration back to RabbitMQ at that point, or commit the team to building a BullMQ-shaped adapter for Fast Track's CRM. Neither is a clear win.

E. Pick a third broker (NATS / Kafka)

Replace BullMQ with a "real" message broker for production async work. Rejected because BullMQ is sufficient at our scale, the operational tooling (redis-cli, BullBoard) is well-understood by the team, and migrating away from BullMQ would be a multi-month project against zero current pain.

Consequences

Operational

  • Debug stalled jobs in Redis, not RabbitMQ: 

    redis-cli -a cache KEYS 'bull:*'                  # list queues
redis-cli -a cache LLEN bull:bet-settle:wait      # pending in queue
    The RabbitMQ management UI (http://localhost:15672, rabbitmq/rabbitmq) will always show vhost ft empty — that's by design.

  • BullBoard (or @nestjs/bullmq UI add-on) is the right inspection tool for queue depth, retries, failures.

  • Idle RAM cost of the RabbitMQ container is ~300 MB. Tolerable on 16 GB hosts; noticeable on 8 GB. Local dev can docker compose stop ebit-rabbitmq without breaking anything (the producer is stubbed), but docker compose down and back up restores it.

Observability

  • BullMQ enqueue → ioredis EVALSHA in OTel traces. Operators searching Jaeger for a "broker hop" between producer and consumer will not find one. Cross-link docs/engineering/observability-runbook.md §5 ("Cross-service tracing gotchas").
  • BullMQ consumer starts an orphan trace. No traceparent is in the job payload, so the consumer's root span is not linked to the producer's. Same workaround as the Redis pub/sub case (ADR-0005): correlate by user_id / bet_id in Loki, not by trace_id.
  • The bet-settled consumer is the most-cited orphan case in the perf programme. Documented in docs/audits/perf-trace-coverage-audit.md.

Code-shape constraints

  • New async work goes through apps/api/src/<domain>/queue/ with a *.queue-producer.ts and *.queue-processor.ts pair, registered via BullModule.registerQueueAsync(...). The bet-queue (apps/api/src/bet/queue/bet.queue-producer.ts + bet.queue-processor.ts) is the canonical example to copy. See add-bullmq-queue.md for the recipe.
  • Do not add to apps/api/src/fast-track/rabbitmq/. That module is frozen pending product decision.

Future Fast Track decision

If product decides to ship Fast Track:

  1. Coordinate Fast Track sandbox creds; populate FASTTRACK_JWT_* in Doppler.
  2. Set disabled = false at fast-track.rmq.module.ts:8.
  3. Verify the 11 call sites against the real producer in a sandbox round-trip.
  4. Update this ADR's status to Superseded by ADR-NNNN (the new ADR codifying the live integration).

If product decides Fast Track is permanently dead:

  1. Remove the 11 call sites.
  2. Remove the FastTrackRabbitMQModule import from app.module.ts.
  3. Remove the ebit-rabbitmq service from docker-compose.yml.
  4. Remove BROKER_URI from .example.env and Doppler.
  5. Update this ADR's status to Deprecated.

Until one of those happens, the stub is correct.

References

  • apps/api/src/fast-track/rabbitmq/fast-track.rmq.module.ts:8 — disabled = true.
  • apps/api/src/fast-track/rabbitmq/fast-track.rmq.producer.ts — interface.
  • apps/api/src/bet/bet.service.ts — × 4 stub call sites.
  • apps/api/src/promo/promo-effect.service.ts — × 7 stub call sites.
  • apps/api/src/bet/queue/bet.queue-producer.ts + bet.queue-processor.ts — canonical BullMQ producer/consumer pair.
  • docker-compose.yml — ebit-rabbitmq: service, comment block.
  • CLAUDE.md — "BullMQ, not RabbitMQ" workspace summary.
  • docs/architecture.md — AF-6 weakness entry.
  • docs/audits/perf-trace-coverage-audit.md — orphan trace cases.
  • docs/engineering/observability-runbook.md §5 — cross-service trace gotchas.
  • docs/recipes/add-bullmq-queue.md — internal recipe for new queues.
  • Memory: project_otel_microservice_transport_gap.md — trace propagation gap.
  • Sibling ADRs: 0005 (Redis pub/sub), 0011 (cross-app calls), 0012 (sampling interaction).