AI Learn

Problem

A basic LLM request/response flow does not handle reconnects, long-running execution, cancellation, replay or multi-node delivery well.

Constraints

Conversation state and execution state must survive process boundaries, events need bounded replay, and delivery must work across more than one application node.

Challenge

The system needs to combine persistent application state with live streaming without making the transport layer the source of truth.

Approach

Persist conversations and runs in MySQL, use a transactional outbox, Redis Streams for bounded replay, Redis Pub/Sub for live multi-node fanout, and expose WebSocket/SSE delivery paths.

Architecture

MySQL is authoritative for conversations, messages and runs. Outbox events bridge committed state to Redis. Streams provide replay and Pub/Sub handles low-latency live delivery.

Key decisions

Keep the database authoritative, give runs and events stable identifiers, and design reconnect/cancellation semantics explicitly rather than assuming a permanent client connection.

Trade-offs

The additional persistence and event plumbing is heavier than a simple streaming endpoint, but it makes execution behavior explicit and recoverable.

Implementation

Implemented with Java 25, Spring Boot 4.1, JPA/Hibernate, LangChain4j, MySQL, Redis, WebSockets, SSE, Flyway and automated tests.

Reliability & security

Idempotency keys, reply/run identifiers, replay cursors and explicit cancellation make duplicate and disconnected operations easier to reason about.

Impact

Provides public evidence of Java backend architecture around durable, streaming AI workflows.

Lessons learned

Durability and live streaming are separate concerns; keeping authoritative state independent from delivery makes the system easier to recover.