Kafka MicroServices with Spring Boot, Docker, Kubernetes, AI

Build real-world, event-driven microservices using Apache Kafka, Spring Boot, Docker, Kubernetes, PostgreSQL, OpenAPI, Spring Boot Actuator, and AI-assisted development.

This course is designed to take you from the fundamentals of Kafka all the way to building, testing, securing, monitoring, containerizing, and deploying production-style Kafka microservices.

You will start by learning the core concepts of Apache Kafka, including topics, partitions, producers, consumers, offsets, consumer groups, brokers, replication, leader election, ISR, and retention. Then, you will move into hands-on development by building a complete Library Events Producer and Library Events Consumer microservice using Spring Boot and Spring Kafka.

Along the way, you will use modern AI-assisted coding workflows such as GitHub Copilot, spec-driven development, PRD documents, implementation plans, AGENTS . md, and reusable coding skills to build applications more efficiently and consistently.

By the end of this course, you will have built a complete Kafka-based microservices system that includes REST APIs, Kafka producer and consumer flows, PostgreSQL persistence, error handling, retries, recovery strategies, transactions, health checks, Docker images, and Kubernetes deployment.

Course Introduction

Welcome to this hands-on course on building event-driven Kafka microservices using Spring Boot, Docker, Kubernetes, and AI-assisted development.

In this course, we are going to start from the fundamentals of Apache Kafka and gradually move toward building a complete real-world microservices application. You will first learn the core Kafka concepts such as topics, partitions, producers, consumers, offsets, consumer groups, brokers, replication, leader election, and retention policies.

Once the Kafka foundation is clear, we will move into hands-on development by building a complete Library Events Producer and Library Events Consumer application using Spring Boot and Spring Kafka. The producer service will expose REST APIs and publish events to Kafka, while the consumer service will consume those events, process them, and persist the data into a PostgreSQL database.

This course also introduces a modern way of building software using AI-assisted development. We will use ideas like PRD documents, implementation plans, GitHub Copilot, AGENTS . md, and reusable AI skills to guide development in a more structured and productive way.

As we progress, we will cover important real-world engineering topics such as unit testing, integration testing, Embedded Kafka testing, Swagger/OpenAPI documentation, producer reliability, consumer retry and recovery, Dead Letter Topics, Kafka transactions, exactly-once semantics, health checks, monitoring, Docker containerization, and Kubernetes deployment.

By the end of this course, you will have built a complete production-style Kafka microservices system from scratch and gained the confidence to design, develop, test, monitor, package, containerize, and deploy Kafka-based applications in the real world.

Quick Breakdown by Section

Getting Started with the Course

• Get a complete overview of what this course is about.

• Understand the main goals and learning outcomes of the course.

• Review the prerequisites needed before starting the course.

• Set the right expectations for the tools, technologies, and hands-on projects covered.

Getting Started with Kafka

• Understand what Apache Kafka is and why it is widely used in modern applications.

• Learn how Kafka fits into event-driven architecture and microservices.

• Explore core Kafka terminologies such as topics, partitions, brokers, producers, consumers, and client APIs.

• Build a strong conceptual foundation before moving into hands-on Kafka development.

Docker Installation

• Install Docker on your local machine.

• Understand why Docker is useful for running Kafka and supporting services locally.

• Prepare your development environment for the hands-on sections in the course.

• Learners who already have Docker installed can quickly verify their setup and move forward.

Kafka Components and Internals

• Learn how Kafka topics and partitions organize data for scalability.

• Set up Kafka locally using KRaft and understand how Kafka works without ZooKeeper.

• Create topics and produce/consume messages using Kafka CLI commands.

• Understand how message keys affect partition selection and ordering.

• Explore how Kafka producers and consumers work under the hood.

• Learn about consumer offsets, consumer groups, rebalancing, commit logs, and retention policies.

• Set up a local multi-broker Kafka cluster and understand replication, leader election, ISR, and fault tolerance.

Application Overview

• Get introduced to the Library Inventory Service application.

• Understand the producer and consumer microservices that will be built in the course.

• See how REST APIs, Kafka, PostgreSQL, and Spring Boot fit together.

• Establish the end-to-end architecture before starting the implementation.

AI-Driven Development and Agentic Engineering

• Understand how software development is changing with AI coding assistants.

• Learn the difference between vibe coding and agentic engineering.

• Explore how AI can help with planning, coding, testing, and documentation.

• Use AI-assisted workflows to build applications faster while still understanding the underlying concepts.

Local Setup: IDE

• Install and configure IntelliJ IDEA.

• Prepare the IDE for Spring Boot and Kafka development.

• Set up the development environment needed for building both producer and consumer services.

• Make sure the local machine is ready for hands-on coding.

Build Library Events Kafka Producer Microservice

• Create the base Spring Boot project for the producer service.

• Set up GitHub Copilot and use AI-assisted coding during development.

• Create PRD and implementation plan documents before writing code.

• Build the Library Event domain model.

• Create REST endpoints to publish events to Kafka.

• Use KafkaTemplate to send messages to Kafka.

• Learn KafkaTemplate internals, batching, buffering, threading model, and auto-configuration.

• Configure serializers, Spring Profiles, custom error handling, and synchronous message publishing.

Unit and Integration Testing — Library Events Producer API

• Understand the difference between unit testing and integration testing.

• Write unit tests for POST and PUT endpoints using MockMvc.

• Build integration tests to validate the full producer API flow.

• Test the LibraryEventProducer component by mocking KafkaTemplate.

• Improve confidence in the producer service by validating controller, service, and Kafka publishing behavior.

Documenting REST APIs with Swagger / OpenAPI 3

• Understand why OpenAPI matters for modern REST APIs.

• Add Swagger/OpenAPI support to the Library Events Producer.

• Generate interactive API documentation automatically.

• Make the API easier to test, understand, and share.

AGENTS . md and Skills for AI Coding Assistants

• Learn what AGENTS . md is and how it guides AI coding agents.

• Understand how skills help AI agents perform focused tasks consistently.

• Create reusable testing skills for unit and integration testing.

• Create controller skills to guide consistent REST controller generation.

• Use AI instructions to improve code quality and reduce repeated prompting.

Build Library Events Kafka Consumer Microservice

• Revisit the consumer architecture and PRD document.

• Create the base Spring Boot project for the consumer service.

• Build a structured implementation plan.

• Implement Kafka consumption using @KafkaListener.

• Poll, read, and log incoming Library Event messages.

• Deserialize raw JSON into structured DTOs.

• Compare StringDeserializer and JsonDeserializer.

• Understand Spring Kafka consumer internals and auto-configuration.

Consumer Groups and Consumer Offset Management

• Learn how Kafka consumer groups enable parallel processing.

• Understand consumer group rebalancing through hands-on examples.

• Explore default offset management in Spring Kafka.

• Learn how manual offset management gives more control over message processing.

• Configure concurrent consumers to improve throughput and scalability.

Saving Library Events in Database

• Set up PostgreSQL in the Library Events Consumer application.

• Integrate Flyway for database schema migration.

• Create entity classes for Library Event and Book data.

• Build Spring Data JPA repositories.

• Implement business logic to persist consumed Kafka messages.

• Test the complete flow from producer to Kafka to consumer to database.

Integration Testing Using Embedded Kafka

• Learn how Embedded Kafka helps test Kafka consumers.

• Write integration tests for the Library Event Consumer.

• Validate that messages are consumed and processed correctly.

• Test Kafka behavior without depending on an external Kafka cluster.

Build CRUD Endpoints to Expose Library Event and Book Data

• Build REST APIs to manage Book data.

• Enable Swagger documentation for the consumer service.

• Create GET endpoints to retrieve Library Event data.

• Expose persisted Kafka data through RESTful APIs.

• Complete the consumer service with both Kafka processing and API access.

Kafka Producer — Exception and Error Handling

• Understand how producer acknowledgments impact message durability.

• Learn how acks and min.insync.replicas work together.

• Configure retries and retry backoff for transient failures.

• Learn how idempotent producers prevent duplicate message production.

• Understand max.in.flight.requests.per.connection and its impact on ordering.

• Explore producer timeout configurations.

• Apply recommended producer settings for reliable production-grade Kafka applications.

• Handle errors in both async and sync producer flows.

Kafka Consumer — Exception and Error Handling

• Create a complete error-handling plan for Kafka consumers.

• Explore current consumer behavior when exceptions occur.

• Understand Spring Kafka error handling under the hood.

• Configure DefaultErrorHandler with exponential backoff.

• Use RetryListener to observe retry attempts.

• Handle retryable and non-retryable exceptions.

• Publish failed records to Dead Letter Topics.

• Persist failure records into the database.

• Implement hybrid recovery by publishing to DLT and saving failure details.

• Update integration tests to validate retry and recovery behavior.

Kafka Consumer Timing Configurations

• Understand broker-side timing configurations.

• Learn fetch-related configurations that affect throughput and latency.

• Explore connection-related configurations for stable consumer-broker communication.

• Understand how heartbeat, session timeout, poll interval, and fetch settings impact consumer behavior.

• Learn how to tune consumers for real-world Kafka applications.

Kafka Transactions — Exactly Once Semantics

• Understand Kafka transactions and exactly-once semantics.

• Learn how transactions help avoid duplicates and partial message publishing.

• Implement transactional behavior using @Transactional.

• Publish multiple messages as one atomic transaction.

• Use KafkaTemplate.executeInTransaction() for programmatic transaction handling.

• Update integration tests to validate transactional behavior.

• Understand how transactions apply to Kafka consumers.

Kafka Health Checks and Monitoring Using Spring Boot Actuator

• Learn the fundamentals of Spring Boot Actuator.

• Enable health checks in the Library Events Producer.

• Configure Kafka readiness health indicators.

• Understand how readiness checks help detect Kafka connectivity issues.

• Enable health checks and Kafka readiness for the consumer service.

• Prepare applications for production-style monitoring.

Package the Apps as Executable JARs

• Build the producer service as an executable JAR.

• Run the producer outside the IDE.

• Pass environment variables to configure applications.

• Build and run the consumer service as an executable JAR.

• Learn how to run Spring Boot Kafka applications independently.

Introduction to Containers — Build and Run Containers Using Docker

• Understand what containers are and why they matter.

• Learn Docker fundamentals such as images, containers, Dockerfiles, and runtime.

• Create a Docker image for the Library Events Producer.

• Run the producer as a Docker container.

• Run the same Docker image with different environment configurations.

• Publish Docker images to Docker Hub.

• Build and run the Library Events Consumer as a Docker container.

Running Spring Boot Apps on Kubernetes

• Understand why Kubernetes is needed beyond Docker.

• Learn Kubernetes core concepts and architecture.

• Set up Kubernetes locally using Minikube.

• Install and configure kubectl.

• Deploy applications using Deployments and Pods.

• Expose application endpoints using Services, Port Forwarding, and Ingress.

• Scale applications to multiple replicas.

• Externalize configuration using ConfigMaps.

• Configure liveness and readiness checks.

• Deploy the Library Events Consumer to Kubernetes.

• Use Kubernetes Secrets to manage sensitive configuration securely.

Final Takeaway

By the end of this course, you will have built a complete, production-style Kafka microservices system using Spring Boot, Apache Kafka, PostgreSQL, Docker, Kubernetes, OpenAPI, Actuator, and AI-assisted development.

You will not just learn Kafka concepts in theory — you will apply them by building real producer and consumer microservices, testing them, handling failures, monitoring health, packaging them, containerizing them, and deploying them to Kubernetes.

This course gives you a complete hands-on path to go from Kafka fundamentals to building real-world, AI-assisted Kafka microservices that are reliable, testable, observable, and deployment-ready.

Objectives Achieved in This Course

By completing this course, you will be able to:

• Understand Apache Kafka architecture and core internals.

• Explain topics, partitions, brokers, producers, consumers, offsets, consumer groups, replication, and leader election.

• Build Kafka producer and consumer microservices using Spring Boot.

• Use KafkaTemplate to publish events to Kafka.

• Use @KafkaListener to consume and process Kafka messages.

• Work with message keys, partitioning, serializers, and deserializers.

• Manage consumer groups, offset commits, rebalancing, and concurrent consumers.

• Persist consumed Kafka messages into PostgreSQL using Spring Data JPA.

• Manage database schema changes using Flyway.

• Write unit tests, integration tests, Embedded Kafka tests, and end-to-end tests.

• Document REST APIs using Swagger and OpenAPI.

• Apply AI-assisted development using GitHub Copilot, PRDs, implementation plans, AGENTS . md, and reusable skills.

• Configure reliable Kafka producers using retries, acknowledgments, idempotence, timeouts, and recommended producer settings.

• Build resilient Kafka consumers using retries, exponential backoff, Dead Letter Topics, and recovery strategies.

• Store failed records in the database and implement hybrid recovery patterns.

• Understand Kafka consumer timing configurations and how they affect real-world applications.

• Implement Kafka transactions and exactly-once semantics.

• Monitor producer and consumer services using Spring Boot Actuator and Kafka readiness checks.

• Package Spring Boot applications as executable JAR files.

• Build Docker images for Spring Boot Kafka applications.

• Run producer and consumer services as containers.

• Publish Docker images to Docker Hub.

• Deploy Spring Boot Kafka applications to Kubernetes.

• Use Kubernetes Deployments, Pods, Services, Ingress, ConfigMaps, readiness/liveness probes, and Secrets.

Final objective:
After completing this course, you will have the skills to design, build, test, monitor, containerize, and deploy production-style Kafka microservices using Spring Boot, Docker, Kubernetes, and AI-assisted development.