Research project / reinforcement learning
Autonomous navigation
with deep Q-learning.
A ROS2 and Gazebo training system where a simulated robot learns to reach a goal, avoid obstacles, and later communicate with an ESP32 through micro-ROS.
01
14D state
goal-aware input
02
7 actions
recovery included
03
DQN
experience replay
04
Gazebo
3D training world
01 / learning + algorithm
Learn a policy from motion, not rules.
Deep Q-learning maps a continuous observation vector to action values. The policy network improves from replayed transitions while a slower target network keeps the learning target stable.
Network
256 → 128 → 7
Replay
100,000 transitions
Batch
128 samples
Target
every 10 episodes
14-dimensional state
Front/rear sensors, position, circular goal angle, normalized goal vector, distance, linear and angular velocity, minimum clearance, and navigation mode.
reward shaping
Progress toward the goal is rewarded strongly; alignment gets a larger bonus, collisions are penalized by proximity, and reaching the goal returns +25 as a terminal reward.
0
Forward
(0.25, 0.00)
preferred
1–2
Forward + turn
(0.18, ±0.60)
fine correction
3–4
Large turn
(0.10, ±1.50)
obstacle escape
5
Reverse
(-0.12, 0.00)
recovery
6
Emergency spin
(0.05, 2.50)
new recovery action
02 / simulation methods
Train safely in a digital arena.
Because DQN training needs thousands of episodes and high compute, the main learning loop runs in Gazebo rather than on the physical robot. The simulator provides repeatable physics, sensor topics, obstacles, and a visible goal.
01
Explore
ε starts at 1.0 and decays to 0.05.
02
Learn
Replay batches update the policy network.
03
Evaluate
Success rate, reward, steps, and collisions are tracked.
03 / software integration
Gazebo teaches. ROS2 connects. micro-ROS carries it outward.
Gazebo
3D world, physics, obstacles, and simulated sensors.
ROS2
Control nodes, launch files, topics, and messages.
PyTorch
DQN policy, replay memory, target network, checkpoints.
micro-ROS
ESP32 bridge for real-time embedded communication.
04 / hardware boundary
Hardware is the endpoint, not the training ground.
The planned platform uses an ESP32, L298N motor driver, two HC-SR04 ultrasonic sensors, encoder motors, and a 7.4–11.1V battery. We did not perform complete high-end onboard DQN training: the compute-heavy learning phase remained in simulation.
important constraint
ESP32 is used for embedded communication and control testing, while model training is performed on a more capable computer.
controller
ESP32
WiFi + micro-ROS client
sensing
HC-SR04 × 2
front + rear distances
actuation
L298N + DC motors
differential drive
power
2S–3S LiPo
7.4–11.1V supply
05 / testing
Test the learned behavior at the edge.
The trained model is evaluated in Gazebo first, then the communication and control path is tested on ESP32 with micro-ROS. This separates learning validation from embedded deployment risk.
validation sequence