Reinforcement Learning for Autonomous Vehicles with CARLA Simulator

This project is trying to reproduce the concepts of a research paper "A Novel Approach to Autonomous Driving Using Double Deep Q-Network-Bsed Deep Reinforcement Learning". For the original publication, please refer to: https://doi.org/10.3390/wevj16030138.

This repository implements a reinforcement learning (RL) framework for autonomous driving within the CARLA simulator. It utilizes a Double Deep Q-Network (DDQN) agent, integrating a Convolutional Neural Network (CNN) for spatial feature extraction and a Long Short-Term Memory (LSTM) network for temporal understanding, to enable an autonomous vehicle to navigate safely and efficiently in simulated urban environments.

For a more detailed information, please visit: https://htliang517.github.io/projects/CARLA_RL

Project Overview

The core objective of this project is to develop a robust decision-making system for autonomous vehicles using reinforcement learning. [cite_start]The entire system is built and evaluated within the CARLA simulator, a realistic open-source platform for autonomous driving research. [cite_start]The trained agent learns to select from six discrete driving actions: hard left-turn, soft left-turn, straight, soft right-turn, hard right-turn, and emergency stop.

System Architecture

The overall project workflow consists of three main components: the simulator, the environment state, and the agent.

• Simulator: The CARLA simulator generates environmental datasets and provides sensor observations, specifically camera frames, for the agent to learn from. It offers a powerful Python API to control various aspects like traffic, pedestrian behaviors, weather, and sensors.
• Environment State: This represents the input data for the model, including observations ($S_t$) of the surroundings (e.g., camera frames) and rewards ($R_t$).
• Agent: This is the reinforcement learning model that is trained to generate actions ($a_t$) (control commands for the vehicle) based on observations.

The driving policy within the agent is implemented using a CNN+LSTM-based DDQN architecture:

1. Input Camera Frame: Image frames from a front-facing RGB camera are captured and preprocessed as the state input.
2. CNN (Convolutional Neural Network): Extracts spatial features like road structure, lane markings, and obstacles from each frame.
3. LSTM (Long Short-Term Memory): Captures temporal dependencies across sequential frames, allowing the agent to perceive motion and context over time.
4. Fully Connected Layer (FC Layer): Processes the LSTM's output to generate Q-values for each potential discrete driving action.
5. DDQN Logic (Only in TRAINING): Uses Q-values to compute loss and update model weights. [cite_start]It does not directly generate actions.
6. Action Selection: During the final action selection stage, the model chooses the action with the highest Q-value, producing real-time control decisions.

Reward Function

A custom reward function was adopted, encouraging both safety and driving speed. The reward function is defined as:

$$ \text{Reward (State, Action)} = \begin{cases} -20 & \text{if a collision occurs} \ +250 & \text{if episode time limit is reached} \ +5 + \lambda v_t & \text{otherwise} \end{cases} $$

where $v_t$ is the vehicle's speed at time step t and $\lambda$ is a positive weight, set to $0.2$ in this project.

Codebase

This repository contains the following Python scripts:

• CARLA_env.py: Defines the CARLA simulation environment, including sensor setup (RGB camera, collision sensor), spawning of NPCs, and action definitions.
• DriverModel.py: Implements the DriverModel class, which represents the neural network architecture (CNN, LSTM, and Fully Connected Layer) responsible for taking visual input and predicting Q-values for possible actions.
• CARLA.ipynb: A Jupyter Notebook that orchestrates the training and testing of the DDQN agent within the CARLA simulator. It utilizes CARLA_env.py for environment interaction and DriverModel.py for the agent's neural network.

CARLA_env.py

This script handles the interaction with the CARLA simulator. Key functionalities include:

• Environment Initialization: Sets up the CARLA client, world, and vehicle.
• Sensor Configuration: Defines and attaches an RGB camera (640x480 resolution, FOV 110) and a collision sensor to the vehicle.
• Traffic Generation: Spawns non-player characters (NPCs) to simulate traffic, with 15 NPCs used in this project to balance computation and traffic load.
• Environment Reset: Resets the environment for each training epoch, randomly spawning the car in different positions to enhance learning in diverse situations.
• Action Space Definition: Defines the six discrete driving actions the agent can take.

DriverModel.py

This file contains the DriverModel class, which is a nn.Module (from PyTorch) that defines the neural network architecture of the DDQN agent.

• CNN Feature Extractor:
  • Three Conv2d layers with ReLU activation.
  • Output size of the convolutional part is $64 \times 7 \times 7$.
• LSTM Network:
  • Takes the flattened CNN features as input.
  • hidden_size: 256.
  • num_layers: 1.
  • batch_first=True.
• Fully Connected Layer:
  • Receives input from the LSTM output (256 features).
  • Outputs 6 features, corresponding to the Q-values for each of the 6 possible actions.

CARLA.ipynb

This Jupyter Notebook is the main execution script for training and evaluating the reinforcement learning agent. It integrates the CARLA environment and the DriverModel.

• DDQN Agent Implementation: Implements the Double Deep Q-Network (DDQN) algorithm.
  • n_actions: 6.
  • gamma: 0.99 (discount factor).
  • lr: 2e-4 (Adam optimizer learning rate).
  • batch_size: 64.
  • online_net and target_net: Instances of DriverModel for stable training.
  • epsilon_start: 1.0, epsilon_final: 0.05, epsilon_decay: 50,000 (for $\epsilon$-greedy exploration).
  • loss_fn: nn.SmoothL1Loss().
  • tau_soft: 5e-3 (Polyak averaging factor).
• Training Loop: Manages the training process, including environment interaction, action selection, reward calculation, and model updates.
• Evaluation: Contains logic to evaluate the agent's driving performance.

Setup and Running

Prerequisites

• CARLA Simulator (version used in project not specified, but typically a recent stable release is recommended)
• Python 3.x
• PyTorch
• NumPy
• Pillow
• OpenCV (if additional image processing is needed beyond basic preprocessing)

Installation

1. Install CARLA Simulator: Follow the official CARLA documentation for installation.

2. Install Python Dependencies:

   pip install torch numpy Pillow opencv-python carla

   Note: The carla package can usually be installed via pip, but for specific CARLA versions, you might need to manually copy the PythonAPI/carla/ egg file from your CARLA installation to your Python's site-packages directory.

Running the Project

1. Start CARLA Simulator: Launch the CARLA simulator (e.g., ./CarlaUE4.sh on Linux or CarlaUE4.exe on Windows).

2. Run the Jupyter Notebook: Open CARLA.ipynb in a Jupyter environment and execute the cells. This notebook contains the training and evaluation logic.

   jupyter notebook CARLA.ipynb

Results

The project successfully demonstrated that the CNN+LSTM based DDQN agent significantly improved driving quality compared to the original CARLA autopilot. The trained agent was able to drive smoothly, avoid collisions, and adhere to lane discipline in both traffic-free and populated environments.

Future Improvements

• Continuous Control: Currently, the agent selects from six discrete actions. Future work could involve modifying the Q-actions to support continuous control of the car (e.g., steering angle and acceleration) for smoother driving.
• Enhanced Reward Function: The current reward function does not penalize stopping, which can lead to the car stalling behind other vehicles. Future improvements could include modifications to encourage lane switching and passing stalled vehicles.
• Advanced Architectures: Exploring more complex or state-of-the-art reinforcement learning algorithms and neural network architectures.

References

• Činčurak, D., Grbić, R., Vranješ, M., & Vranješ, D. (2024). Autonomous Vehicle Control in CARLA Simulator Using Reinforcement Learning.
• Khlifi, A., Othmani, M., & Kherallah, M. (2025). A Novel Approach to Autonomous Driving Using Double Deep Q-Network-Bsed Deep Reinforcement Learning.
• Pérez-Gil, Ó., Barea, R., López-Guillén, E., Bergasa, L. M., Gómez-Huélamo, C., Gutiérrez, R., & Díaz-Díaz, A. (2022). Deep reinforcement learning based control for Autonomous Vehicles in CARLA.
• Wu, J., Huang, C., Huang, H., Lv, C., Wang, Y., & Wang, F. Y. (2024). Recent advances in reinforcement learning-based autonomous driving behavior planning: A survey.