# ThreeJs and Reinforcement Learning in 3D

- URL: https://rdjarbeng.com/three.js-and-reinforcement-learning-in-3-dimensions/
- Published: 2025-02-02
- Author: Richard Djarbeng
- Tags: Machine Learning, 3D, Reinforcement Learning, threejs, JavaScript, TypeScript

I built a simple 3D game in November 2024 mostly for fun.  The objective was to minimize the time taken for a player to destroy all the pillars. Then I got a brilliant _(in hindsight not so brilliant)_ idea 💡. Why not add a Reinforcement Learning (RL) agent to play the game I made? I'd always wanted to do some hands-on RL. Sounded like a great idea with the best of intentions, what could possibly have gone wrong.

![3D reinforcement learning game screenshot showing agent with pillars on a blue-gray environment](/assets/images/3d_rl_game.png "3D reinforcement learning game screenshot showing agent with pillars")

If you wanted to view the current version of the game [find it here](https://3dcharacter.netlify.app/)

## Project Overview

This project featured a robot and an 'alien' character navigating through a custom-designed 3D environment.

- **Gameplay Objective**: The primary goal was to minimize the time it took for the RL agent to destroy all the pillars within the environment. Each pillar, when destroyed, scored 100 points, adding a competitive edge to the gameplay.
- **Environment**: A 3D space where the robot and 'alien' character interacted with pillars. The agent has 4 actions to choose from (up, down, left and right) just like a user playing the game. The agent receives information about the coordinates of the closest pillar from it's current coordinates. Rewards are given for breaking a pillar, for taking actions that encourage exploration; move it outside of a local area.
  

## Reinforcement Learning Agent

- **Initial Setup**: The RL agent started with an epsilon value of around 0.4. This meant:
- 40% of the actions taken by the agent were random (exploratory), allowing it to discover new strategies.
- 60% of the actions were based on exploitation, maximizing reward based on previously learned information.
- **Learning Process**: As the game progressed, the epsilon value decreased, transitioning the agent from exploration to exploitation, refining its strategy over time. To decrease epsilon I divided it by a constant factor I had hardcoded.
- **Current Status**: The agent encountered an issue where it got stuck after destroying approximately 4 pillars. The robot character also had jerky motion that just didn't look good especially at the start. This was where development paused, but _hopefully_ there was potential for further refinement. 

Here's an earlier screen recording I made below

<video width="100%" preload="auto" autoplay controls>
  <source src="{{ '/assets/videos/rl_robot_janky_motion.mp4' | relative_url }}" type="video/mp4">
  Your browser does not support the video tag.
</video>

## Features

- **Destruction Mechanics**: The game featured dynamic explosions when the character impacted a pillar, enhancing the visual and interactive experience.
- **Scoring System**: Each pillar destroyed awarded the player 100 points, encouraging efficiency and strategy in gameplay.

## Technologies Used

- **Three.js**: For rendering the 3D environment and characters.
- **ReactJS**: For managing the game's UI and state, if you were interested in contributing or experimenting with the RL agent.

## Getting Started

If you were interested in continuing the development or experimenting with the RL agent, here's what you needed:

1. **Basic Knowledge**: Familiarity with ReactJS and 3D modeling would have been beneficial.
2. **Environment Setup**: Ensure you had Node.js installed for ReactJS development. Or you could have used an online IDE like stackblitz (honestly that was the same environment I used to code this)
3. The link to the [repository was here](https://github.com/RDjarbeng/3d-character-game)

## Follow the Development

For updates and progress on this project, follow me on [X (formerly Twitter)](https://x.com/DjarbengRichard/status/1883970511312675302).

Potential future ideas:

* Should pass in the distance to a pillar as a variable for the reward function, possibly increase the reward
* Could add a penalty for moving away from a pillar
* Set a time for the agent to complete the tasks or else get stopped out

## Acknowledgements

The initial template for the project was made with [bolt.new](https://bolt.new).

---

Feel free to fork the repository, contribute, or reach out with suggestions or questions. Let's bring this game to its full potential!