Training a coding model to paint watercolours with TRL and OpenEnv

2026-09-03 · Hugging Face

Background

On 23 August 2026, Surya Narreddi posted a video of watercolours painted by a language model. The model writes JavaScript through p5.brush, a library that adds natural drawing tools to p5.js. The video went viral fast, with over 1.5M views at the time of writing.

The author reproduces this idea from the engineering side, using TRL and OpenEnv. The reference pool dataset, the RL environment, the training scripts, and the trained models are all open.

Core idea

The project belongs to the early-exploration era of generative AI art, akin to DeepDream and *Edmond de Belamy*. Its distinctive aspects:

  • The model outputs about 150 lines of JavaScript code, not pixels. The code is readable, editable, and rerunnable.
  • The decision behind each brushstroke is visible.
  • The style comes from a restriction: the model is allowed only ten of the library's methods.

The author also mentions Anna Ridler, who photographed thousands of tulips, hand-labelled every one, exhibited the dataset as artwork, and later trained a model on it. This project does something similar by curating a set of images by hand and training against them.

Reward design

The reward is aesthetic preference; there is no correct answer. The reward function from the original blog is implemented as:

| Term | Weight | What it measures |

|------|--------|------------------|

| gate | 0.05 | The sketch compiles, paints something, does not cheat |

| length | 0.05 | A soft push towards longer code snippets |

| pairwise judge | 0.60 | Style compared against references drawn from a pool |

| HPSv3 | 0.30 | Aesthetic preference on the render |

HPSv3 is an open 7B preference model. Given an image and a text description, it returns a score for how much a person would prefer that image. The pairwise judge is Qwen3-VL-30B-A3B-Instruct, a general vision model called through HF Inference Providers. It sees the candidate painting next to four references randomly selected from the pool, guided by a written description of what to weigh (bleeds, translucent washes, soft edges). Each comparison is shown in both presentation orders.

Training pipeline

The whole pipeline runs on Hugging Face, end to end:

  • Training on Jobs
  • The RL environment and the scorer model as Spaces
  • The pairwise judge through Inference Providers
  • All artifacts on the Hub, gathered in one collection

Once the two Spaces are up, the recipe is one command. Duplicate the environment and the scorer model, set two environment variables for the reward mix, and launch. An example command is:


hf jobs uv run train/watercolour_grpo.py --flavor h200 --timeout 48h --secrets HF_TOKEN -- \
  --env-url https://<you>-watercolour-env.hf.space \
  --model Qwen/Qwen3.5-35B-A3B --lora --all-linear --bf16 --gradient-checkpointing \
  --subject 'a peach hibiscus' --references 4 \
  --top-p 0.95 --top-k 20 \
  --lr 5e-5 --lr-scheduler constant_with_warmup --warmup-steps 5 \
  --scale-rewards none \
  --steps 110 --n-episodes 240 --num-generations 8 \
  --per-device-batch-size 1 --gradient-accumulation-steps 8 \
  --max-completion-length 8192 \
  --run-tag my-run --out <you>/watercolour-grpo --push-to-hub

Experiments and results

The author followed the original blog step by step, changing only what was strictly needed. Three runs were performed, one per reward mix, evolving in parallel. The article includes a list of planned future experiments and the full list of published artifacts.

Significance

This project demonstrates that RL over taste is feasible. The model's code output is readable, editable, and reusable, making each decision inspectable. All code, data, and models are open for the community to build upon.

Source