Samuel Liedtke

Automatic Git commit message with llm, chain-of-thought and structured output

Nov. 10, 2024 • git, automation, llm, generative AI

Writing Git commit messages is a frequent task for developers, so would it not be nice if it could be automated? I have touched on the topic in my June blog post Using LLMs on the Command Line with llm: Practical Examples for Developers . Back then, I wasn't very happy with the results.

Recently, I got inspired by some ideas from the Braintrust episode of the Latent Space podcast, and I decided to give this another go.

Use case and techniques

My use case is this: Generate automatic commit messages for my repository of personal notes (Markdown files, organized with the help of zk, and synced to Github).

My first idea was to let the model "think" for longer before giving an answer, instead of forcing it to provide the final answer right away. This is also referred to as step-by-step or 'chain-of-thought' prompting.

However, this creates another problem. In order to automate creating the commit, I needed to somehow extract the final answer from a longer model response. This is where structured output comes in: Asking the model to respond in a particular JSON format.

Solution

After a couple of iterations I ended up with the following command. It uses llm, a CLI tool for interacting with Large Language Models.

git diff --cached | \
  llm -m 4o-mini -s "summarize the following git diff. based on the \
  summary, suggest a short title for the git commit message. answer \
  in valid JSON (no backticks), using the following schema: \
  {\"summary\": string, \"title\": string}"  | \
  jq -r '.title'

The command takes the diff of the staged changes, and pipes it to llm along with a system prompt. The last part | jq -r '.title' is to extract just the title, which can then be used as commit message, i.e. git commit -m "$commit_message".

I've been really happy with the results from 4o-mini.

Example output:

{
  "summary": "Updated the 'grhm' function to include error handling and fetch information about the latest passing master commit more robustly. Added new SQL notes related to periods not fully covered and resolved issues with stale postmaster.pid files in PostgreSQL. Additionally, included observations about octopuses hunting behavior with additional resources.",
  "title": "Enhance grhm function, add SQL notes, fix Postgres PID issue"
}

Note on local models

I've also tried using the local model llama3.2 3B via llm-ollama. I found it to work quite well, and I'm very exited about small local models, because it means I don't have to send any diffs over the internet. However, I found the quality of the commit messages were not quite as good as those from 4o-mini.

Important caveats

I should note that, while this approach seems to work well for paraphrasing notes, should be used with caution for code changes. Commit messages should provide enough context, either for the reviewer, or your future self! 😉 If this context is not contained in the diff in some form (comments, docstrings etc.), it will be impossible for the LLM to guess.

Recent posts