{"id":2910,"date":"2024-11-09T17:50:20","date_gmt":"2024-11-09T09:50:20","guid":{"rendered":"https:\/\/gergely.imreh.net\/blog\/?p=2910"},"modified":"2025-01-30T10:59:00","modified_gmt":"2025-01-30T02:59:00","slug":"adventures-into-code-age-with-an-llm","status":"publish","type":"post","link":"https:\/\/gergely.imreh.net\/blog\/2024\/11\/adventures-into-code-age-with-an-llm\/","title":{"rendered":"Adventures into Code Age with an LLM"},"content":{"rendered":"\n

It’s a relaxed Saturday afternoon, and I just remembered some nerdy plots I’ve seen online for various projects, depicting “code age” over time: how does your repository change over the months and years, how much code still survives from the beginning till now, etc… Something like this made by the author of curl<\/a>:<\/p>\n\n\n\n

\"\"
Curl’s code age distribution<\/figcaption><\/figure>\n\n\n\n

It looks interesting and informative. And even though I don’t have codebases that have been around this long, there are plenty of codebases around me that are fast moving, so something like a month (or in some cases week) level cohorts could be interesting.<\/p>\n\n\n\n

One way to take this challenge on is to actually sit down and write the code. Another is to take a Large Language Model<\/a>, say Claude<\/a> and try to get that to make it. Of course the challenge is different in nature. For this case, let’s put myself in the shoes of someone who says<\/p>\n\n\n\n

\n

I am more interested in the results than the process, and want to get to the results quicker.<\/p>\n<\/blockquote>\n\n\n\n

See how far we can get with this attitude, and where does it break down (probably no spoiler: it breaks down very quickly.).<\/p>\n\n\n\n

Note on the selection of the model: I’ve chosen Claude just because generally I have good experience with it these days, and it can share generated artefacts (like the relevant Python code) which is nice. And it’s a short afternoon. :) Otherwise anything else could work as well, though surely with varying results. <\/p>\n\n\n\n

Version 1<\/h3>\n\n\n\n

Let’s kick it off with a quick prompt.<\/p>\n\n\n\n

\n

Prompt<\/strong>: How would you generate a chart from a git repository to show the age of the code? That is when the code was written and how much of it survives over time?<\/p>\n<\/blockquote>\n\n\n\n

Claude quickly picked it up and made me a Python script, which is nice (that being my day-to-day programming language). I guess that’s generally a good assumption these days if one does data analytics anyways (asking for another language is left for another experiment).<\/p>\n\n\n\n\n\n\n\n

The result is this this code<\/a>. I’ve skimmed it that it doesn’t just delete all my repo or does something completely batshit, but otherwise saved in a repo that I have at hand. To make it easier on myself, added some inline metadata<\/a> with the dependencies:<\/p>\n\n\n\n

# \/\/\/ script\n# dependencies = [\n#   \"pandas\",\n#   \"matplotlib\",\n# ]\n# \/\/\/<\/code><\/pre>\n\n\n\n
and from there I can just run the script with uv<\/a>.<\/p>\n\n\n\n
First it checked too few files (my repository is a mixture of Python and SQL scripts managed by dbt<\/a>), so had to go in and change those filters, expanding them.<\/p>\n\n\n\n
Then the thought struck me to remove the filter altogether (since it already checks only files that are checked in git, so it should be fine – but then it broke on a step where it reads a file as if it was text to find the line counts. I guess there could be a better way of filtering (say “do not read binary files”, if there’s a way to do that), but just went with catching the problems:<\/p>\n\n\n\n
# ....\n    for file_path in tracked_files:\n        try:\n            timestamps = get_file_blame_data(file_path)\n            for timestamp in timestamps:\n                blame_data[timestamp] += 1\n                total_lines += 1\n        except UnicodeDecodeError:\n            print(f\"Error reading file: {file_path}\")\n            continue\n#....<\/code><\/pre>\n\n\n\n
(hance I know that a favicon PNG was causting those UnicodeDecodeError<\/code> hubbub in earlier runs. Now we are getting somewhere, and we have a graph like this:<\/p>\n\n\n\n
\"\"
Version 1<\/figcaption><\/figure>\n\n\n\n
This is already quite fun to see. There are the sudden accelerations of development, there are the plateaus of me working on other projects, and generally feel like “wow, productive!” (with no facts backing that feeling \ud83d\ude02). Also pretty good ROI on maybe 15 mins of effort.<\/p>\n\n\n\n
Having said that, this is still fair from what I wanted.<\/p>\n\n\n\n
Version 2<\/h3>\n\n\n\n
\n
Promt:<\/strong> Could we change the code to have cohorts of time, that is configurable, say monthly, or yearly cohoorts, and colour the chart to see how long each cohort survives?<\/p>\n<\/blockquote>\n\n\n\n
This came back with another set of code<\/a>. Adding the metadata, skimming it (it has the filter on the file extensions again, never mind), and running it once more to see the output, I get this:<\/p>\n\n\n\n
\"\"
Version 2<\/figcaption><\/figure>\n\n\n\n
Because of the file extension filter in place, the numbers are obviously not aligning with the above, but it does something. The something is a bit unclear, bit it feels<\/em> like progress, so let’s give it a benefit of the doubt, and just change once more. <\/p>\n\n\n\n
Version 3<\/h3>\n\n\n\n
\n
Promt:<\/strong> Now change this into a cummulative graph, please.<\/p>\n<\/blockquote>\n\n\n\n
One more time Claude came back with this code<\/a>. Adding the metadata again, same drill. Running this has failed with errors in numpy<\/code>, though:<\/p>\n\n\n\n
TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''<\/code><\/pre>\n\n\n\n
Now this needed some debugging. It turns out a column the code is trying to plot is actually numbers as strings rather than numbers as, you know, say floats…<\/p>\n\n\n\n
# my \"fix\"\n        df['cumulative_percentage'] = df['cumulative_percentage'].astype(float)\n# end\n\n        # Plot cumulative area\n        plt.fill_between(df.index, df['cumulative_percentage'],\n                        alpha=0.6, color='royalblue',\n                        label='Cumulative Code')<\/code><\/pre>\n\n\n\n
It didn’t take too many tries, but it was confusing at first – why shouldn’t be, if I didn’t actually read<\/em> just skim the code…<\/p>\n\n\n\n
The result is then like this:<\/p>\n\n\n\n
\"\"
Version 3<\/figcaption><\/figure>\n\n\n\n
Sort of meh<\/em>, it feels like it’s not going to the right direction overall.<\/p>\n\n\n\n
But while debugging the above issues, I first tried tried to ask Claude about the error (maybe it can fix it itself), but came back with “Your message exceeds the length limit<\/a>. …” (for free users, that is). So I kinda stopped here for the time being.<\/p>\n\n\n\n
Lessons learned<\/h2>\n\n\n\n
The first lesson is very much re-learned:<\/p>\n\n\n\n
\n
Garbage in, garbage out.<\/p>\n<\/blockquote>\n\n\n\n
If I cannot express what I really want, it’s very difficult to make it happen. And my prompts were by no means expressing my wishes correctly, no wonder Claude wasn’t really hitting the mark. Whether or not a human engineer would have faired better, I don’t know. I know however, that this kind of “tell me exceedingly clearly what’s your idea” is an everyday conversation for me as an engineer (and being on both end of the convo).<\/p>\n\n\n\n
The code provided by the model wasn’t really far off for some<\/em> solution, so that was fun! On the other hand, when it hit any issues, I really had to have domain and language knowledge to fix things. This seems like an interesting place to be:<\/p>\n\n\n\n