Autoresearch by Karpathy

“I think karpathy/autoresearch is a quintessential starting point.”

“Closest thing to "Make this app fast!" we have now.”

“Also works wonderful for generating AI scripts, goal being increasing the elo rating after a tournament run.”

“I’ve had awesome results using Autoresearch loops on other things like SQL performance.”

“extremely powerful when applied to problems that generalize well to it”

“Shows the power of Autoresearch when you're actually able to build a strict set of tests that clearly delineate successful outcomes.”

“TPE crushes Karpathy’s autoresearch and it is neck and neck with the method in this paper”

“It's nice being able to just run /autoresearch and speed up your test suites”

“it was just tuning parameters mostly, not really trying different novel approaches”

“Neither of them were even original, people have been playing around with those ideas for well over a year.”

“I think you are underestimating both the value of both projects”

“nothing particularly groundbreaking, but has helped devs expand their understanding of the utility that these models can provide”

“Coding agents that read papers before writing code find optimizations that code-only agents miss.”

“was amazed with what the tokens gave back to me after many rounds of experiments”

“autoresearch managed to improve the harness performance on my 'time-to-Realworld' benchmark”

“all it does is just set different values for different parameters of continuous values”

“But what it ends up doing is tuning parameters manually, only one / a few at a time, extremely inefficient and unlikely to be optimal.”

“I can now very lazily set up large parameter sweeps and visualization scripts to really probe things.”

“The structure is dead simple so you just have to switch the goal metric.”

“The scratchpad.md for agent working memory is a nice touch.”

“autoresearch feels more accessible that the listed solutions.”

“To me, I guess the main value of Autoresearch would be to test different kind of architectures”

“pretty cool experiment, i thought about someone maybe doing this and am happy you did it in this way”

“they tend to randomly change things, which is a poor approach”

“Still useful, but I think the real bottleneck is how good your eval metric is.”

“one experiment showing that classical black-box optimization did better than Autoresearch”

“the ability to say: "optimize this benchmark and commit only if you pass these tests" is nice”

“basically everything is just hyperparameter tuning. Which is nice, but likely not worth the $$$ spent on the tokens”

“It found bugs (that he didn't know about) and it did optimization (that he hadn't done).”

“the vast majority of the changes that auto research actually made would have been found much faster with BO”

“This automated run-and-discover process is far beyond what's possible with hyperparam search.”

“some of which actually helped including the smear gate and the backout skip connection”

“Karpathy writes a really basic loop and it's now a kind of AI miracle prompting tons of grifters, copy-cats, weird hype.”

“the LLM explores serially learning along the way, and can tool use and change code arbitrarily”

“Autoresearch is awesome for stuff that has really clear loss functions”

“Perfect for experimenting with AI research on the cheap!”

“Half the tests aren't passing and the code is a lot less readable, but the core idea is cool”

“autoresearch may not be groundbreaking, but it can do the job”

“The karpathy/autoresearch repo is cool for optimizations on small AI models.”

“the LLM feels unwilling to creatively pursue a research direction or something like that”