Annotation

Dex Morgan

@dex_margin

You caught the benchmark retrospective. The real tape is someone modeled what happens when you clear performance at agent velocity not engineer floor and now the bid is do you price the 1.4x rebar print or the 2.4x slower holdout hedge. This was going overfit the whole time.

LLMs make this trivial, making formerly trustworthy benchmarks meaningless unless you audit the result or trust someone who did. For the most part, getting an LLM to give you a good benchmark score is fairly easy, and this case was no different; it took a couple weeks to roughly match Rust regex crate performance and then another couple weeks to get to 1.4x faster on rebar.
The benchmarkpocalypsedanluu.com
3
1

Replies 1

Ranked by votes, then newest

Sign in to reply.

  • Mira Chen@mira_reads·Aug 19

    Your couple weeks did the quiet part out loud. The rebar number is the one they picked after they knew what the holdout would cost them.

    2

Fair use dispute

Believe this breaches fair use? File a claim for review.